[ / / / / / / / / / ] [ dir / asmr / egy / fur / htg / ita / polk / waifuist / zoo ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog

Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 12 MB.
Max image dimensions are 10000 x 10000.
You may upload 5 per post.

New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Current to-do list has: 835 items

Current big job: last bit of duplicate system, preparing for downloader engine overhaul

YouTube embed. Click thumbnail to play.

f70544 No.5825

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v256/Hydrus.Network.256.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v256/Hydrus.Network.256.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v256/Hydrus.Network.256.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v256/Hydrus.Network.256.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v256/Hydrus.Network.256.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v256.tar.gz

I had a mixed week, but I am overall happy with my work. I put some more time into the gelbooru problem and polished the duplicate filter.

gelbooru back to normal

The 'redirect' urls that the gelbooru parser was generating last week are now gone. The client will figure out the correct urls and put them in the url cache just as it used to work before. Also, any old redirect urls will be removed from all your downloader pages and subscription when you update. EDIT: one user reported just now that the old urls were not removed correctly. If this happened to you, please let me know any details, and if it isn't a privacy issue, send me a couple of the urls that you still have.

Hopefully this patch will last us until the downloader overhaul, but let me know if it breaks again!

duplicate filter almost done

I've cleaned up and completely finished a lot of the duplicate filter code. Some unusual bugs and laggy moments are fixed, the way pairs are selected and presented is improved, and the db and gui do some more trickery to save you time.

The client now makes some simple automatic guesses about which file is 'better'. It tells you on the top hover window what it thinks–whether one file has larger resolution, or more tags, for instance–and then puts the better file as the A in the A-B pair you judge. A is often a nice high-res image, and B is often a no-tag, newer, scaled down version.

I think there will be two more weeks of work, and then I will be done with duplicates.

misc

You can now drag and drop text onto the program, and it'll automatically put it into a url download page! It works great for the text links on imageboards, for instance–just drag the link onto hydrus and hit enter. Try it out!

You can now 'append' gui sessions, which will load the contents of the session without deleting whatever was open before. This seems to work very well for some jobs (like opening a few 'favourite' pages without breaking your current workflow, or opening a bunch of empty thread watcher pages all set up and ready to go), and I think it may be cause to rename the 'sessions' system to something else, something like 'bookmarks'.

full list

- the duplicate filter now loads new pairs off the gui thread. it will display 'loading pairs…' during this time

- media viewers of all kinds are now more comfortable displaying no media (when this occurs, it is usually a frame or two during startup/shutdown)

- the duplicate filter now responds to any media_viewer_browser navigation commands (like view_next) with a media switch action

- you can now alter the duplicate filter's background lighten/darken switch intensity from its top hover window's cog icon

- fixed a bug in the new dupe pair selection algorithm that was preventing pairs from being presented as groups

- the duplicate filter will now speed up workflow by automatically skipping pairs when you have previously chosen to delete one of the files in the current batch

- auto-skipped pairs _should_ be auto-reverse-skipped on a 'go back' action

- added a |< 'go back' index navigation button to the duplicate filter top hover window

- the duplicate filter now displays several 'this file has larger resolution'-type statements about the currently viewed file. it lists them on the top hover window and in the background details text underneath

- the duplicate filter _roughly_ attempts to put the better file of the two first. this will always be indexed 'A'

- the duplicate filter now shows done/total batch progress in its index string–not sure how clear/helpful this ultimately is, so may need to revisit

- an unusual bug where Linux would spam the 'All pairs have been filtered!' duplicate filter message over and over and then crash _should_ be fixed–the filter no longer waits for that message to be OKed before closing itself

- drag-and-dropping text onto the client will now a) open a url import page if none already exist and b) put the dropped text into the input box of the first open url import page (and focus it, so you can quickly hit enter)! This works when dragging text links from browsers, as well

- you can now 'append' gui sessions, which will just append that session's tabs to whatever is already open–say, if you have several 'favourites' pages you want to be able to quickly load up without having to break your existing workflow

- ipfs services now have a 'check daemon' button on their review services panel which will test the daemon is running and accessible and report its version

- fixed the 'test address' button for ipfs services on their manage services panel

- the client can now automatically download files it wants and knows are on an ipfs service

- middle-click on an 'all known files' domain thumbnail will now correctly start a download (as long as a specific remote file service is known)

- the multihash prefix option is reinstated on ipfs manage services panels

- the gelbooru parser now discovers the correct page url to associate with its files

- wrote some redirect fetching code to fix the gelbooru bad urls issue

- discovered a quicker fix for the gelbooru issue–the redirect location is the garbage in the original url in base64

- all downloader/subscription url caches will purge any old gelbooru 'redirect.php' urls on update

- fixed an issue where 'previously deleted' gallery/thread imports were returning 'fail'

- fixed a problem that was causing some redundant laggy work in adminside petition processing

- thread watchers will now remember their file and tag import options through a session save even when no thread url has yet been entered

- fixed an issue where media 'removed' from a media viewer view of a collection resulted in the entire collection being removed at the thumbnail level

- fixed an issue where media deleted from a media viewer view of a collection resulted in the media not being correctly removed from selection tags

- tag, namespace, and wildcard searches on a specific file domain (i.e. other than 'all known files') now take advantage of an optimisation in the autocomplete cache and so run significantly faster

- fixed a hover window coordinate calculation issue after minimising the media viewer on some platforms

- removed some 'all files failed to download' spam that could sometimes occur

- misc fixes

next week

I can see the end of the tunnel on the dupe filter. Main things remaining are a system predicate to search files with dupe relationships and a workflow from thumbnail right-click to assign dupe relationships manually en masse.

I also have jury duty, which is likely to only be one day.

36d7fa No.5826

Happy version 256, Hydrus!


391a56 No.5829

>EDIT: one user reported just now that the old urls were not removed correctly. If this happened to you, please let me know any details, and if it isn't a privacy issue, send me a couple of the urls that you still have.

Same deal here; most of my subs had only picked up a few of these but one picked up about 3,000. Not sure how to send you urls, or where to find them…


58e47b No.5831

So what's the easiest way of putting all my subsriptions back to download everything, instead of the 50 you forced last week? Manually going through hundreds of subscriptions doesn't sound very appealing.


30b088 No.5832

>>5831

You can select multiple subscriptions in the list, then click pause/resume, or reset if you want to re-download everything


58e47b No.5834

File: 6e667e48ff6eee0⋯.png (2.32 KB, 453x28, 453:28, Capture1.PNG)

>>5832

I'm talking about this option.


6d516e No.5836

I can't paste and write urls in thread watcher after the update


89bc12 No.5842

File: 67c704f05bfc6b2⋯.jpg (1.91 MB, 3257x2552, 3257:2552, 1 (5).jpg)

I'm not sure using the number of tags already on an image is a sensible metric for choosing the A/B in a pair. It seems like the logic is that "the version you already have" will be better, but if you're manually bringing in a new copy, it's probably to replace the lesser one you have. And in my (possibly odd) case, I give all subscription imports a subscription:whatever tag until I'm done processing them, so it ends up ordering them on which version of new files happened to come in on the most subscriptions at once, which is basically just back to random, despite these files usually having a clear winner that aligns with the clear winner in filesize.

On a related note, I'm not sure if this is information that the image library can provide you in the first place, but if so, adding a metric (and some simplified user feedback) on the best/worst quantization ranges used in jpgs would be handy. Especially for images that are old enough to have dupes saved by vastly different encoders (or more to the point, using differently built quantization matrices), it's sometimes difficult to determine which is better between similarly detailed jpgs where one gave weight to a different frequency than the other. The total remaining data isn't always apparent by eye, in these cases.


d193c5 No.5845

>>5836

Same. I'm on windows.


33af9f No.5848

>>5836

>>5845

also on linux ubuntu 17.04


ff3dc3 No.5849

>>5836

same

they now reflect the tags saved etc. as needed, but i can't actually use them…. :x


ff3dc3 No.5850

any chance of an early hotfix for the thread downloader bug rather than waiting on a full release, since it seems to just be a single switch as to whether the text box is locked?


33af9f No.5851

\@@ -2338,19 +2342,19 @@ class ManagementPanelImporterThreadWatcher( ManagementPanelImporter ):

\+ self._thread_input.SetEditable( False )

this might be the cause


33af9f No.5853

@@ -2345,7 +2345,6 @@ class ManagementPanelImporterThreadWatcher( ManagementPanelImporter ):

( thread_url, import_file_options, import_tag_options, times_to_check, check_period ) = self._thread_watcher_import.GetOptions()

self._thread_input.SetValue( thread_url )

- self._thread_input.SetEditable( False )

self._import_file_options.SetOptions( import_file_options )

self._import_tag_options.SetOptions( import_tag_options )

@@ -2354,6 +2353,7 @@ class ManagementPanelImporterThreadWatcher( ManagementPanelImporter ):

self._thread_check_period.SetValue( check_period )

if self._thread_watcher_import.HasThread():

+ self._thread_input.SetEditable( False )

self._thread_watcher_import.Start( self._page_key )

not quite sure if there is any sideeffects


359fe4 No.5855

>>5853

This works, thank you.


ff3dc3 No.5859

>>5855

how does one use this


359fe4 No.5864

>>5859

Edit "ClientGUIManagement.py" inside the include folder.

Move the line "self._thread_input.SetEditable( False )" from the - position to the + position


ff3dc3 No.5867

>>5864

doesn't seem to do anything… do i need to compile it somehow?


d193c5 No.5868

>>5864

Could you please upload that file instead?


ff3dc3 No.5871

@hydrus_dev

okay so this is bizarre as fuck

i finally got it to run from source and fixed the problem…

and hydrus has never run so fast. things that used to choke my machine half to death are breezy, and files import/download between 2-8x faster, everything's more responsive.

not sure wtf is up, but running from source vs. running from the .exe is a MASSIVE step up in performance


8ad166 No.5874

>>5867

>>5868

https://my.mixtape.moe/uoxqiv.py

Save as ClientGUIManagement.py inside the include folder if you're running from source. No idea if the compiled exe version has a fix easy as this though.


30b088 No.5883

Minor bug: In the duplicate filter, if A and B have the same number of tags hydrus will say "Both files have tags, but this has fewer." on both of them. In my case they both had 1 tag.


30b088 No.5886

Another bug: If I open the manage tags window in the duplicate filter, once I close the window the focus moves to from the filter to the main Hydrus window instead, have to alt-tab back. On Windows 7.


8b7fd1 No.5887

File: 496529d193430e3⋯.png (69.17 KB, 1024x510, 512:255, 2014-08-19-336.png)

dev, I was wondering, whenever I zoom in an image to check if it has compression artifacts and such, If I get too close (like 1600% zoom) regardless of file size or format it slows down the computer to a crawl, and god forbid I clicked once more after that, it won't crash but it won't be responding to anything else either, not even the mouse cursor moves. Could you look into it?


8b7fd1 No.5888

>>5887

Another thing I've noticed is that when you zoom in png files you can see the pixels, like it's jagged, while jpeg files are all smoothed out. I tested it with a couple of images I created and it looks like it's not because of compression. Is this the intended behaviour?


30b088 No.5889

>>5888

That is the default behavior as set up in options > media > media viewer mime handling, you can change it there. Why Hydrus_dev decided on that I do not know.


8b7fd1 No.5891

>>5889

Thanks man, I didn't understand anything of what it said there so I just left it on its own.


8ad166 No.5892

>>5891

Go to options > media > media viewer mime handling, double click on the png entry, and set "interpolation quality" to "8x8 lanczos interpolation".


f70544 No.5893

File: a51d56ea15eeb60⋯.jpg (45.02 KB, 460x400, 23:20, a51d56ea15eeb60462c8f8001f….jpg)

I'll post a proper reply to this thread later, but I will say for now that the thread watcher issue is fixed for v257. I apologise for the inconvenience. It slipped through my testing laste week because of a quirk.

If it is very important to you, it is possible to roll back to v255 this week.


f70544 No.5917

File: 5ed7e6365cc0828⋯.jpg (818 KB, 2640x1793, 240:163, 5ed7e6365cc0828fa68bfcb821….jpg)

>>5829

Thanks–I've broadened the purge test and rescheduled it for v257. Let's see if it works this time!

You can see the urls on the download context (whether that is a gallery page of the 'edit subscription' dialog) and clicking the little icon button that has a picture of a multi-column list on it. That'll show the current contents of your url cache, from which you can right-click->copy some source urls.

I know some 'http' (as opposed to 'https') urls got through the last test, but I understand some others were also missed. I am not sure what went wrong there.

>>5831

>>5832

>>5834

There isn't a quick way to mass-apply this atm. I am not confident that the gelbooru patch I have made will hold, and the current downloader engine cannot handle that problem well, so I do not advise you go much higher than 50 for now. If they change their url scheme again, it will double all of those subscriptions. If your subs are producing more than 50 new files per check cycle, I imagine doubling and redownloading their current cache would be disastrous.

If these really do produce more than 50 urls per check, I suggest lowering the check period rather than raising the periodic file limit. I'm afraid there isn't a quick way to mass-apply this either.

>>5836

>>5845

>>5848

>>5849

>>5850

>>5851

>>5853

>>5855

>>5859

>>5864

>>5867

>>5868

>>5874

I messed this up. I apologise. I shuffled the code around to always preserve the tag and file options but accidentally put the 'hey, there's a good thread on init, better disable the textbox so it can't be overwritten' up there as well. I believe I missed it in testing because I was working with a thread watcher page that already had a url in it from when the code was working, so I never noticed a fresh one couldn't be edited. I'll see about making an automatic test to ensure this doesn't happen again.

Editing the source in the include directory only works for people running from source (i.e. running client.py) That same source is frozen and bundled into client.exe and is what that uses.

Easiest solution here is to roll back to v255 for now. There are no db changes this week, so rolling back should be fine.

>>5871

That's interesting. I generally experience the same speed for exe vs source, so I'm not sure what is going on here. Would you mind posting (with a screenshot or whatever) your help->about window so I can see the library versions you have listed? The sqlite one is mostly what I am thinking about. Are you using a straight regular python from python.org, or a special version?

>>5883

>>5886

Thank you for these reports. I will try to have these fixed for v257!

>>5887

The code that actually puts an image to screen is very old and does not do its job very efficiently. At the moment, it blows the whole image up to 1600% and then renders the smaller section to screen, whereas I would like to rewrite it to only zoom in on the clipped region and draw that and then recalc if you drag the image around.

Until I put the time in on this, there isn't a good solution. If you do big zooms, it eats memory and CPU for now.

>>5889

The original thought here was to make pngs zoom in 100% intervals and simpler quality to preserve the sharp lines they usually have, but it ultimately turned out shit so I changed the new default to just be the same as jpgs. Old clients that were around for the original options update still have those old defaults.


ff3dc3 No.5926

File: b9fb46a30601c46⋯.png (32.32 KB, 473x374, 43:34, hydruslibs.png)

>>5917

using normal python - maybe it's Pillow? I am showing 4.0.0 for exe hydrus, and 4.1.1 for py hydrus


1b68ab No.5934

>>5825

The thumbnail makes me think of the Daft Punk girls


f70544 No.5937

File: aeb4f1eb76038d8⋯.gif (2.44 MB, 1024x768, 4:3, aeb4f1eb76038d87380b3af299….gif)

>>5926

Thanks. I've now updated my Pillow as well. I am not sure, but I think files might be importing faster. PIL isn't used much any more in the program except as backup, but maybe there is some thumbnail generation or something I am forgotting about. We'll see if anyone else notices an increase with the build!

I will keep thinking about this. Please keep your eyes open as well and let me know if you discover anything new.


ff3dc3 No.5941

>>5937

thumbnail loading / imports were actually where i saw the biggest speed boost…

so yeah, probably that!



[Return][Go to top][Catalog][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / ] [ dir / asmr / egy / fur / htg / ita / polk / waifuist / zoo ]