windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v310/Hydrus.Network.310.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v310/Hydrus.Network.310.-.Windows.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v310/Hydrus.Network.310.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v310/Hydrus.Network.310.-.OS.X.-.Extract.only.tar.gz
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v310/Hydrus.Network.310.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v310.tar.gz
I had a productive week. A bunch of stuff is fixed, and I finished a first version of multi-file url downloading.
multi-file support
A limitation of the old download system is that it always assumed a single 'post URL' could only ever produce a single file. This week, I have added full support for multi-files-per-post-URL to the new downloader. If a new parser generates multiple 'downloadable' URLs, the file import queue will generate and insert new download objects for each. Tags and associable URLs should all be passed along to the children as appropriate.
As a result, I can finally roll out support for several sites that provide files in this way. I've updated the inkbunny post parser, and added twitter tweet and artstation post parsers, so all these should now be drag-and-droppable onto the client. The twitter parser only supports images–no videos yet. The inkbunny parser may put its multi-files slightly out of order, and in one of my tests it pulled the artist profile picture as well, so it may need some tweaking. IB have a proper API, but it requires some login stuff, so I will look at that again when I have the new login system done.
And of course I have added a pixiv manga parser, which has been highly requested for a long time. This was fairly complicated, but I think I got all the referral URLs and tags lined up correct. It can even add 1-indexed page tags. Unfortunately, pixiv just changed my test login to the new (currently broken) JSON/javascript layout just today, as I did final testing. If it turns out pixiv have switched everyone over to the broken layout on the very day I finally roll out manga support, I think my head is going to explode, but even if so, I am confident the javascript layout is parsable–it'll just take a bit more work. I expect to have a fix for all pixiv users for v311 if another user doesn't post one sooner.
This multi-file parsing is a first version. There may be bugs, so please let me know. I also don't really like how the initial post URLs' file import objects get counted as 'successful' like a regular file, so I may make a new 'successfully parsed' type to distinguish them a bit and alter the file progress counts appropriately.
misc
The new 'x% in' video thumbnail generation works better and fails better now, including some better workflow and error-reporting when regen is manually started by the user. If you had some videos that failed to import or regen last week, please try them again and let me know if you still have problems.
The multiple watcher now has a 'check now' button.
I added a semi-crazy prototype checkbox to options->gui that puts all your page tabs on the left. It needs a restart to kick in. Test it at your own risk.
The new download system now informs the new network engine of file limits in your 'file import options'! So, if you say 'don't get gifs bitter than 32MB', and the server clearly identifies a newly started download as a gif with size 50MB, the download will be abandoned and a veto/ignored status set immediately! It also works with the regular min/max filesize as appropriate. Let me know if it goes wrong!
I cleaned up some file repository thumbnail sync and display code all around.
github
Github got bought by Microsoft, so I am considering migrating somewhere else. I only use Github as a file/code host and the workflow of syncing there is easy, so I may or may not go, or may just put it off and see what happens during the transition. I am interested in your thoughts on the whole deal and what you think of the alternatives.
full list
- updated the inkbunny file page url class to acknowledge that inbunny pages can have multiple files
- updated the inkbunny file page parser to handle multiple file urls (although they may be out of order and possibly sometimes include the artist profile image–this was not super easy)
- added a parser for twitter tweets (only images supported atm, but it can handle multiple!) (hence tweet drag and drop now works!)
- updated the artstation file page url class to redirect to a new api url class
- wrote an artstation file page parser that also handles multiple file urls
- updated/added pixiv file page, manga page, and mange_big url classes
- updated pixiv file page parsers to be ok with manga links
- wrote parsers for pixiv manga and manga_big pages to fetch manga files (with page tags)!
- file import objects can now create semi-duplicate children for multi-file post urls and insert them just after themselves in the file import queue.
- file import objects can now receive and remember referral urls. this referral url is associated with the file if appropriate. the watcher and simple downloader now uses this in addition to the multi-file post system
- jumbled around some parameters and merged the two new file import url commands (import 'file' vs import 'post') into one single simple 'work on this url, thanks' call that is now used across the program
- the parsing system's 'content parser' no longer fetches file urls and post urls, but 'download urls' and 'source urls'. this helps some pipeline logic and also lets post urls be download urls
- when file import objects parse post urls as the urls to download, it now creates 1-n new import objects, just like if multiple file urls.
- improved some file import object file association code
- the new parsing system will de-dupe parsed urls
- refactored the 'seed' code, which handles all basic file import objects, to the new ClientImportSeeds.py
- added a new string transformation type, 'integer addition', for shifting page number tags up and down
- fixed thumbnail generation for some videos that failed to do the new x%-in generation–it reverts more reliably just to the old frame 0 method
- file reparsing popup now has a stop button
- fixed an issue where extremely thin or wide (ratio > 200:1) images would not generate a full-size or resized thumbnail
- the file reparsing/re-thumbnailing now reports errors better (including with full path) and does not abandon the larger job as it works
- misc thumbnail generation code improvements
- improved some thumbnail and file regeneration/moving code when the existing file has read-only status
- the multiple watcher now has a 'check now' button
- added a checkbox to options->gui that will put new notebook page tabs on the left
- for all file download network jobs working in the new download system, the file import options for min size, max size, and max size (gifs) are now applied _during the download_! if the server tells the client the exact file size in the response headers, it will test max and min size before the content is actually downloaded–otherwise, it will test the max size as it downloads. if the server clearly says the file is a gif, the max gif size rules will also be tested in the same way
- cleaned up some bandwidth announcement code–now, if bandwidth is due in less time than override time, that will now correctly be the status text
- the bandwidth status no longer says 'in in' typo
- fixed up some tag repair code from last week
- the 'print garbage' debug function now dumps a whole bunch more data to the log
- the thumbnail cache should now be a bit more stoic about missing repository thumbnails–it should now just present the hydrus default backup without error popup spam
- the repository thumbnail sync will now get as thumbs in blocks as high as 10k at a time, rather than the old 100
- hydrus network requests no longer generate web domain network contexts (and so won't have a default one-request-per-second bandwidth limit and should stream through thumbnails a bit faster)
- hydrus network services are now willing to wait longer for bandwidth, so big thumbnail queues should keep working even if other bottlenecks pause them for a bit
- hydrus network services will no longer sometimes have double-sync popups if synced from the advanced 'sync now' button in review services
- changed the default global 'stop-accidents' bandwidth rule of 120rqs per minute to 512MB per minute. this only affects new users, but users trying to sync to large file repos might like to make a similar change manually
- doing giant full file delete (i.e. purge from trash) jobs should now be a bit gentler on the gui
- improved how the client deletes paths, clarifying in the code when and when not to allow recycle (usually disabled for thumb disposal)
- switched the hacky text widgets on the popup system to a newer object. seems to still render ok, so lets see if it fixes some unusual layout issues some users have seen
- if the temp folder cannot be created on boot, the client will continue anyway
- fixed some url-domain text handling in db storage that was also breaking v309 update for some users
- fixed some additional domain generation error handling at the db level
- the list of url classes in the system:url panel is now the list of all url classes that are considered associable (before, it was file and post urls)
- if a url class now api-links to itself or otherwise forms a loop with n other api url classes, the client will now throw an error (rather than lock up in an infinite loop!)
- in the parsing ui, tag parse test results are now cleaned before being displayed
- fixed misc url matching error reporting bug
- when consulting the current file limit, the gallery page downloader will now try, when it has that number, to consult the total number of urls found it the current search (old behaviour is to only consult the number of _new_ urls, which lead to some bad edge-case workflows)
- misc refactoring
next week
I am going to take a light week next week so I can shitpost E3. I will try and keep up with messages, but I'll only do a little fixing work. I will get back to regular schedule on the 16th, which means v311 will be on the 20th. I will make a 'No Release Tomorrow!' post on the 12th.
I am a little ill and completely exhausted, so an easy week is coming at a great time.
I am still really enjoying making hydrus. Thank you for all your continued support!