windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v319/Hydrus.Network.319.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v319/Hydrus.Network.319.-.Windows.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v319/Hydrus.Network.319.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v319/Hydrus.Network.319.-.OS.X.-.Extract.only.tar.gz
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v319/Hydrus.Network.319.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v319.tar.gz
I had a mixed week, but I got some good work done. It is mostly improvements to the downloaders, and then some stuff for advanced users to play with.
downloaders and urls
Following the success of the new gallery page 'delay' system, where gallery page hits across the program are spread out over time, I have made several improvements: the system's logic is more reliable, it occurs on a per-domain basis, the waiting period and status updates now occur in the network engine itself (so will appear in the network job control ui–I expect to add some kind of 'override wait' action to the cog menu soon), and the thread watcher now also has a delay similar system, also under options->downloading. Also, the network engine will now not permit more than three simultaneous jobs per domain as a global bottleneck, and the overall total num connections is boosted as a result to 15 (from 10). In general, the network code should be just a bit more pleasant and less CPU-spiky while still serving a decent throughput when there is work to be done.
The right-click->share->copy->urls menu on thumbnails and the media viewer is now much more powerful. For the focused file, it can now copy (all/all recognised) URLs, and for a thumbnail selection, it can copy (all/all of a specific URL Class) URLs. So, if you want to select a hundred files and get all their Deviant Art URLs in your clipboard real quick, it is now easy!
Also, the client now does some additional normalisation of URLs (it alphabetises query strings on all URLs, not just those matched in the new system). It will take a minute or two to convert and merge your existing DB URLs on update.
Due to some of the phases of network engine update and normalisation, some subscriptions have been hitting their 'periodic' limits by accident. If you have seen some of these, or get more this week, please do not freak out unless you get additional errors–it should all blow over in another sub cycle.
searchers, or gugs as I like to call them
I have finished the first version of the 'searcher' object. I have also decided to call it the Gallery URL Generator (GUG), as this more precisely describes what it does. Advanced users may like to check the new not-plugged-in dialog under network->downloader definitions->manage gallery url generators to see what I am going for. Let me know if there is anything you think this object needs that I have forgotten!
Assuming this goes well, I'll plug this in next week and we'll start experimenting for real.
full list
- started the new convert-query-text-to-gallery-urls object. these objects, which I was thinking of calling 'Searchers', will be called the more specific and practical 'Gallery URL Generators', or GUGs for short
- the first version of GUGs is done, and I've written some test ui for advanced users under network->downloader definitions->manage gugs. this ui doesn't save anything yet, but lets you mess around with different values. if we don't think of anything else needed in the next week, I will fix this code for v320 and start filling in defaults
- watchers now have a global checking slot, much like the recent change to galleries and subs. it safely throttles dozens of threads so they don't rudely hammer your (or the destination server's) CPU if they all happen to want to go at once (like just after your computer wakes up). the option is similarly under options->downloading, and is global for the moment
- moved the new gallery delay/token management code to the better-fit bandwidth manager (it was in domain manager before)
- the gallery delay/token code now works per-domain!
- moved the gallery delay/token checking code into the network job proper, simplifying a bunch of import-level code and making the text display now appear in the network job control. token consumption now occurs after bandwidth (it is now the last hoop to jump through, which reduces the chance of a pileup in unusual situations) I expect to soon add some kind of 'force-go' action to the cog menu
- the network engine will now not permit more than three jobs active per domain, and the overall limit has been raised from ten to fifteen
- the media right-click menu now supports copying: all of a files recognised urls; all of a files urls; all selected files' urls of a specific url class; and all selected files urls
- reworked and harmonised a bunch of urlparsing and generation code–all urls should now appear as full unicode across the program, generally without %20-type encoding characters unless explicitly entered by the user. character encoding now all happens on the backend in requests
- non-url-class-matched urls now have their query parameters alphabetised as part of the normalisation process
- all urls in the db will have their query params alphabetised on update, and any file relationships merged to the new/existing normalised url
- the manage urls dialog will now normalise newly added urls (but should also still permit the removal of non-normalised urls)
- reworked how gallery hits update file import object caches, particularly for subscriptions
- fixed an issue in subscriptions gallery logging where the gallery log would always state it had found the max number of files and typically redundantly generate an 'ignored' stub–it should now say something like 'found 7 files - saw 5 previously seen urls, so assuming we caught up' as originally intended
- simplified some gallery->file import object creation
- galleries now compact until 100 entries (was 25)
- watchers now gallery-compact after a successful check
- watchers now show the 'just added'/'already watching' status for 15s, up from 5s
- network report mode now reports three time–once each for job addition, start, and successful completion
- fixed an issue with the new 'max width' popup sizing calculation that was sometimes not fitting for new height requirements correctly
- fixed an issue with the new url class next page generation code
- fixed an issue where TIOs with data regarding since-deleted services were failing to initialise at the ui level
- misc status text cleanup
next week
I want to finish all the remaining 'default' gallery parsers. If I can get them all out and get GUGs actually working ok, I might be able to finish off the rest of the GUGs for v321, auto-convert all the legacy boorus people have to the new system, and completely delete the old downloader code! Fingers crossed, we are absolutely in the end-game of the downloader overhaul now.