windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v393/Hydrus.Network.393.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v393/Hydrus.Network.393.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v393/Hydrus.Network.393.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v393/Hydrus.Network.393.-.Linux.-.Executable.tar.gz
EDIT: This release had a hotfix two hours after initial release to fix the CloudFlare code. If you got it early, please redownload. The links are the same, the build is new.
I had a good week catching up on messages and small jobs. There is also a (hopefully) neat prototype solution for some cloudflare issues.
cloudflare and network
CloudFlare hosts content for many sites online. They have a variety of anti-DDoS tech, normally a variation on a 'is this a web browser?' test, that sometimes stops hydrus from downloading. If you have seen unexplainable 503 errors on a site that works ok in your browser, this may have been it.
This week, I am adding a library, cloudscraper, to my network engine to try to solve this. Now hydrus will attempt to detect CF challenge pages when they are downloaded and pass them off to this new library, which attempts to solve the javascript challenge (the part where a CF site sometimes says 'this process may take five seconds…', just as a browser would, and then copies the solved cookies back into the hydrus network session.
This first version can only solve the simple javascript challenges. It cannot do the more serious captcha tests (which usually appear when a site is under active attack) yet, but this is a possible future expansion. I would appreciate any feedback from users who have had bad CF problems. If a page cannot be solved, or if your entire IP range had been flat-out blocked, hydrus should now attach new CF-appropriate error messages.
For users who run from source, this library is optional. You can get it through pip, 'cloudscraper'.
Additionally, I am rolling in the very first basic version of 'this domain is having trouble' tech to my network engine. Now, if you keep having connection errors or CF issues with a site, hydrus recognises this and slows down access for a bit. The default is that no new network job will start as long as its site has had three serious errors in the past ten minutes, but you can edit this rule, including turning it off completely, under options->connection. Subscriptions will also try to wave off sites having trouble, just as if you had hit bandwidth limits. I expect to extend this system significantly in future, particularly by adding UI to see and manage current domain status.
the rest
The Windows build of hydrus is now on a newer version of python, 3.7 instead of 3.6. This makes for a variety of small background improvements all over the place, and some updated libraries, but may also introduce an odd bug here or there in more rarely used hydrus systems. Let me know if you run into trouble!
I did another round of file search optimisations on searches that do not have tag search predicates, such as bare system:rating or system:inbox queries.
Thanks to some excellent work by a user, there is another new DA parser. This one gets full-size images, video, flash, and even pdf! The only proviso is that it needs to be logged in to DA to get most content, otherwise it 404s. I believe the hydrus DA login script works, but if you have trouble with it, Hydrus Companion is always a good fallback.
The blacklist in tag import options gets a bit of work this week. It now tests unnamespaced rules against namespaced tags, so if you have a rule for 'metroid', it will also block 'series:metroid'. This is a special rule just for the tag import options blacklist (it also tests all siblings of the test tag), and to reflect it, the tag filter/blacklist edit panel now has a second 'test' text to show how the test input applies to a TIO blacklist, as opposed to a regular tag filter.
This is a small thing, but if you are mostly a keyboard user, the mouse now hides on the media viewer without having to be moved!
If you have had crashes trying to open a file or directory picker dialog, try the new BUGFIX checkbox under options->gui. It tries to use a different style of dialog that may be better for you.
full list
- cloudflare and network:
- the hydrus client now has an experimental hook to the cloudscraper module, which is now an optional pip module for source users and included in all built releases. if a CF challenge page is downloaded, hydrus attempts to detect and solve it with cloudscraper and save the CF cookies back to the session before reattempting the request. all feedback on this working/breaking irl would be welcome. current expectation for this prototype is it can pass the basic 'wait five seconds' javascript challenge, but only a handful of the more complicated captcha ones
- if a CF challenge page is not solvable, the respective fail reason for that URL will be labelled appropriately about CloudFlare and have more technical information
- .
- the hydrus network engine now has the capability to remember recent serious network infrastructure errors (no connection, unsolvable cloudflare problem, etc..) on a per domain basis. if many serious errors have happened on a domain, new jobs will now wait until they are clear. this defaults to three or more such errors in the past ten minutes, and is configurable (and disableable) under options->connection. this will be built out to a flexible system in future, with per-domain options+status ui to see what's going on and actions to scrub delays
- basically, if a server or your internet connection goes down, hydrus now throttles down to limit the damage
- subscriptions now test if a domain is ok in order to decide whether they can start or continue file work, just like with bandwidth
- serverside bandwidth alerts (429 or 509) are now classified as network infrastructure errors
- I expect this system will need more tuning
- .
- the hydrus downloader system now recognises when an expected parseable document is actually an importable file. when this is true, the file is imported. this hopefully solves the situation where a site may deliver a post url or a file
- .
- the rest:
- the windows build of hydrus is now in python 3.7.6, up from 3.6. this rolls in a host of small improvements, including to network stability and security (e.g. TLS 1.3), and possibly a couple of new bugs in more unusual hydrus systems
- similarly, all the windows libraries are now their latest versions. opencv is now 4.2
- greatly sped up several file searches that include no tags such as bare system:rating, most system file metadata predicates, or bare system:inbox, when the result size is much smaller than the total number of files in the file domain
- thanks to some excellent work by a user, the Deviant Art downloader gets another pass–it can now get high res versions of images where they are available, and video, and flash, and pdf! the only proviso is that you need to be logged in to DA to get most content, otherwise you get 404. the current hydrus DA login script _seems_ to work ok
- tag import options blacklists now test unnamespaced rules against namespaced tags. so if you blacklist 'metroid', a 'series:metroid' will be caught and the blacklist veto signal sent. this can be escaped with the 'advanced' exception panel, which now permits you to add 'redundant' rules
- the edit tag filter panel now explains the blacklist rules explicitly and has a second 'test' green/red text to display test results for a tag import options blacklist, with the new sibling and namespace check
- added some unit tests to test the new tag import options blacklist namespace rule
- when 'default' tag import options are set, the edit panel now hides the per-service options, rather the the previous disable
- the system tray icon now destroys itself when no longer needed, rather than hiding itself. it should now be more reliable in OSes that do not support system tray icon hide/show. if your OS still doesn't get rid of them, and you get a whole row of them, I recommend just leaving it always on
- the system tray now has a tooltip with the main hydrus title and pause statuses
- the timer that hides the mouse on the media viewer is now fired off when the window first opens (previously it would only initiate on the first mouse move over the window), so users who navigate mostly by keyboard should now see their cursors nicely hide on their own
- added some semi-hacky import/export/duplicate buttons to edit shortcuts. I'll keep working on this, it'd be nice to have import/export for whole shortcut sets
- added a semi-hacky duplicate button to the 'manage http headers' dialog
- the 'clear' recent tag suggestions button is now wrapped in a yes/no dialog
- a new checkbox under options->gui now lets you set it so when new cookies are sent from the API, or cookies are cleared, a popup message summarises the change. the popup dismisses itself after five seconds
- the client api now also returns 'ext' on /get_files/file_metadata calls, just as a simpler alternative if the 'mime' is a pain
- fixed a bug when petitioning tags through the client api, with or without reasons
- fixed an error where subscriptions that somehow held invalid URLs would not be able to predict some bandwidth stuff, which would not allow the edit subs dialog to open
- the string transformation dialog's step subdialog is now ok with example strings that are bytes. even then, this str/bytes dichotomy is an old artifact of python 2 and I will likely clean it up sometime so string transformers (and downloaders) only ever work utf-8 and hashes just work off utf-8 hex
- added a BUGFIX checkbox to options->gui that tells the UI to use Qt file/directory picker dialogs, instead of the native OS one. users who have crashes on file selection are encouraged to try this out
- updated running from source help with cloudscraper, a new pip masterline, and some windows venv info
- the 'import with tags' button on 'import files' dialog gets another rename for new users, this time to 'add tags before the import >>'. it also gets a tooltip
- handled an unusual rare error that could occur when switching out a media player inside a media viewer, perhaps during media viewer shutdown
next week
Next week is a cleanup week. I'd like to go into the tag autocomplete code and clean up how the input text is parsed and how results are cached and filtered. Otherwise, I'll do more wx->Qt cleanup and catch up on some small jobs.
