windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.Linux.-.Executable.tar.gz
I had a good week. I fixed some important bugs, and cleaned up some core access and maintenance code. This week's changelog gets pretty technical, which you can safely ignore. Database works better™.
highlights
I fixed a problem importing files with 'only add tags that already exist' checked in tag import options. Sorry, this was a stupid typo. I added a unit test here to ensure it doesn't happen again.
When you search for potential duplicates from the duplicates page, this now happens in a non-interrupting popup in the bottom-right. You can keep searching and browsing while it works. All duplicate pages sync with each other better, too, and share more CPU work.
Thanks to a user's contribution and help, the 'discord drag and drop' BUGFIX mode, which is really a 'export files to a temp location before starting the drag and drop' mode, now supports the same 'filename pattern' you see in the export files dialog. Check it under options->gui. This is experimental, but the first step to automatic export with nice filenames. If you care about export filenames, let me know how it works for you, and what sorts of filenames you want to make in future. I really want to update the old 'filename pattern' system to something with conditional 'if' statements.
If you are a new user and had trouble syncing to the PTR in the past week with an error about an invalid tag, you got caught by my new 'invalid tag catching' code being overzealous. This is now fixed, so if your PTR automatically paused due to the error, please hit services->review services->PTR and then unpause it. When a repository gives an invalid tag (we think there are about 17 really broken tags in the PTR atm, from older days), the client now recovers.
I played around with core database modes this week. Default values are now tuned a little better for modern use. Unfortunately, the planned new mode for HDD users did not pan out due to architectural problems, but I think that HDD users will experience better write performance nonetheless. For those who want to experiment more, I have finally properly documented the different launch switches for hydrus here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html
full list
- misc:
- thanks to a user's contribution, added the export 'filename pattern' to the discord drag and drop mode, under _options->gui_. this lets you auto-rename files in this export mode. I like how this works, but the overall pattern-based filename creation system really needs updating. let me know how this works for you, and I'll finally start the job to update filename generation
- fixed a bug when importing files with the 'only add tags that already exist' filter active, and added a unit test so this should not fail due to a typo again
- fixed an issue where ctrl-selecting on taglists was weird, where any mouse movement during ctrl+click would deselect. drag select and deselect can now only start when the drag crosses two indices
- prototyped a basic profile mode for the client api. it is insufficient (due to the asynchronous nature of twisted), but a start
- when the client catches an invalid tag with the new error handling code, when it shows you that bad tag in a popup, it now clips that to 24 characters (some PTR invalid tags are just a few hundred null characters in a row, wew lad)
- the client now recovers from a repository giving it a new invalid tag definition. all such tags are, for now, called 'invalid repository tag'. a plan to auto-hide these tags clientside and fully eliminate them serverside will come later
- the clipboard url watcher settings should stick a bit more firmly. those users who had trouble, please let me know how you get on
- fixed an issue editing duplicate action options when they contained tag or rating preferences for services that no longer exist
- I think I fixed some issues getting autocomplete results when you type the whole namespace before moving on to the subtag. when you hit 'namespace:', it should invalidate the old cache and start a new search
- when the database is given content updates for services that no longer exist, those content updates filtered out of UI update broadcast
- fixed an issue where URL status check could fail when the url map contained orphan hash_ids. proper orphan clearance will come later
- reduced overhead of tag filtering, which should improve display speed of taglist for very large pages
- parents should now work through repository processing faster. periods of 2 rows/s at the end up of updates should be up to 100 times faster
- .
- duplicates search improvements:
- potential duplicate search now works in the background! it will not interrupt you and is easily cancellable. duplicate search pages disable their search buttons while it is going
- the search distance in duplicates pages is now synchronised across all pages–when one updates, they all do
- all the updates to potential search maintenance numbers are now routed through one cached manager. updates here are repeated less often
- misc cleanup for duplicates page
- .
- database modes:
- a new 'program launch arguments' help page now talks about all the available command line switches, here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html
- added the '–db_journal_mode' launch switch to set the SQLite journal mode. default is WAL, permitted values are also TRUNCATE, PERSIST, and MEMORY
- ensured –db_synchronous_override was hooked up correctly
- the old disk cache options under _speed and memory_ are removed, along with various deprecated disk cache load calls and code
- fixed some shutdown maintenance check logic that was saying 'I think a vacuum is due' when it wasn't actually true
- db_journal_mode, synchronous value, and no_db_temp_files is now shown in _help->about_
- .
- technical database nonsense:
- PERSIST is new to hydrus, and _may_ in future versions of SQLite be boost performance for HDD drives with larger databases (e.g. those that sync to the PTR), although unfortunately in our case (which uses multiple ATTACH databases), it seems current SQLite must ultimately treat this as DELETE, as here https://sqlite.org/atomiccommit.html#_clean_up_the_rollback_journals. damn
- hydrus now tries to always trim WAL (and PERSIST, if it worked) journal files down to 1GB after commits (which happen every 30 seconds), so giganto WALs should clear up promptly after big work is done
- hydrus no longer refreshes the database connection every thirty minutes, meaning WAL journal files will persist (and hopefully regularly clip back to 1GB when exceeded), which should improve some elements of long-running write performance, but may result in some surprise memory issues, we'll see
- in lieu of the db connection not refreshing, the memory database now reattaches every ten minutes, which _should_ stop it leaking in certain situations
- when in WAL journal mode, the hydrus db now cleans up any lingering checkpointing work every half hour
- after testing and feedback from users, the database is now default SQLite synchronous 1 (down from 2) when in WAL. the db is still consistent, so sudden program stop (crash, power cut) should not result in software-caused corruption, but the database may lose more than just the last 30 seconds of work. this speeds up tag processing in an SSD test environment by approx 33%
- the 'no_wal' (TRUNCATE) and 'db_memory_journaling' (MEMORY) launch switches remain valid but are now deprecated
- improved launch switch code generally
- boosted cache size for each of the four db files to ~200MB-this will likely become a launch argument in future, along with some other specific db values
- the client and server no longer disconnect from the db to check whether it is possible to vacuum databases
next week
There are two more work weeks in the year. I will continue working on small jobs and cleanup, but the main focus is now to update ancient network and service code to improve client-PTR sync and communications. I'll fix up buggy janitor tools and hopefully add some nice filters so clients and servers can better manage what they sync.
Since a 'next big work' poll is coming up on Christmas, on Saturday I'll post here and in discord about the current big list and ask for more things to add.
