windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v408/Hydrus.Network.408.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v408/Hydrus.Network.408.-.Windows.-.Installer.exe
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v408/Hydrus.Network.408.-.macOS.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v408/Hydrus.Network.408.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v408.tar.gz
I had a great couple of weeks moving tag siblings forward. The update this week will take some time as a new cache is generated. On an SSD with the PTR synced, about 10 to 20 minutes, depending on how many files you have. Some of the new tag sibling application rules may have 5-10 minute delays on edit as well. If you have a large/PTR client and are on an HDD, or you would rather just wait for the kinks to be ironed out, you might like to wait a week or two for me to further optimise this new code.
siblings
tl;dr: Siblings are faster and better now, you don't have to do anything. Some parents will not appear with new downloads - don't worry about it, they will all fill back in nicely soon.
I have finished the first version of the database 'display tags' cache. This is a long-planned system that keeps track of how tags are 'collapsed' by siblings on an ongoing basis. Until now, siblings were always calculated on the fly at the UI level whenever media was loaded, which lead to slow and inaccurate sibling-based search results and a variety of logical problems. Search results generally load much faster now, particularly on subsequent loads.
On the front end, tags appear exactly they did before, but when you do tag-based searches or tag autocomplete, the files and counts that come back should be more accurate. For instance, if you have a file tagged 'metroid' that appears as 'series:metroid', and you search 'series:anything', that file now shows. Siblings logic is also tighter, and changes apply faster.
And now, if you go to services->manage where tag siblings apply, you can now decide which services' siblings apply where. If you do not like the PTR's siblings, or you have a couple of different preferences you want to add on top, you can say things like 'apply siblings from "my tags" then "PTR"', or you can turn them off entirely. This powerful service-based system replaces the old 'apply all siblings to all services?' checkbox that used to be in the options.
This is some complicated code affecting core systems. I have tested it extensively, but IRL situations, particularly with large sessions or storage, will be more complicated and useful than what I have here. Please let me know where this new system is laggy or miscounts anything. And of course, if anything outright breaks, let me know!
The help on siblings is going to be increasingly outdated as this work continues. If you don't know what siblings and parents are, please wait until I am done with this multi-month job, and I'll have some nice new help written.
sibling and parent virtualisation
On the back end, the important change is that siblings are now virtual, which means that when the tag 'samus_aran' comes in, while it may appear as 'character:samus aran' in all normal views, the old systems that would substitute the 'good' tag in for the bad at the storage level (i.e. what you see in manage tags) no longer occur. This means that all siblings going forward are completely undoable–if you want to try something different out, you can just change the current siblings or application rules and it will all recalculate from the untouched (albeit uglier) source.
This 'virtual' change is needed now. Siblings can come from anywhere (particularly with the new service application rules), and trying to manage a mix of 'soft' and 'hard' siblings has proved too difficult to manage in one system. Parents are not yet virtual, but they will be soon - I hope in four or five weeks to do the same for them as I have done here: virtualisation, improving search and application logic, permitting complete undo, and allowing service-based parent application.
We are therefore in an interim period. Parents are still hard-added, so I have disabled sibling-based parent application (so if 'character:samus aran' has 'series:metroid' as a parent, a file that only gets 'samus_aran' will not get that parent). This unfortunately means that workflows for the next month may be a little awkward, as parents will not fill in for certain downloads. Please hang in there - I am still working on this, and when parents become virtual, all the missing parents, old and new, will be filled in (and finally undoable!). I want to improve some of the autocomplete sibling-add workflow, something like grouping siblings together, a bit like how parents hang off a tag that will add them. Let me know what you think.
I know there are uses for 'hard' sibling and parent rules that will actually change and add to storage tags. Trying to support both in the same system has proved unworkable, so I will bring hard-replace and -implication in as separate systems in future.
full list
- tag siblings cache:
- tl;dr: siblings are faster and better now, you don't have to do anything. some parents will not appear with new downloads - don't worry about it, they will all fill back in nicely soon
- wrote the first version of a 'tag display' cache, which stores not your tags as they are, but how they appear after display rules such as siblings, parents, and filtering are applied, meaning this data need not be calculated every time on thumbnail load. this week marks the first concrete step forward in an improvement of siblings and parents storage, and begins with just siblings. all siblings and front-end tags work should be generally faster and more accurate
- part one is for tag domains cross-referenced with file domains. it maintains virtual sibling-collapsed mappings and autocomplete counts through mappings added, deleted, pended or pend-rescinded, files added/deleted, and siblings added/removed
- part two is for tag domains on the 'all known files' domain (i.e. no file domain). it maintains virtual sibling-collapsed autocomplete counts through mappings added, deleted, pended or pend-rescinded, and siblings added/removed
- both parts also support full table drop/regen (under the new database->regenerate->tag display mappings cache) for when my logic inevitably miscounts something. the existing regen 'tag mappings cache'/'tag siblings lookup' commands also regen the display mappings cache, since it relies on them
- when tag siblings on a repository are petitioned to be deleted, they are now instantly discounted from tag sibling application (previously, they had to be uploaded and committed to count, now both pending and petitioned offer a quick preview of outcome)
- the display cache supports the tag sibling service application rules under the 'services menu', and regen when that changes, so you can now turn siblings on and off, and apply them across services. as a result, the old 'apply all siblings to all services' option is now gone! as parents will undergo a similar change soon, and the siblings changes this week may lead to some undesired parents in the interim, the 'apply all parents to all services' option is also gone
- tag autocomplete counts in the form (x-y) due to siblings are eliminated. it will still do it when combining the same merged tag across different services, or when an unnamespaced tag includes how many potential namespaced will also be found
- the following search types now obey tag sibling application rules accurately: number of tags search, namespace:anything search, wildcard search, tag search (on a per-tag-domain basis, previously it was globally hacked to all siblings), tag-as-number search. for instance, if you search series:anything, a file that has 'metroid' tag-siblinged to 'series:metroid' will now correctly appear.
- the above search types are now exact to how the tag displays. if you have for files that are tagged 'samus' on either tag service A or B, and service B has a sibling for that to 'character:samus aran', searching for 'samus' gets the results in A, 'character:samus aran' gets the results in B. previously it was an expensive logical mish-mash of 'sure, try and get everything behind the scenes'. now it searches what you see
- searching for files in the advanced 'all known files' domain currently has no sibling support for the above search types. autocomplete counts should be good, and the results that come up should have the correct tag display, but the actual results are calculated based on storage tags. getting this to work without doubling the size of the db is tricky, so it will have to be ongoing work
- all tag siblings are now completely virtual. this means that when a tag comes in via a downloader or other means, it will not be automatically coerced to its ideal sibling in the actual db storage tables (the true tags you see in manage tags dialog), but remain as it is. there is no change in sibling appearance in normal operationit still _displays_ and searches as its ideal sibling. the same will happen to parents in the coming months, and in the interim period, parents no longer apply across siblings. as siblings can come and go from anywhere, they are now divorced from actual stored tag mappings. in a similar way, the manage tags dialog no longer supports the 'hard-replace siblings and parents' command, nor the 'auto-replace with sibling' command. this may be jarring to workflows and preferences, so please bear with me and let me know what feels particularly bad. and please don't worry too much about parents not always being added in the meantimeI hope to do the same transition for them in four of five weeks, and all gaps will be filled in. also, in the coming weeks, I expect to improve manual tagging workflow by indent-grouping edit-view siblings together (ditching the old 'will display as' text) for easier review and selection, a bit like parents. actual 'hard' siblings and parents that do always get irreversibly renamed/added in storage will come in the future as a separate system
- the 'add_siblings_and_parents' client api parameter no longer adds siblings, and soon will be retired completely
- I had wanted to completely eliminate the old UI level siblings manager this week, but there are still some systems, mostly tag autocomplete work, that need it and are tricky to swap. I stripped it down, at least, and reduced its update delay to 2 seconds. therefore, the 'loading tag siblings' step of boot still occurs, albiet a _little_ faster. I hope to have it gone soon
- this is some complicated code affecting core systems. almost everything 'siblings' is now different. there are likely to be laggy parts, awkward new workflow, and possibly some update or miscount bugs as I iron it out. the good news, now they are all virtual, is that problems are undoable. please report any issues, and I will work on it as I keep pushing on this and on parents
- please expect your client.caches.db file to expand in size about 10-30% or so this week. the update itself will take a few minutes as the improved tag lookups and new caches are regenerated from empty
- .
- boring mostly db optimisation list:
- after some thought, moved those new options for tag sibling application down to the db. previously, they were stored in an UI object for convenience, but since everything is going down to the db, it is worth doing it properly down there. thus they reset this week to the default
- I also removed that complicated 'all known tags' page in the tag sibling application options–it wasn't doing enough to justify itself
- tag siblings lookup cache now obeys the tag sibling application rules and regenerates the appropriate cache when the options change
- tightened up the db tag siblings lookup cache and wrote more tools for it. it had a couple of blind spots for getting all siblings in a chain. also optimised the lookup for en-masse tag operations
- tightened up my tag sibling structure builder object, which was not eliminating loops but collapsing them to (generally harmless, but not desireable) (A,A) pairs
- extended mappings and siblings lookup caches to perform various sorts of tag sibling collapse filtering, to determine files that do or do not have another tag mapping on a tag sibling chain
- optimised the existing mappings cache in several ways
- optimised cross-domain file cache mappings filtering, and cleaned the code
- optimised autocomplete count fetching from the mappings cache, particularly for large result sets
- optimised how the combined autocomplete count generates from nothing
- optimised how tags are loaded for search results (thumbnails)
- optimesed basic tag search
- greatly optimised how the mappings caches request cross-domain file cache filtering
- broke up the rescind_pending/add mappings job into simpler separate parts, which was needed for accurate display cache counting. this may end up fixing the other weird pending miscount bug we had
- the cached 'display' tags are now loaded with regular media results, not generated on the fly on first request (unless in the advanced 'all known files' domain, where it is done quickly on first load at the db level)
- converted the db over to using its local sibling lookup cache for all sibling jobs
- all data-level content updates to media result objects now occur in the database loop, reducing lag and eliminating a single UI event loop gap when the objects the UI relies on were desynchronised
- optimised how the tag and hash id-to-definition cache maintains itself
- cleaned up cache code generally
- wrote a ton of unit tests to cover construction, tag, and tag sibling operations on the siblings and display caches
- wrote a second optimised method for regenerating 'all known files' display cache autocomplete counts from nothing, which, when multiple siblings have wildly different counts (e.g. 50, 100, 200000), instead of counting them all, counts the smaller tags sans the largest, and adds this to the already pre-computed largest count
- the old ui level siblings manager has been pared down to some final tools that will be trickier to replace
- .
- the rest:
- fixed the stupid manage tag siblings dialog input/ok bug I introduced last week
- fixed the pair preview label in manage tag siblings dialog when it asks to enter a reason for a remove petition
- I believe I fixed the annoying recent bug where the top-right hover window would sometimes not position itself correctly on a window size/move until the top hover was shown once
- fixed a bug where the 'do you want to do shutdown work?' dialog was not abandoning shutdown if cancelled (rather than yes/no)
- updated the 'has free space to do db transaction?' checker, which needs to test device partitions, to do two sweeps–first only fast local devices, then potentially mega laggy network discovery if the mount point is not found (hydev was wondering why it was suddenly taking nine seconds to close his test client!)
- fixed another issue with double-clicking some addremove/queue listboxes when no edit button is set–now in this case they all delete on a double-click
- fixed a little bad error handling on pending content upload. an error with petitioning certain IPFS uploads is not yet fixed
next week
I was unable to completely replace the UI-side siblings manager. It is a bit smaller and faster, but you will still get the slow 'loading tag siblings' on boot. There are about four or five things it still does that I still need to move down to the new cache, so I will keep hammering away. I'll also improve the speed of these new caches' regen. Other than that, I have two weeks of other jobs to catch up on, so just some misc work.
I just realised I forgot to fix the annoying manage tags/media viewer focus stealing! I'll fix it for 409!
