/hydrus/ -Version 414

Name
Email
Subject	REC STOP
Comment *
File	Select/drop/paste files here
Password	(Randomized for file and post deletion; you may also set your own.)
Archive	Archive [500 char limit]
* = required field	[▶Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field)
Allowed file types:jpg, jpeg, gif, png, webp,webm, mp4, mov, swf, pdf Max filesize is16 MB. Max image dimensions are15000 x15000. You may upload5 per post.

YouTube embed. Click thumbnail to play.

Version 414 Anonymous ## Board Owner 10/21/20 (Wed) 22:46:06 06c97f No.14838

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v414/Hydrus.Network.414.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v414/Hydrus.Network.414.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v414/Hydrus.Network.414.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v414/Hydrus.Network.414.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v414.tar.gz

This is the first release of a complicated new system. If you are a cautious user, feel free to wait a week for me to iron out any surprise bugs.

I had a good four weeks converting tag parents to the new 'virtual' database cache.

The release does not need to do a big update this week. It will ask you if you have an SSD or an HDD (EDIT: with a special caveat for PTR users, see below), but that is it. All the mandatory heavy work that came in with 407 now happens in the background.

parents

tl;dr: You don't have to do anything. If you haven't heard of a tag parent before, no worries. The database should work better now.

I did the same to parents as I recently did with siblings. They are now virtual, which means when the database needs to 'fill in' a missing tag because of a parent relationship, that new tag now does not actually exist in storage, only in a special computed database cache. If parents change, the implications in this cache can be recalculated with no permenant changes made. Parents are now undoable.

Parents are also comprehensive. Previously, due to different logical holes, not every parent was filled in correctly. There were several retroactive holes. Parents now always apply to all their children, and they adjust to siblings and sibling changes.

Also, you can now apply parents across services. This matters less than siblings, but if you have particular parent needs, you can now set your 'my tags' parents to work on the PTR, or remove all parents entirely, under the new tags->manage where tag siblings and parents apply dialog.

This took a bunch of work. I am happy with all of it except how parents now display in manage tags. It was complicated to insert 'ghost' tags to represent parent implications and I just ran out of time, so for now, tags with parents just get a '(2 parents)' suffix after their tag label. You can now right-click on tags in most places to see all their sibling and parent data.

smoother sibling/parent calculation

LATE EDIT: It seems the PTR has one very large parent structure that causes a one-time lag of a few minutes even for SSD users. This is happening because we are catching up on all the parents at once. You will get a popup when you update asking if you are on SSD or HDD. If you select SSD, be aware the client is likely to lock up for a few minutes soon after boot. Just let it do its work, or select 'HDD' on that update dialog and it will happen when you are not looking.

I was not happy with the large lag introduced when I added siblings. Ten minutes to see a new PTR sibling application is too long, and HDD users were almost completely nuked. Rolling parents into the same system would have made it worse. So, another aim of this big work was to reduce that calculation time and smooth it out in the background.

This is now done. Applying changes to siblings or parents should be very quick in all cases, but those changes may not instantly appear in your actual tags. The client now keeps track of what it currently has and what it should ideally have, and migrates silently in the background, in pieces. Most of the time, changes should appear within a second or two, but if a whole bunch just changed, it will take longer. You can check what is going on under tags->sibling/parent sync.

I also improved the recalculation code. The sibling and parent systems are now unified and expressed in a more granular 'tag implication' algebra, and rather than clearing bad numbers and regenerating them from source, the client now defines the difference between the current implication structure and the ideal, and only applies those differences. Reducing all structure changes to 'add implication' and 'remove implication' allowed me to implement this migration tech and optimise the queries significantly.

This is an entirely new system. If it causes lag for certain situations (e.g. if you have a 100k+ file session open), let me know. You can always turn it off under the sibling/parent sync menu. If you select 'hdd' during update, the migration will only happen in idle time, but I'd be interested to know how it performs in active time as well, if you are feeling adventurous.

If you are running a pre-407 client, you should be able to update straight to this week and skip almost all that work. Should™.

full list

- tl;dr: you don't have to do anything. if you haven't heard of a tag parent before, no worries. the database should work better now

- .

- top level:

- parents are now completely virtual! this means that when you add a tag parent, the tags that 'fill in' to make it show do not really exist in storage, only in a computed cache. if you decide to undo the parent, the implications are recalculated and the virtual tags disappear, with no permanent changes made. also, petitioning a parent will 'preview' the delete, just as siblings now does

- siblings and parents are now unified, and the logic is improved. all parents apply to all siblings, so no more worries about retro-active filling-in. the siblings and parents code is now basically 'nice'. this was a lot of quite complicated work, and it solves a number of lingering issues from the original prototypes I made several years ago. I will still do some smaller work and little fixes I am sure in the near future, but the 'big' siblings and parents work is done!

- like with the recent siblings change, the client no longer needs to do the 'loading parent tags' step when booting–everything is now handled at the db level

- like with the recent siblings change, you can now edit which services apply their parents to which service, now under _services->manage where siblings and parents apply_

- in the _manage tags_ dialog (and some other places), tags with parent implications now show a '(x parents)' after their label, much like the 'will display as' sibling suffix. I do not like this, but I ran out of time. I hope to add a more advanced actual listing of virtual tags with a nice 'ghostly' colour or similar in future

- right-clicking on a tag in a specific tag domain now shows a 'siblings and parents' submenu with detailed info on all known siblings and parents in that domain

- 'tag' menu entries are moved from the top 'services' menu to a new 'tags' menu. 'pending', when available, is also moved right

- the process of changing siblings or parents, or which services apply where, is no longer a CPU-laggy process! actual changes, however, may not appear immediately. a maintenance task now tracks what is currently applied and what is 'ideal', and slowly migrates to the ideal in the background in little chunks. in most situations, the changes are very quick, but if you are behind due to big recent changes, they may be delayed. you can manage when this maintenance runs and see the current status under _tags->siblings/parents sync_. this is an entirely new thing, so feedback on IRL work would be appreciated–there may be some kinds of siblings or parents that cause a whole bunch of annoying lag

- the PTR has a lot of non-virtual parents that were hard-added in older versions over the years. most are fine, but some are like the 'shadow'->'shadow the hedgehog' debacle. now the source of the problem is fundamentally solved, this problem will reduce over time. with luck, before the end of the year, no more will be added at all, and thanks to the janitors, the worst offenders should be chipped away

- during all this work, a bug with tag siblings and parents repository processing has been revealed (some users do not 'get' all siblings/parents for some reason). now the system is nice and undoable, this will be more easily addressed in coming weeks, with automatic retroactive fixes rolling out to all clients

- .

- boring details:

- like with siblings, wrote a parents structure object that constructs the parents tree without loops more simply and reliably. it populates a new parents quick-lookup table in the database, for which a full suite of lookup and maintenance methods are written

- parents and siblings virtual tag presentation is now unified into a single 'display' (i.e. vs 'storage') system with a more granular tag implication algebra (essentially 0-n rows of 'if A is in storage, show B in display' for every tag) that can calculate new and updated display tags and counts without having to do the expensive 'clear-and-regen' that 408-413 used

- wrote functions to quickly add or remove a display implication to the 'all known files' or specific file service tag display cache

- migrated all the combined and specific tag display cache update code (add/remove files, add/remove mappings, add/remove sibling/parent, add/remove sibling/parent application, and misc regen maintenance calls) to use the implication system instead of the sibling 'ideal' system (basically moving from 1->1 to 1->n)

- completely rewrote the complicated 'all known files' cache 'with tags' and 'with and without tags' lookup routines to use much less overhead in general and to use a single, albeit complicated, count-based query that carefully chooses whether to select the 'with tags' and 'without tags' portions using tags or files where available as the primary selector based on existing autocomplete count data

- replaced all usage of the old ui-side 'tag parents manager' object. as parents pop in virtually and do not need to be bundled intentionally to various content updates, this was mostly just clearing now-surplus code, but for instance in 'write' autocomplete searches, the parents that appear below search results are now generated at the db level on first search, rather than looked-up live in UI time

- the parents and siblings lookup tables are now split into two views: what the display cache currently holds, and what it ideally should hold. when adding new sibling or parent data, only the fast ideal table is changed

- a new complicated maintenance function now takes actual and ideal data for a particular unsynced tag, hashes out the implication changes needed to effect a migration, and performs it

- a new maintenance manager and accompanying db code now track and manage calls to migrate actual to ideal display presentation, and to update UI afterwards

- as tag display changes are now more frequent, I have made the routine that refreshes tag UI after sibling/parent changes more efficient. tag display now only refreshes for files that have the affected tags in a particular change

- wrote the UI panel and dialog to show and hurry up current sync status, and all the background hooks for that

- added 'tag parents lookup' entry to the database 'regenerate' menu. this routine and the 'siblings' variant are now very quick thanks to the new actual/ideal maintenance system

- updated my sibling unit tests to deal with the new actual->ideal syncing

- improved the speed of mappings cache updates when deleting files

- deleted all the old combined/specific 'regen chain' code and the sibling-based 'get sole/any tagged files' search code

- optimised and generally cleaned a bunch of the new cache code, particularly cutting out overhead for unusual/small situations

- fixed a counting bug with 'all known files' tag counts when rescinding pending tags

- fixed a bug in the siblings display code where deleting or pend-rescinding all of the multiple tags that have the same ideal sibling in the same transaction (e.g. if both A and B sibling to C, and a file has both A and B, and you remove them in one manage tags dialog apply) would not remove the current/pending ideal (issue #571)

- the 'add_siblings_and_parents' parameter on /add_tags/add_tags client api command is now obsolete! the help is updated to reflect this

- cleaned up just a bunch of db/ui/tag code mate

- .

- the rest:

- fixed an issue where long-running 'similar files' search was not cleaning its memory use properly as the job was going on, resulting in out-of-mem errors on very large clients (issue #669)

- thanks to user submission, rolling in a fix for the default pixiv tag search downloader

- cloudscraper updated to 1.2.48

- removed surplus executables from linux and macOS builds (win32 upnpc exe was causing anti-virus false positive on mac lmao)

next week

Back to small work. There are a thousand things to catch up on, so I'll mostly grind away at that. This work was big and difficult, and some IRL stuff hit me for the first couple of weeks, so the year is running out. My overall plan is still to have some network upgrades done and then a 'what to work on next' poll up for Christmas, but I will take a few weeks of simple work first.

____________________________

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 10/22/20 (Thu) 09:31:58 a1c9b2 No.14839

Thanks for your hard work hydrus_dev!

Will the button to turn parents/siblings into real tags in the manage tags window return?

And to everyone else: Remember to go to "tags > manage where tags siblings and parents apply" and add the PTR if you want that for your local files.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 10/24/20 (Sat) 20:39:16 e2d1e4 No.14849

It doesn't appear that parents for the PTR can be accessed yet. I thought this was the update that would bring that back. Is there something else that needs to be done before users can change them again? I didn't see a mention in the changelog.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous ## Board Owner 10/25/20 (Sun) 00:29:39 389e39 No.14851

>>14839

I do not plan to bring it back with these systems, but I will introduce something similar in future. It will probably be system wide, rather than manually fired off for each file in turn.

The ideal that I will now keep to is that siblings and parents are always virtual. They now never hard-add tags and will always be completely undoable as a result. A system that had both virtual and non-virtual components was too difficult to maintain and meant I could not allow cross-service application. The more a user can customise what their siblings and parents do, the less I can allow for hard-replacing. An increasing problem in the PTR was due to one or two users with unusual parent rules spamming up tag changes for thousands of files to the shared PTR with that parents/siblings button, and trying to undo what did get through became unmanageable.

So, I will instead at some point write a different system-wide tag 'rename' ability (maybe called 'alias' or similar, as how some boorus do), that will hard-replace in past and future and be be more focused on tag domains you control. You'll be able to hard-replace tags on 'my tags' no problem, and janitors/admins on repositories like the PTR will be able to efficiently fix typos and so on. A similar 'hard' parents system may have a place at this point, or we may feel we don't need it.

>>14849

I am no longer involved in the decision making, but I believe the current plan is to bring it back when I do a network update (hopefully before the end of the year). I will be adding long-delayed janitorial tools, and if I can find the time some clientside filtering tools so you can screen out some things you don't want, which will require incrementing the network version. At that point, old clients will have to update to the new version to keep talking to the PTR, so everyone uploading will be on the new undoable caches, and then they will either turn it on for all users or set up a different account key system for users that want to pend parents or siblings up. We'll see how my improvements to janitors' petition approval workflow goes, I guess!

For now, I recommend you create a new local tag service under services->manage services called 'my parents' or something, and then set whatever you like there, and under tags->manage where tag siblings and parents apply, add that to the PTR. You'll then get whatever parents you personally like, and when there is an avenue to upload up to the PTR, you can either spam them all up with the tags->tag migration dialog, or pick and choose what you think is appropriate.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 10/26/20 (Mon) 08:33:30 788153 No.14854

Thanks for your hard work! I just updated from ver.395 (don't worry, i jumped from 395 to 400 to 405 to 414.) Is there a way to remove tags implied by a parent to stop showing up in suggested tags in the tag manager? Hydrus used to remove "male" from suggested tags if I had added something like "1 male" which used to add "male" to the file.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 10/27/20 (Tue) 21:47:13 ae3dea No.14859

File: 06202c065f954af⋯.png (73.48 KB,1121x326,1121:326,ClipboardImage.png)

I updated hydrus, selected SSD (because I have one) and now it freezes all the time.

What do now?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 10/28/20 (Wed) 05:50:14 b6d90b No.14862

File: 85f5ac74c7432e2⋯.jpg (156.25 KB,1057x860,1057:860,Untitled.jpg)

>>14838

Got a small issue with favorite tags. I added a few recently and for some reason the series tag became "grouped" underneath a character tag for that series. Despite them being "grouped," clicking on it in the Manage Tags window only adds the character. I tried removing the series tags from favorites, but they still show up as grouped anyway. Oddly, it doesn't happen for the other character/series tag I have on there.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous ## Board Owner 10/28/20 (Wed) 07:26:25 f444a8 No.14863

>>14854

Ah, damn, I see. I had to compromise a couple things in manage tags to get this working in time. I will see if I can fix this next week.

>>14859

I am sorry for the trouble. 415 tomorrow should have much less laggy background maintenance. Please give it a go, check your sync progress under tags->sibling/parent sync->review tag sibling/parent maintenance, and let me know how it goes.

Most SSD users with a typical PTR-syncing database should have seen the lag spikes disappear after no more than, say, 40 minutes of use. If you still get it in 415, and the sync in that review window does not seem to be progressing at any reasonable speed, please try disabling the maintenance in active and idle time in that same menu, and let me know.

>>14862

Damn, thank you for this report! This should not be happening, I will check out what is going on.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Release Tomorrow! Anonymous ## Board Owner 10/28/20 (Wed) 07:26:50 f444a8 No.14864

I had a mixed week. While 414's parents cache seems to have gone well, making the new maintenance code less laggy proved difficult, and it monopolised my time this week. I did a small amount of other work, but not much, so 415 will mostly be a 'polish' release that makes 414 nicer for all users.

The release should be as normal tomorrow.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.