/hydrus/ -Version 419

Name
Email
Subject	REC STOP
Comment *
File	Select/drop/paste files here
Password	(Randomized for file and post deletion; you may also set your own.)
Archive	Archive [500 char limit]
* = required field	[▶Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field)
Allowed file types:jpg, jpeg, gif, png, webp,webm, mp4, mov, swf, pdf Max filesize is16 MB. Max image dimensions are15000 x15000. You may upload5 per post.

YouTube embed. Click thumbnail to play.

Version 419 Anonymous ## Board Owner 11/25/20 (Wed) 22:59:27 182f87 No.14948

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v419/Hydrus.Network.419.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v419/Hydrus.Network.419.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v419/Hydrus.Network.419.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v419/Hydrus.Network.419.-.Linux.-.Executable.tar.gz

I had a great week finishing editable system predicates and making some long-running search situations faster and snappier.

editable predicates

As a reminder, if you shift+double-click on the search terms (predicates) in your current search, you can now edit them! I also added an right-click 'edit' menu entry, if you do not like the shift+double-click.

OR and system:rating predicates are now editable. Also, 'invertible' predicates like inbox/archive or tag/-tag will now be stacked in the editable panel as buttons you can click to flip.

You can now set the default for any editable system predicate type! Little 'star' buttons beside the panels now let you set the current value as the new default.

The editable system predicate panels also remember the five most recently entered predicates for each type, and these are stacked up top as quick buttons. The UI here gets a little tall and jank, but see how you like it!

You can now shift+double-click tags in a place other than your current search predicates (e.g. the 'selection' tags in center-left) to add the selected tags as a bundled OR tag. Shift+middle-click does the same for a new search page. There are also right-click menu entries for these.

All tag lists now support drag selection, so you can click a tag, drag down three rows, and all will be selected without having to click more.

faster and snappier searches

First off, any 'wildcard' search for autocomplete results or files that is A) 'complex', which means it has an asterisk in a non-rightmost character position, like 'bl*ah' or 'bl*ah*', and B) does not start with an asterisk, should now be much faster than before to find results. Tag/file wildcards that start with an asterisk are still basically hellmode in all situations.

Secondly, pretty much every component of tag-based search, from autocomplete lookup to tag, namespace, wildcard, or system:num_tag searches, now reacts faster to 'cancel' signals. This means that if you enter system:num_tags>4, and it is taking ages to finish, if you stop the search or update it with a new search predicate, the old search should stop its work more quickly. Also, when you type in autocomplete, old unfinished searches are often being quickly cancelled in favour of new ones as you type. This process is now much faster, particularly when wildcards are involved.

client api virtual tags

The Client API now provides 'display' tags, as calculated by the new virtual siblings and parents systems, if you want to show 'pretty' tags to your users. Check the help for /get_files/file_metadata for the new service_names_to_statuses_to_displayed_tags structure.

a note on macOS Big Sur

Hydrus currently does not run on Big Sur. It is a library compatibility issue, as my 10.12 macOS laptop is getting a little old. I am working with some users to figure out a better build solution, so please hang in there.

full list

- tag lists and editing predicates:

- you can now set the default value for any editable system predicate. a star button beside each panel lets you set or clear the custom default

- all editable system predicate panels now put 'recent' predicate buttons up top, for the five most entered predicates of the respective types. this is a little jank and grows pretty tall with multi-pred-type panels, but let me know what you think

- all tag lists now support drag-selection!

- taglists now have 'open a new OR page' menu entry when more than one tag is selected

- when taglists can change the current search, they now have an 'add an OR to current search' menu entry when more than one tag is selected

- OR Predicates are now editable! they launch their own little autocomplete input that is a little jank because you can technically make nested ORs, but it works!

- system:rating is now editable! it launches the whole stack every time. the stack alignment is messed up though :/

- invertible predicates (inbox/archive, tag/-tag, etc…) now flip on double-click only if you have one selected. if you have more than one selected, they appear as invertible buttons along with the rest of the edit UI

- the active search predicates taglist now has an 'edit search terms' menu entry, if you find shift+double-click a pain

- when you shift+double-click on more than one tag to add them to the current search, this is now added as an OR

- similarly if you shift+middle-click on more than one tag, the new page is now an OR

- when editing predicates, edited predicates now stay selected

- shift+clicking on an already selected tag no longer adds any new selections (i.e. shift+click filling-in). this should make it nicer to do shift+double-click on selections. furthermore, the 'last clicked' focus ghost (from which a shift+click selection cascade starts) on tag lists is now cleared on edits or removes, which should reduce some other crazy/annoying select behaviour here

- the list of active search predicates now correctly initialises sorted

- entering hex hashes into system:hash or :similar_to now has unified hash parsing, auto-removes 'md5:'-style prefixes, and presents detailed error information when a hash is too long or short

- .

- faster and snappier file and tag searching:

- searching for files by complicated wildcard (i.e. a search phrase that includes an asterisk in a non-rightmost character position) is now greatly optimised when the tag does not start with an asterisk (e.g. 'sm*l' is now much faster, '*all' is still hellmode), and now cancels (due to hitting the stop button or changing the query before results come in) much faster thanks to a new unified results fetching and cancel-checking routine

- rewrote my autocomplete tag search to use the new namespace and subtag lookup code from the virtual siblings and parents system, unifying lookup logic and benefitting from the same new complicated wildcard optimisation and fast-cancel tech

- autocomplete tag count aggregation (a later step, after the initial lookup) benefits from a little faster cancel tech

- all file queries based on tag, wildcard, namespace, tag count, and tag existence now use the new fast-cancel tech. if you put in a 'has >4 tags' query and it is taking ages, changing the query or just hitting the 'stop' button should now free up the db pretty fast

- related tags suggestions also gets the cancel tech and is now more timing precise for tags with either huge or tiny count

- .

- client api:

- the /get_files/file_metadata call now returns a service_names_to_statuses_to_displayed_tags structure, which reflects the sibling-collapsed and parent-added tags, as displayed to the user in UI. the help is updated to reflect this

- the client api version is now 15

- .

- the rest:

- fixed an issue where regenerating the tag definition search cache would not tidy up the 'I am busy' modal dialog once it was done, resulting in a soft lock

- fixed another upnp error handling bug, this time in the upnp daemon

- updated Qt to 5.15.2 on Windows and Linux builds. this should fix the unusual button clicking area problem for some custom styles

- .

- boring specific code changes:

- wrote widgets to edit invertible preds and OR preds

- pulled the messy rating code out of the rating system predicate ui code to their own widgets

- wrote some special predicate ui definitions and initialisation handling for OR preds and grouped 'multiple' preds (for ratings)

- refactored search and predicate ui code to a new 'search' module

- refactored collect and sort widgets away from search code

- misc layout improvements for system pred edit ui

next week

I was not able to get to the siblings/parents logical bugs, nor was I able to catch up with my messages like I wanted. These will be top priority next week, and then I am going to start planning the network update that I hope to finish the year on.

____________________________

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Post last edited at 11/25/20 (Wed) 22:49:04

Anonymous 11/26/20 (Thu) 09:14:03 c4f8c7 No.14949

File: 9c69ae73f6411fc⋯.jpg (191.23 KB,1196x1168,299:292,ERFwrRyWsAIl38D.jpg)

Fuck me, I accidentally deleted all my databases! Fortunately the actual files remain because I kept them separated from db+thumbnails. But my big db had 200k+ files in it! I hope the PTR has most of my tags (I was a pretty diligent tag uploader)!

What worries me most is the duplicate process though, because I had resolved all duplicates before but I will now have to do it all again. Fuck my life!

Well at least I have now set up a way to regularly backup my Hydrus databases. By the way, syncing with the PTR takes about a whole day on an SSD.

Uh, thanks for the new version, hydrus_dev!

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 11/27/20 (Fri) 11:37:52 000000 No.14950

Not sure if this is the correct place to mention it, but I've found that Hydrus creates really fragmented write-ahead log files since sqlite is demanding regular flushes to the drive every time even a little bit more data is processed. This flushing is normal behavior for sqlite, since it's intended to ensure writes go through. It's also fine on flash-based media because there's essentially no seek time, but for platter media it's absolute hell. A bandaid is to defrag the wal file while it's very large, or to grow a separate file, defrag it, shrink it, and then rename it to the wal file and copy over the content which gets the new wal file its own little section of the drive to play with. Sadly Hydrus resets this on a regular basis.

The root cause of the resetting fragmentation is that Hydrus' copy of sqlite doesn't have the "sqlitefcntlpersistwal" file option set, which means the wal file is deleted and recreated every time Hyrus starts. This means that the new wal file is shoved into whatever tiny 2kb hole the OS can find to start with, and since the wal is for file integrity, the file is flushed to disk without caching, resulting in several hundred 2kb-sized blocks scattered around the drive. It slowly fragments again until it's hitting hundreds of fragments, and the end result is incredibly high seek times on platter media.

Defragging a wal and the datbase and reserving space for growth allows for approximately 20k rows/s if it's the only thing running, whereas once the wal is deleted and recreated, it fragments and demands lots of seeks for sync writes (and for reading the wal to apply to the database) and drops to low single digit rows/s. I know the instructions say to use an SSD if syncing with external databases because HDD performance is so bad, but if it's reasonably easy to do, setting the wal to be persistent would significantly improve the performance for people who don't have an SSD.

Thanks for your time.

Reference:

https://www.sqlite.org/draft/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpersistwal

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous ## Board Owner 11/28/20 (Sat) 22:28:32 4b8074 No.14951

File: 778f92b26789c23⋯.jpg (58.19 KB,500x322,250:161,778f92b26789c23bf1b52dfa3c….jpg)

>>14949

Fuck, that sucks! There isn't a huge amount you can do, if you know the files are gone. I went through this myself, losing I think 75,000 files about fifteen years ago, and the good lesson I learned from it was to be crazy about backups. I have never had a problem or worry since.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous ## Board Owner 11/28/20 (Sat) 22:30:02 4b8074 No.14952

Btw, I messed up the drag-select on taglists, if you scroll it still selects the top items. I'll fix it!

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous ## Board Owner 11/28/20 (Sat) 22:30:51 4b8074 No.14953

>>14950

Thank you, I saw your post on Endchan, I'll paste my response there here:

Thank you, this is very useful information. I am not sure if I have the ability to set these opcodes from my python layer, but I will investigate this. If I can set a permanent WAL of a certain healthy size, it sounds like that would be a great improvement for HDD users. Can you say more how you achieved this to get 20k rows/s? Did you use a custom sqlite.dll, or were you able to set an environment variable to change my code to do this?

Another potential option here, although it is a bit crazy, and probably stupid if the user is on an older system with an HDD, may be to tell SQLite to use a memory journal, or memory temp file. I wonder if there is a similar problem as the fraggy WAL with fraggy temp files (SQLite creates files to store temp table data when it does larger transactions).

I have discovered that some PTR syncing is creating absolutely gigantic temporary files, sometimes >20GB. I want to investigate shrinking this significantly, although I worry that making more smaller commits will just result in more HDD work overall.

Also ultimately, I will be adding filters to the PTR in the coming months so users can just sync with a subset of PTR data (e.g. just creator/series/character tags), which I hope to produce several speed and storage improvements for users on slower drives.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 11/29/20 (Sun) 09:38:45 1ac8f6 No.14956

Did the twitter (nitter) media lookup importer start failing for anyone else? I think it's a problem with the source site changing settings or something, since it happened while using the same version.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Release Tomorrow (later today)! Anonymous ## Board Owner 12/02/20 (Wed) 10:10:29 14c632 No.14961

I had a great week. I fixed bugs in drag-select, an undisplayable tag, some autocomplete result weirdness, and logical problems that were causing some grandparents and siblings not to appear in the new virtual system; I fleshed out some maintenance modes and added better autocomplete lookups in manage tags; and I sped up several autocomplete, sibling, parent, and general tag routines.

The release should be as normal tomorrow, maybe a little late.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous ## Board Owner 12/02/20 (Wed) 22:54:38 a733c7 No.14963

>>14956

Sorry, I don't know, I'll check it out.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 12/20/20 (Sun) 03:47:24 1b5038 No.15019

>hydrusdev fell into the rabbit hole

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.