/hydrus/ - Hydrus Network

>>13527

Does it work for videos? I'm using 378 and the url downloader just skips videos since it grabs them by the filename rather than hash.

>>13523

Thank you. I have just fixed this now, let me know how it works for you. There are some other fixes in this area in today's release as well–please check out the changelog and let me know what else I need to catch up on.

>>13530

>>13527

No worries. I am rolling out 8kun thread url recognition and parsing today, so you should be able to paste 8kun threads into a watcher page and it'll all work, videos and everything, in 380.

I had a great week. I focused on a variety of smaller jobs: better downloader UI, nicer search workflow, m4a audio file support, better select/remove menus, and more file search speed optimisations.

I caught the coughing cold that is going around, so my schedule is a bit out. The release will like be quite late tomorrow.

>>13469

>>13470

Thank you for the follow-ups. I believe this is fixed for 378. Thankfully, your sessions are not deleted from the db, they were just not showing in the menu.

A recent change in how the 'pages' menu is generated was relying on the 'backups' list for the main 'session names' list, but in the case where a session only had one recorded save, no backups, it was not included in the backups list. This would particularly hit older sessions that were saved before the backup system was established. I have fixed the way these lists are generated to check both areas for session names.

>>13469

For your time-skip issue, I think this is the fix:

Same deal with the sqlite3 executable console as in >>13461 above. We will rewind all your subs a bit just to make sure a new session gets saved on top rather than be discarded as the new 'oldest' one. First, we need to pick a number of seconds to rewind. If you reckon a couple of months difference, let's try rewinding them six months, or 86400 * 30 * 6 = 15500000:

.open client.db
UPDATE json_dumps_named SET timestamp = timestamp - 15500000 WHERE dump_type = 13;
.exit

Now any new session (with a timestamp of 'now', 1576526960 or so) should save on top of the backup stack. I will fix this issue this week and make sure that even if current backups appear to have been saved in the future, any new save will fast-forward. If you need to do some more clock-changing in future, I'll still advise you do it while the client is off, as all downloader/subscription/thread watcher timings are based on current time and may go haywire.

>>13468

Hey, I am sorry you have run into this. I believe you got hit by the tricky 335 update, which was the jump from python 2 to python 3. Your db is safe.

Hydrus needs a 'clean' install to clear out old python 2 dlls that conflict with py3 stuff. Forgive the tumblr URL, but here is the release post from then with full instructions:

https://hydrus.tumblr.com/post/181885931124/version-335

The basic thing is 'don't delete the db directory', since that is where all your stuff is stored. Delete everything else, reinstall/reextract, and you should be good. Please let me know if you still have any trouble. A big jump of 40-odd versions may have other issues, so you might like to try 345 or so first, boot it once, then 355, 365, 377.

Please also bear in mind that a few weeks ago a complete overhaul of the UI engine happened, wx to Qt. It is as significant a change as the py2->py3 step was. Hydrus looks a little subtly different now, but performance is significantly better. There are no special install instructions for this wx->Qt stuff.

>>13472

Thank you so much, that fixed it! I don't expect to do any clock-changing for the foreseeable future, but I'll follow your advice if that changes.

>>13473

Thank you very much, Hydrus dev!

I followed your instructions, cleaning out the old program and reinstalling thru 335 to 377 every 10 or so updates, and now the latest version is running well so far. I encountered 2 minor hurdles in the form of a "missing cache to rebuild" and the migration to the new PTR, both done smoothly.

If I encounter any unusual, less-than-optimal events I'll be back with details.

Thank you again for your help!

ok nevermind it turns out that it does not work with 8kun

>>13457

Yeah, unfortunately the DDoS protection is a block for now. You may have luck using Hydrus Companion to copy your browser's cookies across to hydrus, but I don't know if this will work for 8kun specifically.

https://gitgud.io/prkc/hydrus-companion

Once the whole situation calms down, I will roll out what I can to get hydrus working again with 8kun and the popular bunkers.

- .

- new file maintenance jobs:

- added a new 'check for missing files' file maintenance job, where if the file is missing and has urls, those urls will be queued up in a new url downloader for redownload. the file record is not removed, preserving archive/inbox and import time

- added a new 'check for invalid files' file maintenance job that does the same deal as above with an additional expensive byte-for-byte content check if the file is not missing

- added a new 'check for invalid files' file maintenance job that only cares about invalidity–if the file is present and invalid, it is moved out but the file record is not removed

- .

- the rest:

- network jobs that receive low-bandwidth error codes from the server now use a separate wait routine (previously, they piggybacked on the connection fail retry system). they have a separate cog-menu action to override these waits

- the time delay multiple for connection errors and serverside bandwidth problems are now editable under options->connection. old default was 10 seconds base, now 15 and 60 seconds respectively

- updated the danbooru login script

- improved the precision of the thumbnail size estimate in database migration

- the alphabetisation of a url class's GET paramaters on normalise is now optional. it is a new checkbox on the url class edit panel

- when a default object fails to load from a png path, a simple error is now written to the log

- misc cleanup

next week

I got hit by some IRL stuff right at the end of this week, including some Thanksgiving surprises, so I couldn't fit in the new downloader fixes I wanted to, but I have some fixes for pixiv tag search and twitter video download waiting to be added. I also couldn't get to Qt theming again, nor macOS tab DnD! So, I'll focus on those first thing so they definitely get done.

Otherwise, next week is scheduled to be a 'medium size'Post too long. Click here to view the full text.

test BO trip

Curiously this thread only shows up in the catalog. At least for me.

>>13441

Yeah, I have had to force-refresh a couple of threads on the site to get new posts, and a couple of the BO things didn't work immediately after migration. And things are obviously sluggish. Seems to be getting better though!

My only real peeve is that file downloads now don't have the content-length header sent or something, so my web browser doesn't have a nice progress bar as they download. And the anti-DDoS blocks hydrus html for now unless, I assume, you send the vanwatech cookies with Hydrus Companion or similar. JSON API seems to work, so maybe that is better.

I keep meaning to do this job but keep not getting around to it, but I'll roll out new thread watcher support for the big bunkers, and 8kun, in a release soon.

>>13399

Not yet. I had a tentative job to explore post-duplicate metadata merge/permanent synchronisation in this iteration, but adding that system was bigger than I had time to do well. It'll be slightly complicated, and I don't want to mess it up with a quick hack.

So just to be clear, for now the metadata merge only works on the pairs you see in the dupe filter. But since both representatives from the dupe filter are the Kings of their respective groups, and therefore 'good' data travels up from groups to kings, you can generally assume that Kings will have fairly 'complete' metadata based on previous decisions for now.

Since duplicate records are permanent, once I do add something like this, we'll be able to retroactively apply whatever merge settings you prefer, so non-Kings of groups will all sync too in an ongoing basis.

I suspect when I do this I'll have different or better default duplicate merge options here. Some tags aren't appropriate to merge backwards, like 'high resolution' or 'webm', and it'd be nice to have examples of those as default, or easily selectable, rather than making users awkwardly figure it out on their own.

>>13408

I am not certain. I think I remember reading about one user getting it to run on FreeBSD some time ago, but I can't be sure. The program is devved on and works best for Windows, and I build the Linux release on Ubuntu, so some more esoteric flavours of Linux and their Window Managers can have trouble with my build. Your best bet is probably running from source, for which I have some help here:

https://hydrusnetwork.github.io/hydrus/help/running_from_source.html

There's an Arch package maintained by a user, but I don't know enough about FreeBSD to say what is compatible:

https://aur.archlinux.org/packages/hydrus/

I assume that wouldn't work for you, right?

If you have a Windows machine, I recommend you try hydrus on that just to see if you like it first, rather than going through the rigmarole of getting it going on your FreeBSD machine and finding it isn't what you like. If you do give it a go, let me know how you get on!

>>13417

Not yet, but as it happens I am on the verge of doing some heavy work on local tags and I hope to add it then. I'll do a couple of weeks on some audio metadata stuff and then start on all that.

>>13425

I had added the lookup script for Deep Danbooru but couldn't get it to work; it would get stuck in the uploading stage and quietly fail out. But the errors didn't pop up till maybe 20 minutes after my various attempts so I wasn't sure if they were related or not.

>>13426

Like I said I think, the reason this one works for the jpeg being larger is due to the text, at least that what I think. to get clean text on a png, it's more or less par for the coarse and doesn't eat much space, but for a jpeg, to keep it without any kind of flaw, at least visually, it would need a stupid amount of resources. edge cases like this, or potentially others like this where a jpeg is made, and my best guess is one of the joi threads on 4chan had someone blanket convert with the highest quality settings on them, hit this image and it tossed out a larger file then put in. i'm guessing that 2/3 or above of the image being something jpeg does not handle well had a part in why this exists the way it does.

as for an auto decision, I don't see how the difference between 'the pixels are the same choose jpeg' or 'the pixels are the same choose lowest file size' either way, 99%+ of the time it will go jpeg anyway, this would just introduce a safety net for the sub 1% of the time jpeg isn't the smallest choice.

>>13332

This is stupid but my laptop only has 4GB ram, and normally I'm able to keep my browser and Hydrus open at the same time, and Hydrus is often frozen but still usable sometimes. But in trying to load a tab with 139,198 images at once, Hydrus remains frozen for 40+ minutes at a time for me, and I had to sit there watching it in my peripheral vision to catch it for the seconds in a row it was unfrozen.

I managed to load the images, highlight them all, press f3 to tag, and paste the desired tag in from a text document where I manually typed the tag when Hydrus was frozen. I even seemingly successfully pressed accept, because a subscription window partially loaded, signifying I returned to the main client window. But on this partially-loaded subscription window on the main client window, it's been frozen for like 10 hours now.

My measly 4GB ram never dips below 95% usage wih only Hydrus open in this state, with Hydrus using from 3 mil to 3.2 mil. So I was just wondering if there is a way to tag without loading images… like if I could perform the same search with arbitrary exclusions if possible, but without the burden of the client loading it all, or any of it in one tab.

>>13317

>Then I suggest you open a big search for all files with a pixiv url but no id tag and then ctrl+a and right-click->known urls->copy->pixiv file page urls, then open a new 'url downloader' page and set the tag import options to get tags even if urls recognised and file and in db, and then paste all those urls in. It should catch you up with the missing tags without trying to download the actual files again.

I just did this, and the number it shows is 0/12,025, when my search of "r-18" and "-pixiv work:* only showed 8,499 results. So I was worried it would only fix the r-18 tagged images, but it must've gotten everything instead. So I think it works, thanks.

>>13329

Great. Although the new downloader system is amazing compared to the old rubbish I had, it is not always pleasant to work with with the duplicate objects and so on. The next iteration here will have versioning or something and auto-updates.

>>13331

>>13332

I hope to add namespace siblings in some tag repository work that will start in the next 3-5 weeks! This has long been requested.

>>13348

Yeah, hydrus performance bombs out at about 30-50k thumbs in a single page. And manage tags doesn't work well for selections above about 8k thumbs. I am afraid the code just can't handle this stuff yet. Next step here is to improve manage tags so it does big operations asynchronously with a nice cancel button, and then yeah, I'd like some sort of faster tech for various large operations overall. The main problem for manage tags is when you try to add a tag, it needs to check all the files in the selection if they already have the tag in order to present you with 'pend to 3 files/petition from 2 files' kind of decisions. It does not scale well (and I think the final 'approve' stop is O(n²)), so for 10k files it just takes an unreasonable amount of time. As well as code optimisations, some CPU-faster entry actions (like 'woah lad, there's a lot of files here, let's make some add-only decisions beforehand') will come at some point.

>>13354

Great!

>>13367

Thanks for taking the time to read and respond to my random feedback on trying to add "pixiv work" tags to those that missed them the first time around.

I wanted to say too though that actually I was mistaken- the "url downloader" page only added "pixiv work" tags to the images tagged "r-18".

I was able to tell as much because I found a western artist who used to tag his images "r-18g", which is the gore tag, rather than mere "r-18" to mean adult. So every one of his images before he started using the "r-18" tag still didn't have a "pixiv work" tag.

I think I may not have a meaningful way to add "pixiv work" tags to every image that failed to get it the first time, in the end. Because even if I manage to do the "url downloader" for as many popular tags as I can, there will always be images that weren't tagged correctly.

But the only reason anything even has an "r-18" tag is because I used a "gallery" tab and put in a pixiv member id. I thought I could use the fact I've done that to fill in anything missing a "pixiv work" tag somehow.

>>13361

ok

>>13349

That resolution stuff is an interesting thought. I think trying to solve the problem of what is a useful resolution is beyond the time I have left in this current big job, and I should leave it to human eyes for now. I can add some common resolutions for now, though, and I can detect odd-numbers vs even-numbers easy, and we can iterate on that later if we see other easy answers. I'll make a job for nice resolution ratios, but it may come later.

is there a way to filter 'pixel for pixel' duplicates?

would love to be able to get those hammered out and could do that very fast too.

>>13373

Not yet–this data is currently generated on the fly in the duplicate filter. But I think it is very worth caching it in the db at some point. I can search it incredibly quickly to find pixel dupes and get started on some auto-dupe-resolve rules like 'if two files have the same pixels and one is a png and the other is not, delete the png' so we can finally and automatically clear out Clipboard.png trash dupes. It would be fast enough that I could even add a hook for it on import!

This would be optional and probably default on. More complicated stuff like 'if two jpegs are the same, keep the one with smaller/larger filesize' would be default off and up to the user.

>>13378

I would set both rules to delete the larger file size, it's very rare, but I have come across some jpegs were saving as png did lower file size.

>>13259

Thank you for this feedback. That's definitely the direction I'd like to go. The file maintenance system has proved a good step forward, so now I'd like to do similar for db-wide actions like analyze and repo processing. Get everything working on the same pipeline and timers and checks, and then I can neaten up the user feedback of 'hey, I need to do x, y, z big job, is this a good time?' into one system rather than the current cobbled-together system that consults several different maintenance clocks all with their own settings.

As a side note: standard python is restricted to using one core as a language. I can bump up CPU if I go into low-level C++ hard math stuff, or by calling other processes like ffmpeg, but up in my high-end, the language has a Global Interpreter Lock (GIL) that I can't get around. I have threads and everything, but none of them actually execute simultaneously on different cores.

The biggest sources of lag and latency in hydrus are due to my shit code rather than hardware/language limitations though. I just need to put the work in to make it neatly async, like the recent autocomplete improvements.

>>13260

Unfortunately, the current downloader system doesn't support the idea of 'removed' results. It always treats a new gallery request (or thread check) as an 'append any new stuff' action and doesn't take into account urls that have dropped off (which in the gallery context could have just cycled out to the next page). I think any clever additions to the downloader system will have to wait for the next big iteration of work on it.

I like the idea of marking threads in some way for later processing. I had always wanted to make the individual thread items drag-and-droppable so you could migrate them to other/new watcher pages, but I ran out of time last time. My best suggestion for now is to multi-select and right-click to say 'show all selecteds' presented files' or whatever, which will show all the files for several at once. If it all looks legit, and the threads are DEAD, I then recommend you drag and drop the big selection of files to a new processing page or give them a later-filtering-lookup rating/tag, and then clear out the DEAD threads.

>>13266

The new 361 should let you add specific potential pairs from the right-click menu. I am vacillating on whether I should/can easily add something like a 'launch dupe filter from here' command from a thumb selection. I think adding a temp tag/rating you can find again in the dupe filter is a good idea, or just picking your favourite from the thumbs in front of you may also just be easier, at least for smaller alternates groups. A quick browse through ten files in a row and choosing 'yeah, the one where all the girls have a benis and runny makeup is the best' when they are all in front of you and just deleting the others may be faster than going through them by pairs later on.

>>13268

Hey, just fyi: the 'speculative' search and the other options are inclusive of the more narrow choices. Speculative is really just 'hamming distance <=8', so it includes the others, which are 4, 2, and 0.

When I eventually add videos, I'll add gifs as well. It'll work on frames, so your pokemon gif should match enough of the still images to get linked up with the eventual dupe/alternate groups you create here, and will appear in 'speculative' searches and so on.

For finding things in trash, if you change the search domain, either on the duplicate files page or a page initialised with a 'similar files' system pred, from 'my files' to 'all local files', that should include trash. You may need to be in help->advanced mode to see this option.

>>13271

>>13270

Hey, you should be able to do this from manage tags right now, although it is advanced and debug-tier UI.

First, make sure help->advanced mode is on. Then select your thumbs and open manage tags. Hit the cog icon and then 'advanced operation'. It'll load the same mass-copy dialog you can do service-wide from review services, just limited to that selection. Double-check you have everything set how you want, and then click Go!. Make a backup before you get into this, as it is a powerful tool and if it goes wrong, there is no easy undo.

This dialog will get a pass in the upcoming PTR/tag services work I am planning. I'd like to add the tag filter object to it so you can more finely determine which tags are moved around.

>>13285

>>13286

Yeah, the current duplicate files search is a big atomic job, and if it goes on too long with a manually started job, the UI gets caught up until it eventually finishes. I don't like it, and as I move it to this new unified maintenance pipeline, I'd love to have it work in smaller async chunks like the file maintenance stuff.

Since you have a big db, I recommend you have it work in idle time (the cog icon on the duplicates page should allow this) and let it catch up in that system, which has improved cancellability.

I generally recommend users start with 'exact match' (i.e. 0 hamming distance) search to start with and work that queue completely before going for larger distances. This is much faster than the other searches and gives you easy decisions first. Because of the internal logic of the duplicates system, easy/actual duplicate decisions are more powerful/useful than alternate decisions–they eliminate the queued pair count faster.