/hydrus/ - Hydrus Network

Apparently the IPFS path redirect stuff is not completely fixed for client path locations outside the main db dir. I will give it another go this week.

>>13225

oof, on the second step of regenerating their metadata, this could take between 10 and 18 hours…

Wow wow wow tagging feels so much snappier now. Loving it! Thank you so much!

I stopped getting "pixiv work" tags over a week ago. It's been killing me but I thought it wasn't just me. But is it really just me? I just updated Hydrus and still didn't fetch the "pixiv work" tag from a subscription.

I don't know what I could've done to break it except I once dragged those image things into Hydrus in the hopes of making ripping from pixiv work before it "just werked" eventually. I remember the window I dragged the images onto was an anime girl, which scared the shit out of me at first since I thought it was a screamer. It didn't do shit as far as I could tell, but I wasnt sure how to undo what I did, so I just sat on it until Hydrus started doing the pixiv ripping out of the box. But I figure that broke this now.

Found a bug: If you press custom action in the dupe filter, after you enter your custom action, Hydrus will ask you which files to delete.

Tagging feels definitly more fluid now although it will take some time to get used to not searching via partial namespaces in some cases, but that is fine.

I definitly like that they work again once i already search for a tag - (as in i can just write 'char' to see all characters that a certain creator has used)

One thing i would like to see added is a way to look at the existing tags within a namespace inside the manage tags window - outside you can write 'character:*' and see all existing character tags but this does not work when you are inside the tags window - would that be possible to add?

Thanks for all the hard work!

>>13228

>>13225

>>13231

>>13233

Great, I am glad this is fixing your problem. Thanks for the updates. My intention is for the new file maintenance manager to be able to handle jobs competently in the background if the user doesn't want to burn CPU manually for 18+ hours straight. The current system has the default throttle of 100 jobs/day, which assumes all jobs have equal weight, but I think I'll have to reweight relatively quick jobs vs heavy ones, or maybe add a '100 jobs AND ten minutes' or something since if you had let your ~5M jobs run at 100/day, it would have taken 150 years of idle work every day.

I am still trying to grapple with the human workflow of some of the numbers we have here. Hydrus is a powerful magic wand, able to do more than we are used to.

>>13229

>>13239

Thanks lads. This took a while, but it was definitely a case of slow and steady wins the race. There's still some edge cases to sort out and some logic to improve, but the snappiness here is the ideal for where I want most of the ui to be. I can improve the lagginess of db access across the program in similar ways, it'll just take the work to clean up the code.

>>13232

I can't say confidently what is going on with your client, but a pixiv tag search test on my dev machine here seems ok.

Pixiv update their access frequently, so if your client is now running on and old or accidentally broken parser due to the downloader import, this would explain what was going on.

This has happened a couple of times with different sites now and there is no excellent 'just reset this site to default' button, which suggests I should write one. Until then, I think your best solution is to do something like:

Hit all the manage dialogs under network->downloader definitions and delete anything pixiv related. Those dialogs should have a 'add defaults' button, which will let you reload the default objects that come with the program. Add all the default pixiv stuff back in. You can probably skip the login stuff, if that still seems to work for you. Then hit manage url class links under the same submenu and hit the try to fill in gaps… button to relink everything back together. Check your pixiv subs are pointing to the right 'GUG's (i.e. downloader entry, should be called 'pixiv tag search') and then give it another go.

That is a bit of a hassle, so you could first try just seeing if you can manually switch back to the default parsers under manage url class links. I don't know exactly what your client has, but it probably has the defaults still under it. If the problem is your pixiv is now pointed at custom parsers for the gallery/file url classes, just switch them back to the simply named defaults and you'll likely fix what is wrong. Failing that, do the complete wipe/reset as above.

>>13234

Great, I am glad you are back up. The speedup is likely due to the clone, yeah–it reads and writes the whole db, so it works like a great disk cache for the next x hours.

The namespace situation is getting out of control! Once I am done with the current duplicates work, I'll be doing some PTR and tag-repo and tag siblings work to try and put some fires out. Namespace siblings, like 'call all "idol:" tags "person:" tags' is a top priority. At the moment, the only solution would be to do individual siblings en masse, which is not practical.

Tag siblings are great overall, but the whole system is still running on the 1.0 attempt, and I feel it does not account for human preferences enough. Shit like 'bodypart:' is creeping into the PTR, which is a fundamentally correct namespace–we can all agree that 'breasts' are a 'bodypart'–but different users disagree on whether they want to actually see it as a namespace. Permitting client-local control of namespace display cuts the knot of this problem.

Eventually, tag siblings will have a 'tag definitions' system, group-based like the new duplicates system, that will place semantically similar tags on equal footing along with extra info like 'this tag is in japanese language, it is slang' and let users choose their language preferences and whether they want to see 'vagina' vs 'pussy' based on proper vs slang rather than for each pair again.

>>13240

Thank you for this report. I was going to say 'this is actually intentional', which it is, but I just did a bit of testing here and I saw that choosing 'forget it'/'no' on the delete dialog actually cancels the whole action. The intention was to add an optional delete to custom actions, not mandatory. I am 99.7% confident I

can fix this for next week.

>>13242

I just answered a question on tumblr about the namespace searching:

https://hydrus.tumblr.com/post/186423570874/loving-the-better-autocomplete-in-v360-only

It is complicated to explain nicely, but I will fix up the wildcard logic so you can put in 'char*:sa' and have it implicitly enter 'char*:sa*' and return 'character:samus aran'. I think moving to the workflow of just putting 'c*:' before your input may solve the problem nicely and still give you options.

Thank you for the report about 'character:*' in the 'write' autocomplete. I am not sure why it is not fetching all the results on that call, although a bunch of the logic here is still from the old system, which protected certain inputs like '*' from being searched due to the lag. Now we have some more control and no ui blocking, I'll see if I can undo some of these limiters and give you more power.

>>13245

Thanks for the reply. I tried nuking all the pixiv-related stuff under the dropdown but after doing that and "try to fill in gaps" I still can't fetch "pixiv work" tags. I even went back to the "manage default tag import options" and selected "get tags" to fetch "all tags" but still it doesn't fetch "pixiv work" tags for me anymore.

I didn't check my pixiv subscriptions- I just opened a new gallery tab in Hydrus and pasted someone's pixiv member ID or however, after selecting "pixiv artist lookup" from the dropdown. It counted up until completion as normal, but didn't fetch "pixiv work" tags.

I didn't redo the log in for pixiv because I didn't think it would help if Hydrus can fetch the images and every other tag fine.

>>13243

the 4 million jobs ended up taking around 2.5-3.5 hours, I'm not really clear on exact times as it would go fast and slow so I wasn't paying close attention, but that second task did end up taking 18 hours.

Here is my suggestion, and its probably going to only get more apt going forward looking at cpus, you sould have a big multicore more that more or less ignores background use, because lets be honest, the program on my cpu meter even in the time intensive stuff never even registered as cpu use, and the most the program ever eats cpu is when it does things with ffmpg or whatever it is because that will use an 8core16thread cpu, hydrus itself will only use 1 thread

It would probably also be a good idea to have a toggle for things that will hard lock the use of the program from things that will only momentarily lock it when it has something to do. similar metadata from the duplicate window, that will lock you out of the program while its doing its thing, but the metadata regeneration I was doing, while time intensive, wasn't locking me out. I think that would be a better way to divide, because personally, 8/16 cpu here and even when hydrus is doing everything it can to lock itself up, I dont feel it, but when something hard stops the program form being used, that I do feel.

I also think there is value in sorting things that can get done fast vs things that take time, but I think the program telling you something like 'dude, I gots some shit I need to get done, mind if I go at it for a while' kind of popup would be appreciated, also, if there was a way to estimate time based off a data set of what has been done vs what needs to get done that would also be nice.

>>13260

sauce on the middle image?

>>13266

Thank you mate. But I tried reverse image searching with yandex, iqdb, tineye, saucenao, all returned nothing. But google showed one result from /aco/:

https://boards.4chan.org/aco/thread/3331595/blacked-twinks-and-traps#p3331645

I recognize the artist name "onta", and I ripped their twitter "doxyonta" before, but dude posts way too many sketches, crops, plus typical twitter garbage to be able to reasonably find any specific thing there. The md5 didn't match anything in Hydrus for me, and the look for similiar files function found nothing.

Realistically it's just another twitter crop or something though. I last ripped his twitter 6 mo. 22 days ago.

>>13270

<using the copy button in multiple pictures copies all of the tags and moves them all as a single block of tags.*

>>13266

artist?

>>13274

the best you have to go on is the signature in the lower left, never looked into it but someone else said its likely a crop.

>>13275

ah sorry didn't notice it

thanks.

Thank you, based Dev.

Ok, so in getting duplicates sorted on a rather large image dump with many similar images, an exact match duplicate search is not picking up the similar ones, so I decided to try out going a few steps down, due to how small the image set was.

so… doing this on a speculative search gives me 1 extra file done every 30 or so seconds… so noped the fuck out of there

similar while not as bad, seems to be 30 files every 10-20 seconds. nopeming back off again I go to very similar.

this is about 30 files every second, ok, I can wait for this to finish and hopefully get all the files I want… granted this will take longer then manually filtering similar, but it seems that exact/verysimilar/similar/speculative searches are all kept over time and this will get done one way or the other.

Now, I thought of 2 things

1) the program just hard locks if I keep speculative search going, I have had to open a fresh session to be able to get things done and hopefully not lock the program up. Is it possible to get this to work in chunks of 1000-10000 and commit? I will admit that this is almost entirely a problem with existing large databases and any new database would not have these issues, it may even just be my session fucking over the program… either way a commit from the program every so often would be helpful.

2) now, the image set I want to work with is only about 2000 files big, would it be possible to set a specific file set up to be processed in? especially with speculative being so long to even do one file.

I don't know if 2) would require a smaller subset of the full archive duplicate processing, or if it could be done along with the full dup processing, but it would be nice to be able to work with smaller filesets when needed.

>>13285

still hung forever even on a fresh session… it seems like 20k files and any more then that at a time causes it to hang indefinitely, it's possible that after several hours it may show up done, but that's not something i'm willing to leave to chance, going to keep filling this out a few thousand at a time, but i'm not expecting a whole lot.

>>13257

Here is my reply: >>13317

>>13259

Thank you for this feedback. That's definitely the direction I'd like to go. The file maintenance system has proved a good step forward, so now I'd like to do similar for db-wide actions like analyze and repo processing. Get everything working on the same pipeline and timers and checks, and then I can neaten up the user feedback of 'hey, I need to do x, y, z big job, is this a good time?' into one system rather than the current cobbled-together system that consults several different maintenance clocks all with their own settings.

As a side note: standard python is restricted to using one core as a language. I can bump up CPU if I go into low-level C++ hard math stuff, or by calling other processes like ffmpeg, but up in my high-end, the language has a Global Interpreter Lock (GIL) that I can't get around. I have threads and everything, but none of them actually execute simultaneously on different cores.

The biggest sources of lag and latency in hydrus are due to my shit code rather than hardware/language limitations though. I just need to put the work in to make it neatly async, like the recent autocomplete improvements.

>>13260

Unfortunately, the current downloader system doesn't support the idea of 'removed' results. It always treats a new gallery request (or thread check) as an 'append any new stuff' action and doesn't take into account urls that have dropped off (which in the gallery context could have just cycled out to the next page). I think any clever additions to the downloader system will have to wait for the next big iteration of work on it.

I like the idea of marking threads in some way for later processing. I had always wanted to make the individual thread items drag-and-droppable so you could migrate them to other/new watcher pages, but I ran out of time last time. My best suggestion for now is to multi-select and right-click to say 'show all selecteds' presented files' or whatever, which will show all the files for several at once. If it all looks legit, and the threads are DEAD, I then recommend you drag and drop the big selection of files to a new processing page or give them a later-filtering-lookup rating/tag, and then clear out the DEAD threads.

>>13266

The new 361 should let you add specific potential pairs from the right-click menu. I am vacillating on whether I should/can easily add something like a 'launch dupe filter from here' command from a thumb selection. I think adding a temp tag/rating you can find again in the dupe filter is a good idea, or just picking your favourite from the thumbs in front of you may also just be easier, at least for smaller alternates groups. A quick browse through ten files in a row and choosing 'yeah, the one where all the girls have a benis and runny makeup is the best' when they are all in front of you and just deleting the others may be faster than going through them by pairs later on.

>>13268

Hey, just fyi: the 'speculative' search and the other options are inclusive of the more narrow choices. Speculative is really just 'hamming distance <=8', so it includes the others, which are 4, 2, and 0.

When I eventually add videos, I'll add gifs as well. It'll work on frames, so your pokemon gif should match enough of the still images to get linked up with the eventual dupe/alternate groups you create here, and will appear in 'speculative' searches and so on.

For finding things in trash, if you change the search domain, either on the duplicate files page or a page initialised with a 'similar files' system pred, from 'my files' to 'all local files', that should include trash. You may need to be in help->advanced mode to see this option.

>>13271

>>13270

Hey, you should be able to do this from manage tags right now, although it is advanced and debug-tier UI.

First, make sure help->advanced mode is on. Then select your thumbs and open manage tags. Hit the cog icon and then 'advanced operation'. It'll load the same mass-copy dialog you can do service-wide from review services, just limited to that selection. Double-check you have everything set how you want, and then click Go!. Make a backup before you get into this, as it is a powerful tool and if it goes wrong, there is no easy undo.

This dialog will get a pass in the upcoming PTR/tag services work I am planning. I'd like to add the tag filter object to it so you can more finely determine which tags are moved around.

>>13285

>>13286

Yeah, the current duplicate files search is a big atomic job, and if it goes on too long with a manually started job, the UI gets caught up until it eventually finishes. I don't like it, and as I move it to this new unified maintenance pipeline, I'd love to have it work in smaller async chunks like the file maintenance stuff.

Since you have a big db, I recommend you have it work in idle time (the cog icon on the duplicates page should allow this) and let it catch up in that system, which has improved cancellability.

I generally recommend users start with 'exact match' (i.e. 0 hamming distance) search to start with and work that queue completely before going for larger distances. This is much faster than the other searches and gives you easy decisions first. Because of the internal logic of the duplicates system, easy/actual duplicate decisions are more powerful/useful than alternate decisions–they eliminate the queued pair count faster.