/hydrus/ - Hydrus Network

Ok, i'm going though duplicates now,

got a problem, better/worse pairs are not registering numbers, alternate pairs are, not sure about same quality, as I have just been picking one from between two when one of them pops up.

also, a general 'copy all tags' and 'copy all ratings' would be helpful, especially with ratings, as the deeper I go the more often I will have to edit that setting to add more ratings to take into account.

Just got this error

RuntimeError

Failed to gain raw access to bitmap data.

File "include\ClientGUICanvas.py", line 5722, in EventPaint

self._Redraw( dc )

File "include\ClientGUICanvas.py", line 5680, in _Redraw

wx_bitmap = self._image_renderer.GetWXBitmap( self._canvas_bmp.GetSize() )

File "include\ClientRendering.py", line 145, in GetWXBitmap

return wx.Bitmap.FromBufferRGBA( wx_width, wx_height, wx_data )

Not exactly sure why that one is about

>>12164

my sides

>>12182

yea gotta love the shotgun approach.

that said I want to say that the dup filter is less random than I thought as I think I have dealt with every one of them now.

How would I go about making pictures a child and parent of each other? For example I got a set of images, image 1, image 2, image 3. I want to order them like this since they go together. How would I set image 1 to be a child of image 2, and image 2 to image 3 while linking it to image 1? I want them to be like a set of images that goes together and not individual ones

>>12184

That functionality does not exist yet. I think if you want this, you can vote for

>Improve duplicate db storage and filter workflow (need this first before alternate files support)

in the upcoming "next big thing" poll.

>>12153

> Now, when you trash a file, a context-appropriate 'deletion reason', such as 'Deleted from Media Page.' will be saved. These statements are mostly trivial, but duplicate filter actions will specify a bit more about the duplicate processing action type. This text will be recovered in an import status window for 'deleted' status results, just as a help if you want to investigate closer (e.g. perhaps you are not sure why a particular file failed to import, but then you see the reason is you already decided you have a better duplicate version of it). Any files deleted before this system will just give "Unknown deletion reason."

Is one of the deletion reasons, removed via duplicate processing rules?

>>12186

alright thanks, is there a way to make hyrdrus show picture that aren't with a certain tag? For example picture 1 has a tag with "character: person 1" and picture 2 and 3 has no set tag. How do I filter out picture with a set tag so it only shows picture without that specific set tag?

>>12193

oh nvm found out using "-" will exclude a tag for me

>>12194

That is good, if by change you want to see tags with the least tags you can do a system:everything

system:limit=100

and sort by amount of tags

>>12164

ICARUS HAS FOUND YOU!!!!!

>ICARUS HAS FOUND YOU!!!!!

>>ICARUS HAS FOUND YOU!!!!!

>>>ICARUS HAS FOUND YOU!!!!!

>>>>ICARUS HAS FOUND YOU!!!!!

>>>>>ICARUS HAS FOUND YOU!!!!!

>>>>>>ICARUS HAS FOUND YOU!!!!!

>>>>>>>ICARUS HAS FOUND YOU!!!!!

>>>>>>>>ICARUS HAS FOUND YOU!!!!!

>>>>>>>>>ICARUS HAS FOUND YOU!!!!!

>>>>>>>>>RUN WHILE YOU CAN!!!!!!!!!!!

>>>>>>>>RUN WHILE YOU CAN!!!!!!!!!!!

>>>>>>>RUN WHILE YOU CAN!!!!!!!!!!!

>>>>>>RUN WHILE YOU CAN!!!!!!!!!!!

>>>>>RUN WHILE YOU CAN!!!!!!!!!!!

>>>>RUN WHILE YOU CAN!!!!!!!!!!!

>>>RUN WHILE YOU CAN!!!!!!!!!!!

>>RUN WHILE YOU CAN!!!!!!!!!!!

>RUN WHILE YOU CAN!!!!!!!!!!!

RUN WHILE YOU CAN!!!!!!!!!!!

>>12159

I assume the numbers for better/worse that you are not seeing increment are on the little panel on the duplicate filter page, right? They are staying at 0? I think I should redesign this or something, I am not sure, but that is actually correct behaviour with default settings.

With default settings, setting a better/worse pair trashes the worse of the two. This removes that file from 'my files' local file domain, which that panel defaults to looking at. And since then not both sides of the pair are in the domain, their better/worse relationship doesn't count for it. If the worse files are still in your trash and you change the file domain on that page to 'all local files', it will include files in the trash, and you will see positive counts.

I agree on the copy all. I made the initial version for maximum customisability, but tbh most use is far simpler.

>>12163

Yeah, the current phash similar-looking search technique is, except in extreme cases, immune to stretching or changes in HSL. I am overall really pleased with it. It can also deal with slight clipping, like 2-5% border clips and so on. It can't do flips or rotations, but I expect I will add that in the next iteration on this.

>>12164

I haven't seen an exact dupe situation this bad before, but I guess that since these images are of such extreme banter value that some anon out there was adjusting a single byte over and over (to change the hash) so he could keep posting them to 4chan maybe? Or otherwise keep desu-spamming them to the same thread that had a similar hash dupe check. Or maybe, ha ha, the different versions are an intel op that wanted to trace where certain identifier keys placed in the different versions then ended up on twitter and other big sites, to better map out which twitter users go to which places.

>>12164

>>12205

To continue, the current dupe system is unfortunately very bad at handling large groups like this. Before I can do any real extensions on the dupe or alternate system, I want to rewrite the current pair-based storage to a cleverer and more compact group-based on. If you try and action that big mess in one go, it will warn you about like 120,000 pairs. I forget if it in n-squared or n-factorial, but it is definitely n-shit for large groups. Don't try and action it that way, or you'll just clog up your db for now. No good solution atm.

>>12167

This looks like a variant of an out-of-memory error. Were you zooming way in on a big file during dupe comparisons? Unfortunately, my image viewer is not good at big zooms atm, and if you like 2000% on a 4,000x3,000px png or something, it'll hit a limit somewhere trying to make a bigass bmp for the whole thing. You may have noticed the client getting laggy during bit zooms like this. A future iteration will have a tiled rendering system that only keeps the bit on screen in memory, but for now it is still basic.

>>12164

Oh yeah, your odds of getting the same image in a row are fairly high, as my current 'get some dupes to process' job batches decisions together into various similar types. You'll likely get several pairs in a row that feature the same file as A or B.

>>12190

Yeah, it'll say the action as well. In the filter it will be something like:

"Deleted in Duplicate Filter (better/worse): worse file deleted."

If you do a mass action from thumbnail right-click, it says this variant:

"Deleted from duplicate action on Media Page (action): one/both deleted."

Just did another clean up as hydrus was eating 6gb on launch and dups were starting to lag the program to much, its now starting sub 1gb with next to nothing changed since the last time I did this which got it down to 2.3~gb

still honestly see no reason for the watchers once they 404 to eat that much space, but good to see the program is getting even better with ram management.

>>12205

as far as the images themselves go, yea, they are likely changing 1 pixel and re saving it, my shotgun approach to downloading has managed to seemingly sweep up everyone ever made but there are a few times where I knew the image was never getting posted again I just straight deleted the shitposts, but this time…

I thought I got rid of all of them, but apparently it's somehow finding more.

>>12208

after a while I thought this too as I was getting somewhere of 455k images, it was really odd that there would be 3 or 4 images that were dups of the same image that would come in.

>>12204

I have to ask, is it possible to have a custom sup finding thing? sadly I went though these a number of weeks ago, but I was checking an import archive, and apparently what I used to download corrupted images, it's more or less the standard at some point the image below this line is no longer loaded and its solid grey lines.

with images like this, would it be possible to have a 'corrupt dup' setting where you either point out where the image is good, or where the image is bad (a fill tool may deal with bad better then good making it easier to do) and have it discount that section from a dup search? my logic here is if I have a large enough archive, or the interests see me downloading similar images regularly, there is a good chance I have the un corrupt version somewhere else already

and just to head this off, these are images that have gone from a 120gb maxtor, to a 1.5tb seagate and survived 3 hdd failures there, to be moved to a 4tb and finally to am 8tb hdd, and finally to a currently 3tb image archive drive. odds are one of the drives fucked with the files in some way, or it was my horrendous method of acquisition all those years ago so the issue isnt my hdd corrupting images, its more I have corrupt images that may have to much corruption to be dup found

>>12239

No, I am afraid sectioning off good and bad areas of images is way too complicated for my current system to deal with. I encourage you to delete all corrupt files.