/hydrus/ - Hydrus Network

>>12458

This is a temporary issue, as its likely going to be completely sorted out by the next update.

I have 2000~ files in the trash at the moment and because they are in the trash, they can't get a custom reason despite none being there. I don't think this will be an issue that manifests itself going forward but if there was an override for the reason they were sent to trash to get a new one that would likely solve all future instances of this too.

other then that, I am loving the system

>>12459

Hey, I am glad it is working for you. The reason is applied as the file is added to the trash, so if you would like to set a new one for those files, I think untrashing them (shift+delete) and then advanced retrashing will sort you out.

>>12458

Am i missing something or is the duplicate filter not doing anything? I ran the filter for a day and clicked through a bunch of different pictures but then the next time I launch the duplicate filter it shows me the same ones I already sorted as filtered and "comitted 150+" changes.

>>12464

The shortcuts system is another one in mid re-write atm. There is a still lot to do just in getting all the simple commands working on the new system, so I'm afraid I don't think I can get advanced commands like 'delete with this reason' going soon. Can you maybe navigate the new delete dialog with tab and arrow keys? This is probably inconvenient, but could you maybe macro 'tab tab down tab down down tab enter' to your keys for the different reasons, just for your delete sessions?

Yeah, the single manga pages is a tricky issue. My belief is that files' value is not in their existence, but in what you get out of them. If you will never look at them or want share them again, just delete them. If you want to keep some single good pages, keep those. Otherwise delete. Don't spend today worrying about integrating them into comic pages 15 years from now as we'll have a whole bunch of better tools then that'll solve all these problems better. Focus on what you want to look at in the hypothetical today. I'd say do 3.

>>12476

The SFM Community™️ has a long history of these 1080p gifs and other silliness. Same as artists putting up 'original' quality 60MB 14,000x22,000px pngs on their patreon. Artists tend to be excellent at aesthetics and less so at practical file formats. There is no solution in the client atm, but I can definitely see some optional auto-conversion in the future. I taught myself how to do some more advanced ffmpeg conversion the other week, so I know a bit more what I am talking about here.

I think we'll see how video dupe filtering goes, once I've added that, and then chase up with various systems for easy conversion/optimisation. Big gifs are stupid almost all the time, and I should think most users would rather have video if we can figure out a decent conversion regime (the dupe checker tech could help with this). And I wouldn't do it myself, but I know some users are thinking of converting all their jpegs to webp or flif or whatever, and hydrus could facilitate that sort of thing as well. I wouldn't be surprised if these older file formats will be completely subsumed by better systems in future. Especially so as we move to higher colour depths in all formats. You may not have seen, but hydrus supports bmp importing, but it auto-converts to png.

>>12480

Thank you for this report. Can you say a bit more about it? What sort of tags are you using in your duplicate filter search? If this is on 349, that had an undercounting bug, so if you were clearing your apparent queue down to '0 pairs' and then next boot it went up again, this is now fixed in 350.

If you are seeing similar files in subsequent filters, is it possible that you are just seeing some 'similar' looking files, and not exactly the same ones? The way the current pair-processing queuing logic works, I have to separate decisions on 'groups' of similar files into separate batches of 'committed' so they do not step on each others' logic.

Can you see if you can find out more about a potential group here? I am thinking something like this:

Open the dupe filter and work until you find a file with some decent tags that you can find again (Even better if you think you have seen that file in the dupe filter before)

Do not process that pair and exit the filter

Find the file(s) again in a regular search page with their tags

Right-click->view this file's duplicates->potential (you may need help->advanced mode on to see this)

Are there only two files, or more?

This isn't surefire, but if you discover you have seven very very similar files, that would explain this. You could have dupe-filtered A-B on one cycle and A-C on another and A-D and then B-D, all depending on what their relationships were.

Alternatively, can you reduce the filter domain so it is only, say 10 pairs? Just add some tags to the search until it gets small. If you process those 10 pairs and refresh everything, does it stay '0', or does the count go back up? If it stays at 0 then your pairs are being processed, and that suggests you are seeing multiples of very similar files. In fact, in general, if your 150 changes (or even just 2 on a test run) reduce the 'potential pairs' count by at least 150, then I feel fairly confident it is working basically ok here.

All that said, I know there are bugs in the system. I don't think there are any big enough to drop 150 pairs in a whole batch, but I suspect one or two transitive implications are under- or over-applying. I am looking forward to likely working on the db-level here as part of my next big job, as the poll seems to be going atm.

>>12482

v349

I didn't use any tags. I just let the duplicate filter try to find potential results. I'm 90% sure that it was giving me the same files I already sorted. Does the dup filter only move tags or does it put them in the trash? I couldn't find any recently filtered ones in the trash and it gave me the same files after running it again, this time it did move some to the trash though so maybe it wont show up in the dpilicates processing again.

>>12458

>and deleting files without leaving a delete 'record'

Right when I was going to ask about that. Thanks.

I think a better term for this action is 'forget'.

How does deletion reason/forget/… work with "archive/delete filter"?

>>12481

With delete, I don't mean delete with reason, what I mean is in current hydrus, there are various hotkey presets depending on what you are doing

So when you get into a delete menu, its hotkey preset takes over, and like I said in my case it would be 1-8, pressing the number could select one of the reasons.

Like I said before, my use case for this is still quite a bit down the road because mass culling is still easier and will be for several months. no idea if this is possible in the current shortcut method but if it is, it would be something to do to fill time if its easy enough.

as for manga i'm leaning on keep the 'money shot' pages

If I read the manga, it will make me laugh/make me want to re read it, if I didn't, I still may get something out of it, if its porn, well… that one obvious.

as for conversion…

personally with that 112mb attrosity, I got it down to 1.5mb after fucking with settings in handbrake for a while

For what to do in the future… that a a bit harder

first you would need a 'lossless' conversion, as some animated gifs are bloated to hell and back, but they have very clean lines, and a conversion even as lossless may end up at a lower file size then non converted.

second you would need a near placebo preset, not quite there but still high quality

third would be a high quality compression, a good catch all for everything,

forth would be another relatively high quality preset, lower but still a catch all

and below that some fuck this presets, I have a few videos where they made a music video of sorts, but it's over 200mb and the only thing that's worth anything from it is the audio, but the visuals are a nice touch. a good variable bitrate could/should crunch that down to acceptable levels, but at least with handbrake currently, I don't think there is a way for me to do that without fucking with audio, whatever presets you make, if its possible to do nothing to the audio, all the better.

at least as far as I can see, these are the only presets you would need in hydrus, as if anyone wants to do more, an outside conversion would likely be better.

as for other formats, If I can go lossless to lossless and save space, I would convey everything immediately, then convert back on a share.

for jpeg or lossy, if I can convert to, get 95%+ of the quality, and save a receivable amount of space, it's another no shit for me, but that's just me.

as for bmp to png, I think it may have come up a few times dealing with my older dbs, but for the most part, bmp has never been something I collected/found.

On the topic of sfm, I know a little about 3d rendering, you don't render out a movie, you instead render out images, just because if the program crashes or fucks up 1 image, its a lossless way to re render. you can also stop the rendering and resume easier. so putting said images together into a gif opposed to figuring out editing program is probably a lot easier. granted im able to straight drop these into handbrake so the value of the gif is iffy at best.

on the side of 14000x22000 artist images, again this is where I know quite a bit. working at a higher resolution, even if you dont save and distribute it at said resolution, you are able to do color mixing and are more easily able to ignore minor fuck ups then if you were making a 1080p at 1080p, it also allows you to use something like a textured brush so you get smaller details without trying without making it obvious you used a textured brush. on the side of the artists selling them, the ones who do it right are selling them as a teaching tool, and the morons who only buy them for the highest quality… got a very low opinion of these idiots.

that said you make a great point. if duplicates is able to deal with video, then what ever I do on my end to shrink the shit, will get a higher quality version found if a higher quality version exists.

was honestly not even considering that when I asked. that makes this a bit easier for me to move forward

>>12488

I dove into the code here as part of my research into the dupe db overhaul yesterday and discovered some problems in my original 1.0. I feel less confident about it as I did for certain edge cases where 4+ files want to figure out what worst->best order they should go in. What happens to tags and whether the 'worse' of a pair is deleted is up to the duplicate merge options (check the cog icon on the dupe filter to edit these). I believe the defaults are to delete the worse of a pair, but if for some reason yours is not set this way, I think this will help your processing. Clearing 'bad' files out of the local file domain clears out some unusual AB, BC, AC pair comparisons the current system wants to do.

My plans for the new storage system will do away with the over-autistic comparisons here and push for a simpler model with groups of dupe files with a defined 'best quality' King.

>>12498

The archive/delete filter still uses the old delete system atm, just sending files to trash with a default reason. I would have added the new delete dialog to it, and likely will in future, but the batched way archive/delete filter stores up and commits its actions make it more complicated to integrate, so I put it off.

>>12499

I will not hardcode number shortcuts for certain controls for now. I think macros to tab&space navigate the dialog are your best bet atm.

That's interesting about SFM, I didn't know about the image-based rendering. I had heard the software tends to crash a lot, especially as artists are often working on older laptops etc… Sounds like a nightmare. I assume there is no easy one-click open source workflow–that for instance just neatly eats up their pngs and throws them at ffmpeg–for these users, unless Handbrake can do something like that?