/hydrus/ - Hydrus Network

More awareness of exif/xmp and other embedded metadata.

For example, dupe checker being able to tell that two jpgs are 100% identical when metadata is stripped. Maybe even automatically mark as "same quality".

I've played around a bit with exiftool,

( https://www.sno.phy.queensu.ca/~phil/exiftool/ )

And a lot of photoshop jpgs have metadata about the original name it was saved under and even ID numbers pointing to the source image if it's an edit.

Sorry for not splitting this off into this thread in the first place. Here's a copy pasta:

Would you be averse to fleshing out the hover tag pane on the left of the media viewer with more functionality? I picture a possible workflow inside the media viewer being:

1. text input gets captured automatically by a new search box added atop the hover tag pane when pane has focus.

2. The hover tag pane shows tags already on the image when the search box is blank. When search box contains text, it's populated with autocompleted tag suggestions, and any autocomplete suggested tags that are already on the image have a highlighted gold background.

3. If there's no autocomplete tag suggestions, then the tag doesn't exist and therefore the Enter key should create and append the typed tag to the image. Idk what the performance implications of this would be, so maybe namespacing can be used to cut down the tag pool so it's less laggy?

4. up/down arrows can navigate autocompleted tag suggestions without needing to use mouse input to change focus from the pane. Enter key appends tag to image, delete key removes tag from image. Shift/ctrl click to select multiple tags perhaps?

5. Left/right arrows can navigate to the next image without needing to change focus via mouse click.

6. right click>mange>file tags for more advanced tagging

wish i could have multiple pages open with 90,000+ images each without lag

reeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

A way to exclude or only include items in a search that I've marked as an inferior duplicate or superior duplicate

Could shift+del force the advanced deletion dialog to appear no matter your settings?

Normally I don't want a confirmation dialog when deleting files, but sometimes if I'm deleting many files at once I like to enter a reason, or delete without leaving a deleted record. If would be handy if I could just shift+del instead of going into the settings to change it then later change it back again.

So some of the menus, notably the tag sibling/parent menus, have protections on them to keep you from closing them accidentally and make you confirm you want to close them if you have uncommitted changes.

Can those protections be added to menus like subscriptions, the various parser config menus, and other configuration menus where pressing Esc one too many times can undo a stupid amount of work?

Also can you implement a tiny tweak, basically switch the order the ListBox that shows your recently added tags in the tags menu and the clear button above said ListBox are added to the layout. This will make it essentially impossible to accidentally click 'clear', which I do on a frustratingly regular basis. Just switch lines 128 and 129 in ClientGUITagSuggestions.py.

How about having the deletion prompt auto focus on the deletion reason so you can easily change it with up/down arrow keys and then press enter to select.

If it's possible I would like to have the option to have the pages tabs form multiple rows instead of disappearing off-screen with the little left/right scroll buttons when you have too many of them. It's just too slow to navigate or get a good overview with the scroll buttons.

Another useful thing would be to have each subscription make its own page of pages and give each query a separate page within it.

Could the new favourite searches feature be repurposed for adding multiple tags to files at once? For example I could make a new favourite tag group of 'brown hair, brown eyes' and add both tags with one selection in the manage tags window

When I'm gooning I like to have 16+ slideshows going at once across my monitors. This is a hassle to set up each time manually, I wish there was an easier way!

Could you make it so that tags that you have as a favorite show up with some sort of highlight or bold or something else that makes them easy to pick out from other tags in the single and multi file views. I don't know about everyone's use-case, but I add tags to my favorites if it's a tag that I care about, so it'd make sense that I'd also want to be able to know at a glance if these tags are present on a file I'm looking at, or a group of files I have selected.

I'm making this suggestion because I've noticed that with the increasing number of tags that are appearing on files in the PTR (which is normally a good thing), it's becoming increasingly difficult to tell if the tags that I'm actually interested in regularly using are properly applied to the files they belong on without needing to carefully look through the long list of tags. This problem becomes even worse if you don't have the tags listed in alphabetical order, which is a shame because I'd prefer to use another sort, like incidence. Highlighting tags in your favorites would make it much easier to find those tags in the long lists, with the benefit of being easy for users to understand and (probably) easy for you to implement.

Of course, this functionality could also be decoupled from favorites if you think that would be a better idea. In that case, you could just have a separate list of tags that users add tags to, and tags in that list get highlighted. For even more granular control, they could set different highlight colors for different tags, making it easier to separate them into groups, and even easier to quickly see if a certain tag is on a certain file, even if the file has many tags.

The "manage url classes" and similar windows REALLY need a confirmation dialog when clicking the X with unapplied changes. Just lost all the work I did by mistake thinking they were already saved because I was hitting "apply" on the sub windows.

The "manage url classes" and similar windows REALLY need a confirmation dialog when clicking the X with unapplied changes. Just lost all the work I did by mistake thinking they were already saved because I was hitting "apply" on the sub windows.

The manage tags window really need some more quality of life functions for when editing tags of many files at once.

It needs a search or filter function so that I can find the tag I want quickly. Say I want to remove the tagme tag from 20k files. I gotta scroll and scroll the tags list until I find it so I can double-click to remove it. Many lists in Windows allow you to press any letter key and it will jump to that letter but even that doesn't work in Hydrus.

A "replace tag with" function would also be really useful. I get that for the PTR it is better to use siblings but for my local tags sometimes I really just want to remove a tag and replace it with a better one. It can be done now but it requires a separate search for just that tag. It would be helpful if from the manage tags window I could just rightclick a tag and replace it with another that I enter on just the files I have selected.

>>14009

>Say I want to remove the tagme tag from 20k files. I gotta scroll and scroll the tags list until I find it so I can double-click to remove it.

You can just enter tagme again into the tagging box and it will remove it.

>>471

"A (will display as B)" is pretty verbose, it would be nice to have a danbooru style arrow A → B

>>14011

Good point, thanks

Simple Exif data manipulation please.

Something I often use Hydrus for is sharing photos on imageboards.

Some of these photos are taken using my personal device.

I like to strip these photos of all Exif data before posting – for privacy reasons, minor as they may be.

Hey, thanks for your work. I'd like to suggest a few things:

Use Firefox's privacy.resistFingerprinting useragent as the default one. It's a generic UA used by a lot of people. I don't think we gain anything from broadcasting an unique UA. It rarely updates and is currently: Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0

Something to do about CloudFlare. At least let us solve the catchas it gives. Ideally hydrus should simulate a browser like firefox better.

Tor integration.

Having a tag service for subcription tags separate from "my tags" and enabled by default.

Be able to apply lossy compression for files. Possibly with an option to retain metadata like hashes.

Make floats be a part of the main window instead of spawning new ones. They bug out some WMs.

Cache tag repo sync for a while before writing to disk. Currently it writes massive amounts of data to disk, taking a while and thrashing SSDs.

Split up the database files into services and services into chunks.

Keep files in different directories based on predicates. (e.g. system:inbox or system:rating = 5/5)

Purge a file or mapping from the db as if it never existed.

In the same vein as the previous one, journalling that you can revert at will.

Option to download only thumbnails until user decides to download the whole file to save bandwidth on images you obviously don't care about.

Separate sessions for subscriptions with their own cookies.

Hope you can at least do the first two soon. Unique UA isn't doing us any favors and CF kills off use of a lot of boorus.

It would be nice to be able to read doujins double-paged, LTR or RTL directions. There's often two-page spreads that I can't view easily.

>>14267

> Use Firefox's privacy.resistFingerprinting useragent as the default one. It's a generic UA used by a lot of people. I don't think we gain anything from broadcasting an unique UA. It rarely updates and is currently: Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0

Yes

> Something to do about CloudFlare. At least let us solve the catchas it gives. Ideally hydrus should simulate a browser like firefox better.

100% the most critical issue now

> Tor integration.

Already has socks5 support, you shouldn't really even run tor on the same computer you access it from. At a minimum use the qubes or whatever model of VM's

> Having a tag service for subcription tags separate from "my tags" and enabled by default.

i guess?

> Be able to apply lossy compression for files. Possibly with an option to retain metadata like hashes.

get necked.

> Make floats be a part of the main window instead of spawning new ones. They bug out some WMs.

your shitass WM doesn't concern us, the pop out windows are part of why hydrus is good.

> Cache tag repo sync for a while before writing to disk. Currently it writes massive amounts of data to disk, taking a while and thrashing SSDs.

This would need to be an advanced option only for people with battery backups

> Split up the database files into services and services into chunks.

y

> Keep files in different directories based on predicates. (e.g. system:inbox or system:rating = 5/5)

directory = bad

> Purge a file or mapping from the db as if it never existed.

yes

> In the same vein as the previous one, journalling that you can revert at will.

eh, low priority. At risk of sounding like an asshole, just stop making mistakes. How many fucking times do you need to revert something???

> Option to download only thumbnails until user decides to download the whole file to save bandwidth on images you obviously don't care about.

actually a good idea

> Separate sessions for subscriptions with their own cookies.

yes

> Hope you can at least do the first two soon. Unique UA isn't doing us any favors and CF kills off use of a lot of boorus.

Agree 100%, CF integration is code-red priority. Literally everyone except a few shitlers (like 8kun lel) uses them.

>>14370

>I don't want to sperg over integrity if I can cut down 20mb files down to 1 without noticeable differences.

Compression can be done outside, there's literally no reason to have compression built in. I myself reencode some files whenever they're retardedly big.

I do agree that some local "receive tags from this hash" would be nice for duplicates. Obviously nothing that automatically petitions

tags to the PTR to prevent easy abuse and mistakes, but a "pull tags only" thing would be nice.

There's often duplicates that are the exact same without tags, and if you choose one over the other and another user uploads tags to the other via the PTR you're not able to get those tags without possibly unsafe DB fuckery.

>Do you not realize that even Qubes has a specialized WM that splits up the windows?

I honestly don't think it's a problem for anyone but you considering that's the first time I've heard it mentioned. Maybe make it optional, but even then there's a ton of more critical stuff.

Optionally this could be used as an excuse to make hydrus autistically modular, but that would require way too much work.

>Not your army.

How new?

>This is the same idea.

It isn't. Having folder naming that depends on the user completely fucks with many of hydrus' axioms. Plus considering you can easily tag pics accidentally it would cause many files to keep moving around with a lot of room for error, not to mention how it would work if you have multiple client_file directories. Considering you were just complaining about the number of r/w operations, that seems like a really stupid idea.

>>14432

>Compression can be done outside, there's literally no reason to have compression built in.

That's a poor argument. Doing it outside breaks metadata integrity. Doing it outside also involves more work in a different program when that's supposed to be managed by the db.

>I honestly don't think it's a problem for anyone but you considering that's the first time I've heard it mentioned. Maybe make it optional, but even then there's a ton of more critical stuff.

hydrus dev assisted with fixing WM interactions before. Moving the interface to the main window should not be an issue with Qt.

>How new?

Are you?

>Having folder naming that depends on the user completely fucks with many of hydrus' axioms.

You're still missing the point. The idea is to separate the main root folder into several roots with the same branching structure. You can already do a similar thing in migrate database.

>Plus considering you can easily tag pics accidentally it would cause many files to keep moving around with a lot of room for error

If you're prone to accidentally tagging pics then maybe you should use the feature sparingly. It's up to the user. Also I didn't say "tags", I said "predicates".

>Considering you were just complaining about the number of r/w operations, that seems like a really stupid idea.

Two completely irrelevant problems.

I asked about adding the third option to the pending PTR changes menu, in addition to the existing "Commit, Forget" ones - to migrate the changes to local tag service, when you neither want to publish your changes online (since they might make sense only in your local collection), but don't want to lose them either.

Now I have a bigger problem, somehow after working a lot in the duplicate filter I ended up with 800 pending changes to PTR. I don't know how to view them, but I'm sure they shouldn't be published, while I still need them to be available locally.

I assume this is due to the way the tag merging works when choosing a better picture in a pair. If I have a file that has local tags and a dupe of it that has PTR tags, depending on which one I choose as a better alternative, the local tags might get copied to PTR - and I don't need that.

So, the question is, would it be possible to configure Hydrus to avoid that sitution without losing the tags, or it would require a new feature in the program?

>>14441

>Doing it outside breaks metadata integrity

A different file is a different file.

>Doing it outside also involves more work in a different program when that's supposed to be managed by the db.

It isn't. Hydrus is a tool to tag files, not one to modify them.

Again, setting a file to receive tags from another hash seems like a good idea, but making the program treat two separate files as the same is a bad idea that opens a lot of room for errors and requires a lot of work that just isn't worth it.

>The idea is to separate the main root folder into several roots with the same branching structure

I guess that's not that bad, but moving existing files that much doesn't strike me as efficient.

hi, just a few suggestions for Hydrus to add in the next updates:

-there needs to be a on-click selection tool, at least make it doable in Hydrus so i don't have to click on every image to export separately

-is there a parser that can download off from private twitters? sounds a bit creep, but i follow a couple of locked twitter artists and TMD seems to have a problem downloading from private twitters. there's also some private twitters that are still pending for a follow, i still want to grab what's there. (especially since several artists are tryingt to move to pillowfort.)

-web.archive.org should get a parser too, sounds challenging but hopefully its something you can conquer.

>>14472

>-there needs to be a on-click selection tool, at least make it doable in Hydrus so i don't have to click on every image to export separately

What do you mean by that? If you're talking about multi-select you can already ctr+click and shift+click images.

>-is there a parser that can download off from private twitters?

You'd need access to the twitter account somehow. If your account is allowed to view their posts then you could probably throw the cookies in hydrus and it should work.

>>14447

>A different file is a different file.

Directly derived from the original, thus most operations still make sense on the derived file.

>It isn't. Hydrus is a tool to tag files, not one to modify them.

It's also a tool to make thumbnails and download galleries and so on. There are plenty of use cases.

>bad idea that opens a lot of room for errors

…which is why hydrus should handle it.

>requires a lot of work that just isn't worth it

At worst hydrus would keep two sets of metadata that could be requested for the same key file. It really is a simple change as long as you know how your way around the db.

>moving existing files that much doesn't strike me as efficient.

It's fine, it's not supposed to happen back and forth. It could be even deferred to maintenance if efficiency is a concern.

Since animated duplicates are still far off, how about a duplicate API? DupeGuru works well enough with gifs, so it's be nice to have a way to feed those into hydrus' pontential duplicates.

>>14495

>Directly derived from the original, thus most operations still make sense on the derived file.

They don't. A jpg saved as a png is a different file. A mp4 reencoded as a webm is a different file. A different file is a different file. It doesn't make sense for hydrus to suddenly give up everything it takes as granted for files and add clutter just because you're too lazy to move tags.

>It's also a tool to make thumbnails and download galleries and so on. There are plenty of use cases.

Both of those are related with being an image database. Converting files isn't.

>…which is why hydrus should handle it.

It opens rooms for errors in hydrus. If you want to convert files there are already safe alternatives like FFMPEG.

>At worst hydrus would keep two sets of metadata that could be requested for the same key file. It really is a simple change as long as you know how your way around the db.

But what if you want to convert it again, now it has three, then four and so on. Even if you only convert it once now everything is considerably slower, you can't take what were previously axioms as truths, you have additional operations that shouldn't make any sense because, for whatever reason, batch converting files outside the program and automatically moving tags isn't enough for you.

I constantly do operations that involve taking the hashes of tens of thousands of files and it takes a long time. Having to test whether they have alternative hashes of other files, having to test whether that one hash is the current hash, and many other operations that are fundamentally absurd is going to considerably impact those operations.

How about a button in the manage tags window to check the file's urls for tags again? In case it was downloaded from a booru and there have been more tags added since then.

I want to be able to export transcode to 4chan compatible webms

Ideally I don't want to keep multiple encodes of files depending on where I'd like to post it. I'd just like to keep original mp4s and have a simple menu to export to board friendly formats.

It would be good to have a better deduplication assistance, e.g. letting this evaluate image quality and only just confirming that it did correctly identify the worse images:

https://github.com/idealo/image-quality-assessment

It might also be good to have more powerful image similarity detection:

https://github.com/mkettune/elpips

Also the lossless JPEG recompressor ( https://brunsli.dev/ - code at https://github.com/google/brunsli ) that will also be part of the JPEG XL standard is so great that likely I will want to re-compress all JPEG with it. I assume hosters will see it the same way.

Is there some way to transmigrate checksums from the PTR so they can still apply to the recompressed files?

>>14619

>lossless JPEG recompressor

That's a converter, it turns jpg into jxl. It has the same problem as FLIF, HEIF, etc. in that literally nothing uses it. Webp barely has a presence and that's only because Google shilled the fuck out of it after making it. If you want a true JPEG recompressor, check out mozjpeg ( https://github.com/mozilla/mozjpeg ). That being said, JPEG XL is better at compression than even mozjpeg, so if Hydrus had JPEG XL support I would be interested.

I would also like some sort of way to take a file in the Hydrus database, compress or otherwise modify it, and move the metadata from the old file to the new one. How that is done is the problem, though. Maybe we could just have whatever external compressor output a tsv file of old to new hash pairs, then import that into Hydrus and have it migrate metadata. This would work well for local collections; no clue about PTR though, I don't use it.

>>14521

>It doesn't make sense for hydrus to suddenly give up everything it takes as granted for files and add clutter just because you're too lazy to move tags.

It doesn't have to give up anything, it just has to alias the file metadata. If you move the metadata to a modified file then you won't be able to receive any updates on it because it only applies to the original. Likewise, you wouldn't be able to update the original if you only touch the modified file. (e.g. PTR)

>If you want to convert files there are already safe alternatives like FFMPEG.

What do you think hydrus uses under the hood?

>Even if you only convert it once now everything is considerably slower

Would be O(1).

>Having to test whether they have alternative hashes of other files

Would take a trivial amount of time because it's a hash table.

>having to test whether that one hash is the current hash

Would be unnecessary if the file is addressed correctly.

Related to this sort of issue:

>>13431

I have files I download through other means, so I'm not using the Hydrus downloaders, but instead am using the "try to load tags from neighboring txt files" option to have my other download scripts fill in all the meta about the files. I can add a "url" into that TXT file as a tag, but that gets imported as a "tag" on the file, not a "url" object on the file. Some means to have those side-by-side TXT files also indicate the source URL(s) would be nice, or a "magic" tag that setting it actually sets a "url" object for the file.

Any way to have a parent/child/sibling system between posts similar to the one on Danbooru? It's a really good system for posts that are related to each other.

>>471

How about a "download anyway" option in the detailed file import window?

I have some tags that I blacklist because I usually dislike images tagged with them, but in some cases can stand.

It would be nice if I could right-click on a file that was ignored because of the blacklist and have a download anyway option that would force the download.

>>14267

>Option to download only thumbnails until user decides to download the whole file to save bandwidth on images you obviously don't care about.

I was about to suggest this. I have a habit of downloading several thousand files and then going through them all at once when I have the time. It would be nice if I could batch delete those fucking huge images with several dozen alternates without downloading every single one.

In the import folder settings there are some options for "when a file fails to import". Why is there no "Leave the file alone, try again later" option, only a "Leave the file alone, do not reattempt it"?

For an example where this becomes and issue, if a file is currently downloading to the folder Hydrus will mark it as "File is zero length!", then ignore it forever, even if it has been downloaded completely later. You have to go into the log and select try again to get it to import again.

It would be nice if there were the option to have persistent per-parser/site tag import options. This is a really shitty example, but you could do something like blocking furshit unless you were using a parser specifically for that kind of site.

>>15036

You can set tag import options by url class in network>downloaders>manage default tag import options

Unless I'm missing something, this seems to be missing:

I've been trying to figure out, is there no way to have stuff autotag per watched threads? Didn't see anything in the docs about this.

I know you can set tags to be automatically applied on the watcher, but can you not do it PER watched thread?

So if I add a watcher url, and want all files that get fetched from that to be auto-tagged with series:pokemon, then I add a second watcher url, and want all posts from that to only be tagged (automatically) with series:kingdom_hearts is there no way to do this?

If I am correct, I would love to see this. It's a complete PITA to have to do it manually for each thread for basic tags.

I'd also suggest this be done anywhere else it is not possible on the other download tabs, although I rarely use anything besides the watcher so not sure if they are missing this.

My other suggestions are:

Under manage tags, the recently added tags only save about 22, while there's an entire vertical space for it to save around 60. This means you constantly have to keep doing a search as you go through tags. Would be nice for this limit to be bumped up a lot higher.

And lastly, the ability to shift click items, or shift hold and drag to drag highlight items. When dealing with lots of thumbnails, this would save so much time. Currently it seems it's only possible to Ctrl + A to all, or ctrl click each individually.

Thanks

>>15091

If you set tag import options in the watcher page it is applied to the threads that are queried while those are set, so you could change it to have your tags for that thread then change it again for another thread.

You can change the amount of recent tags in options, options>tag suggestions>recent.

Shift clicking to select a thumbnail range should work already.

>>15094

>If you set tag import options in the watcher page it is applied to the threads that are queried while those are set

It would be very nice to have a prompt or option area to add a default tag if the user wishes. Then it just resets after the page is submitted. The user could also right click the watch and edit the tags and it would easily update them for all the previously grabbed files in that watch and any newly tagged files. The way you explain is a bit inefficient imo.

>You can change the amount of recent tags in options

Ah, a bit hidden, thank you.

>Shift clicking to select a thumbnail range

I was behind like 2 Hydrus updates, but just updated and that works now. Maybe it was a bug then. Works now though. Thanks.

Does Hydrus have any support for multiple audio/video streams in one file?

Say I have 4 videos with the same audio, just slightly different video (say different levels of clothing). You can use ffmpeg or what have you to combine them into one file with multiple video streams and one audio stream, which mpv can play (at least externally, cycling with the _ keybind).

You could do similar things for multiple videos which only differ in audio, say language (keybind #). You could even have both multiple video and audio streams in one file, if you really wanted.

I could have each video+audio stored separately in Hydrus, but to me they're the same video, just different variants. Being able to combine them would help reduce clutter in the search and lets you be sure just from the thumbnail that the variant you want is available.

I think it would be cool if Hydrus could switch video/audio streams without having to launch in an external program. Don't know the technical details on whether a keybind or right click menu or both would work, though.