/hydrus/ - Hydrus Network

>>12233

I kind of agree with this, some things just shouldn't all mix together. Doujins just add a whole another layer of tagging that just doesn't mix well with single images, videos, txt files, etc. All this stuff is really starting to clash with hydrus. Hydrus needs to be split into a different programs if you really want all that extra stuff or running multiple instances of Hydrus(which I'm currently doing for videos but mostly just testing it out).

I suggested this once but a separate version of Hydrus that's meant solely for doujins would be a pretty nice idea as some people might not want to use a webclients like happypandX or LANraragi. Managing doujins is a whole lot easier than single images so I doubt you'd need to do much work to manage a program like that.

Siblings handle some cases where you want to replace namespaces, but this doesn't work well for single images or long tags (which you have to retype).

I'd like an option to change the namespace for selected tags in one or more on the fly, e.g. change an unnamespaced tag to an actual name space, or replace one for another.

For example, I'm manually changing many creator: tags to blog:, tumblr: instagram: or whatever because the account owner is not the actual creator of the work. I can do this with some SQL-fu, but it woul be much easier to say, right-click a tag in the Manage Tags screen, select "change namespace", then type in the new namespace or select it from a dropdown.

It would be nice to have some kind of search-and-replace interface for tags globally when doing this kind of maintenance.

>>12235

>Managing doujins is a whole lot easier

I might be slightly biased towards this(I'm the dev of one of the webclients and it's not HPX), but doujin providers are much more annoying to scope for metadata than boorus. Alongside this, you also can run into encoding issues with the filenames inside of the archive.

Hydrus bypasses encoding problems by renaming files currently (and it's probably much better off thanks to that), but that wouldn't apply to files inside of a cbz.

I'd personally rather see support for more esoteric formats like xml/svg than manga, but that's just me wishing for stuff that'd fit my workflow more. With the original HP dead there's not much left in terms of desktop clients for doujins.

>>12218

lol well aware of those reasons, the bmp has eaten all my ram before stopping its attempts to render a few times.

Oh, one thing hdev

I have gone into this in length in the past with manga viewing/viewer version.

If you go down this road to actually doing a manga version before you go further would you be able to have a pause like this for input?

personally i'm of the mindset that you would have to 100% make all storage user parseable. with images, I have clumps of who gives an actual fuck what its named, so if hydrus dies tomorrow, I lose nothing, but as far as a manga reader goes, if hydrus dies, there goes my entire db/collection.

Personally, I think that it would be for the best to branch hyrus into 2 programs at that point, one for manga, one for images, which while having the same base codebase so they can both be worked on at the same time, they would function in how they archive and present things in 2 radically different ways.

>>12233

>>12234

>>12235

Hentai panda x is fucking garbage

Hydrus with its current tagging system and a public repository could tag every manga in existance and it would all come down automatically with little to no hassle.

because hydrus would take control of the hentai/manga itself, it would never run into the 'woopse, you moved a file, all the info is now gone' that every current option with tagging has,

it has nearly all the ui elements that I would need to replace acdsee as my manga reader

The only issue is its not user parseable. which would need to change, and why I propose a split into a manga version and an image version, using the same codebase, just different things get ticked for different versions, images would all be stored in hashes, manga would all be stored in group-author/manga/autor folders group-author for hentai/doujin manga for known manga and author for oneshots.

This way if hydrus fucks itself, it has saved everything in a non destructive method and likely saved it in a way that would put anything we currently do to shame

every few months I get so pissed at acdsee that I go out looking to see if ANYTHING can replace it, and long story short, nothing can, but hydrus is SO fucking close to being able to do it that I would love to see it try, most cbr/comic readers fail miserably because they rely on outside programs to do something, usually acting as a browser, and this sucks because I cant one hand read, what I mean is with acdsee I get to a manga I want to read

enter, down, up, enter

i'm now reading the manga

page down read till done

enter, backpage im out

down and i'm on the next chapter

repeat above and i'm now reading the next chapter, modify for manga of choices distribution method.

The closest replacement I have found is comicrack, but the whole system is near perfect, yet a few steps away from perfect at the same time. I have to use the library in program to save any data and I can't modify files on hdd through library so quick removal of shit is not possible, I also can't do easy dup searches and have to rely on things being named correctly. so shit is very suboptimal from a archive management perspective, the way I have it set up it wont straight up loose info I add to it if a file moves, but i'm not able to even attempt to file manage anything anymore or else shit will get fucked.

I'm honestly using this just for porn as its a far better way to do it then acdsee at this point, and hentai panda x cant handle a 45000 file archive without shitting itself horribly. comic rack, while its dead, does a good enough job as an intermediary between acdsee isn't good for porn, and when hydrus adds manga to its functions. the worst that will happen is I either need to manually redo everything I do in this program, which honestly may be fairly simple, I don't plan on going overboard, due to the last time I did that with a tagging program, I lost 10k tags because I thought they were file based and moved with files, not program library based.

Have an idea for a duplicate auto mode.

going through the dup finder, I am encountering quite a few jpeg images that were converted to png at some point, the png being 3-5 times the file size for the exact same image. would there be a way to check for duplicates that are jpeg and png, then auto pair against each other to see if they are literally the exact same quality, or within a tolerance where the increased size does not make up for the extrem minor gain in quality if there is even a gain at all?

>>12243

Just make a separate database if you don't want to mix manga with images

>>12243

>>12249

Be sure to check https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/issues/70

Personal opinion: It is better for Hydrus to have a rewrite in Qt before we should have complete comic support.

>>12272

human-oriented sorting is a problem.

Also > duplicate thumbs

>>12272

https://github.com/SethMMorton/natsort there we have it, HyDev take note.

>>12272

Also if you want a fast implementation https://github.com/sourcefrog/natsort

>>12247

Yes, that's basically something like these: https://github.com/andrewekhalel/sewar or even https://github.com/lidq92/CNNIQA or https://github.com/lidq92/CNNIQAplusplus … and so on.

Having these available in Hydrus' duplicate filter (or its revised version) should help a lot.

That said, you should only expect imperfect reliability in fully automatic mode. You won't ALWAYS get very accurate scoring or even just be able to identify the "better" image automatically. These also generally can misidentify variant images as "the same but worse", and other mistakes like that.

>>12152

Like the other anon implicitly did, I also propose the ability to run some more of these image quality / image similarity metrics as a possible feature. Would be nice to have more options to populate the filter and assisted automatic scoring from within the duplicate filter.

At he same time, it would probably be a good idea to make the duplicate filter more modal. E.g. "this is above the certainty threshold you set up, so here's how the weighted scoring of the algorithms you picked would solve this and the corresponding scores - confirm?". Probably also needs some thought put into a colored multi-selection GUI thing that makes it quick to see the scoring & resolution to manually deviate and fix mistakes.

I don't know if this is a feature already or not but a sibling thing that could have multiple dependencies so that it could go like

>if [arist] + [ocname] then swap to artist:[artist] + character:[ocname](artist)

would be very convenient, expecially for scrapping tags from places like furaffinity where people don't use underscoring properly

On the subject of character name.

Would there be a way, far in the future to link 2 tag together.

Instead of:

character:character_(series)

use:

[[character]]x[[series]]

Where x is a dynamic taglink action.

And while we're at it, instead of namespaces, multiple list of tagtype a tag can be presented as for that specific media.

Tear me up, it's brainstorming more than a definitive solution. I don't even know if it is possible.

>>12277

Never trust Neural Network systems verbatim. Always find an expert system that can work well first.

>>12282

>>12277

true, it wouldnt be perfect, but right now I have to parse between two images and when the quality is close, what could take 1-5 seconds takes 10-20 seconds.

if we had dup tiers, like the current one is a blunt and stupid shit looks close, that is perfect, sadly I don't have any examples on had as I deleted them, but there was a scalie thread on /trash/ where some jackass came in, made the shittiest 'corrupt' images that were 7-8 times the good images file size, in a thumbnail looked passable most of the time, but full size, it was unrecognizable. if a dup filter was more accurate, these duplicates would get overlooked, what I would like is this

current duptetector as a base line

from this a more stringent dup detector to check the base lines work. this should filter out alternative images from normal dups

and from there a far more stringent one that would do something like jpeg to png comparisons, and if they are close enough to trigger here, in nearly all cases the jpeg was at some point converted to png, so in this case, scrapping the png would be done as it would save space without needing to go though with a fine comb by hand,

with that final level, imagine that the two images are just shown, right click is keep png, left click is keep jpeg, and due to how close they are you almost never click png. at least this is how i imagine it, dups going through 2 dup filters and then a third png to jpeg filter would turn the 10-20+ second checks into 1-5 second confirmations.

>>12285

I will add more information to the Optimization thread on the list of expert system vs NN repos

>>12286

Personally I preferred to KISS and just suggested a few python frameworks that might be easy to hack into Hydrus - but sure.

- parser revisioning

- remote fetching

- version tracking

>>12152

I'm not sure how big this is but here is a suggestion: implement an icon and search command for files which has any notes attached to them.

A probably small thing that would help me a lot would be an "delete both" button in the duplicate filter - i know that i can press del but i usually dont have a hand at the keyboard while filtering and then i also still need to press delete both anyway…

>>12232

PeTR

Public

Emacs???

Tag

Repository

- Cookie management from API

- Tag statistics from API so if you search for a_totally_sfw_tag, and it produces a lot of creator:cname , you know you should sub to cname

>>12298

Public Integrative Tag Repo

Public Incorporative Tag Repo

Public Interdependent Tag Repo

Public Ingrained Tag Repo

(PITR)

>>12166

You might wanna get https://github.com/deanmalmgren/textract (this is some good stuff for text-like documents)

>>12230

Thanks for clarifying. Unfortunately, this is not trivial to do for hydrus. Any paged system needs sorting, and hydrus supports many clever kinds of sort, so in order to implement this, I would need to load 'media' metadata for every file in a search result before I could fetch the first page of results. This would not save much time from the current system, where most of a search delay is in fetching that same media metadata. The proper solution, and I imagine how the boorus probably do it, is by having a sort cache (and they have page caches as well, and generally simpler searches to cache), to cross-reference search results against to figure out page slices. This is more complicated than I want to make hydrus search code at the moment. I am happy with being able to display and manage thousands of results at once in the main gui, and I also don't want to further complicate the viewer with paged management and load code. As you say, I encourage users to add 'system:limit=x' if they want less laggy searches.

I would be interested in your further thoughts if you have certain scenarios where search is very slow. If there are particular instances where the client runs very slow for you, I'd love to help it run faster.

>>12213

IPFS repos would be important, also an advanced API that can trade IPFS hashes and images would be sweet

>>12234

>>12233

>>12235

>>12243

>>12245

>>12248

Yeah, I am mixed on cbz. I like the idea in the sense of waving a magic wand and having great support, but I can't do that and I know I fall to feature creep too easily. If this is voted on, I would try to make very simple support and see how that goes, and then iterate on it in future if it proves popular. I can't out-compete the programs already out there, but I can do some simple stuff, and ancilliary code like navigating multi-page single-file media will have uses for things like file alternates.

I really want all future big jobs to be small improvements and experiments, ideally 6-8 weeks and pref no more 12, so I don't get bogged down like the downloader engine overhaul. I am open to experiments that fail and don't want to get emotionally attached or fall into sunken cost fallacy.

>>12236

Thanks. Yeah, I would like easier sibling workflow, including from the right-click menus, as part of a tag sibling/parent improvement. I would push in this direction with "Improve tag siblings/parents and tag 'censorship'".

>>12240

Yeah, your dialog mock-up is exactly the sort of thing I was thinking of. I'll have a new options panel somewhere that turns on the advanced mode of this dialog and let you set up some favourite reasons and custom entry. Now that I have the 'set a reason' infrastructure in place, this will not be super difficult to add. I expect to have it in in the next few weeks.

>>12247

>>12277

Yeah, my first push here will be to set up a system that permits auto-decisions in a sensible and generic way, along with user ability to control what is permitted, and then in future hang new auto-decision systems on it. The 'this is a png copy of a jpg' seems like a nice simple way to start, and I know I can do very quick detection of that by just hashing image pixels. Then maybe explore some 'this jpg is definitely lower quality than this one of same resolution' stuff. The way to slice through dupe mountain will be through automatic systems to reduce the human drudgework, but I am similarly leery >>12282 >>12283 of anything too clever/vapourware to start with. Most of all I want to get the infrastructure and maintenance processing code in, and then we can test all kinds of different comparison systems for our exact purposes.

>>12272

>>12273

>>12274

>>12275

Thanks. I have 'human' number sort capability in hydrus already, although I am sure there are still places to apply it. All numbered tags should sort like this atm. I am pretty confident I can get directory listings and file access of rars and zips with the python libraries I already have. Any first version of a cbz viewer would be simple and just read through the internal pages one by one, no bookmarks or per-page metadata or anything. Just something that lets you penetrate the 'list of numbered jpg' zips already in your db in the media viewer (and rename to .cbz or whatever, so you can 'open externally' to your preferred comic reader).

>>12277

For new image recognition techniques, yeah, I designed the search system to make this possible. Much like >>12319 , the main push of duplicate detection 1.0 was to build a search system that could handle many search systems and hang one simple 'looks like' system on it. I can fairly easily add new techniques to support rotation or colour similarity or whatever on it now. This would not be my urge at the moment, as this simple system we already have the biggest problem is there are way too many to go through, so the processing workflow is now the weakest link, but once we have that more under control I can work on this.

>>12278

Yeah, that's a tricky one. I know exactly what you are talking about, and I would love to have a nice system for it, but the actual guts of how 'if … then' tag relations would work are way more complicated than I am confident I can currently support. For siblings and parents, my first priority is to improve the data store behind the whole system first. Once that isn't on fire behind the scenes, I'll consider carefully adding this sort of power. I am sure it could go very wrong if not thought about, so it'll be baby steps until we have some real world experience.

>>12281

Both of these thoughts are on my mind. Adding tag siblings revealed to me a big set of pain in the ass problems related to tag definitions I had not considered before.

For both of these, I think the ultimate far future way to solve them is to have a tag definition structure "Add tag metadata (private sort order, presentation options, tag description/wiki support)", where clever metadata can be applied to tags.

So you could say:

character:shimakaze (kantai collection)

And the tag definition, which would essentially be a cleverer iteration of the current siblings and parent system, would say "this has 'series:kantai collection' parent" and also perhaps "this can be displayed to the user as 'character:shimakaze'" without destroying the unique tag identifier through merging with some 'character:shimakaze (my oc series, donut steel)', basically being aware of the (kantai collection) after the main tag. I am very much on the side of letting users display tags how they want, as there are many different spergy desires here, and having a system that recognises info about tags lets us do mass management rather than the current per-tag mess and endless firefight.

Same for namespaces. I am experimenting with 'clothing:' namespace on the PTR, but I know some users hate that. It would be ideally better if the tag 'bikini' had the 'property' of "clothing" rather than an explicit namespace to argue over, and then a user could say 'when a tag has "clothing" property, display it as namespace'.

As it is, I expect my next step here will be more in line with little patches. Namespace sibling control (like saying 'display all creator: tags as artist: please' or 'display all clothing: as unnamespaced') seems an easy-ish next step.

"Tags were a mistake." - t. hydrus_dev

>>12288

Thank you, I am adding 'Add version tracking to downloader system objects and explore remote fetching of updates' to the list.

>>12296

Thanks, this is actually a small thing. I assume you want no duplicate action applied, just a basic 'get rid of these two shits' and move on to the next decision? I'll see if I can add a button to quickly do this for 349 or 350.

>>12299

Thank you, I have api cookie management in my current to-do. I won't work on client api as a big job in this cycle just so it has some time to breathe.

Can you explain the 'tag statistics' idea a bit more? Could this be something to apply to the program more generally, rather than just the API, like something to click that says "show me what artists I like and do not sub to?"

Subs need a db-level data overhaul before I can do clever inspection about them btw. But this is something else to integrate into the client api as well–managing subs.

>>12293

Thank you. There are multiple jobs for improving notes that I have not been able to get to. I am adding "Expand file notes system" to the list to cover a general push in this direction. This would include multiple notes and likely connecting the notes system to the downloader as well.

>>12325

Yes, thank you very much!

>>12316

sadly the only program that you would be up against is comic rack and acdsee in terms of full featured program and in terms of comic rack, nearly every feature they have, you also have, and in terms of acdsee you either use crash prone versions from 10+ years ago (version 8/first pro to version 9) or you use the recent versions which are pure bloat for unicode support, there is very little middle ground with acdsee due to so many of the in between versions fucking with features.

>>12319

I honestly never want an automatic sort, as much as it would be good for going though my cluster fuck, I am still getting images that are full featured images going against 23 byte black boxes. running into this makes a pure auto system unacceptable to me, I would accept an auto system that looks at two images, and then has me spot check

lets say it finds a better worse pair,

it presents me the better and it presents me the wrose

green border is better, red border is worse

I scroll though like normal, but left click confirms green, right click confirms red.

Confirmed green acts like better worse does currently,

confirmed red takes it out of the auto figure it out area and into a manual pick.

Are there any plans at all to (at least have the option to) keep metadata of individual tag mappings, like a file's tag history or keeping track of which process added this tag to this file?

Like was it typed manually, was it imported while scraping, etc.

I assume this would bloat the DB by a big factor but at least for my personal use I think that might be worth it down the line

Poll >>12358 !