/hydrus/ - Hydrus Network

>>10500

>>10505

Thank you. I am adding this to the list now as "Improve file lookup scripts and add mass auto-lookup".

>>10524

>>10525

I will give my top seven

Ebook and Comic support

- pdf/epub/mobi/djvu/chm MIME, maybe Office-related MIME

- https://github.com/adolfosilva/libgen.py for a start, maybe https://github.com/evilhero/mylar and https://github.com/Xonshiz/comic-dl

Text and HTML support

- https://github.com/JimmXinu/FanFicFare (both ebook and html form) and maybe some more in >>9927

- standard wget or curl for direct web archiving, because why not?

- HTML and CSS combined/separate option (maybe useful for similar themes)

- Twitter Tweets, Tumblr post, *chan posts etc. support as text/HTML downloader scripts

Fuzzy searching for tags, images and music

- >>10232 and >>10272 (tag fuzzy search using phonetic hashing)

- >>10290 (audio fingerprinting and deduplication similar to audio player/managers)

- >>8320 (dupe automation, but with better algorithms to detect JPEG quality)

- >>9281 and >>10361 (advanced image de-dup algorithms and OpenCV3)

- Further IQDB coverage with other sites not included in the official site e.g. e621

Better tag management

- tag cloud and clustering with statistics on relatedness (possible tag parents/siblings)

- Tag translation and dictionary (*booru and dA English tags <=> Pixiv's japanese tags)

- Tag auto-converter for between-booru compatibility (e.g. episodes for derpibooru, extra meta for danbooru)

- Derpibooru-level sophistication on tag logic e.g. AND/OR/NOT and hotkeys (https://derpibooru.org/search/syntax)

Multi-processing, Multthreading andmultiple desktps

- Allowing parallel downloads from different sites or servers to waste less time

- Assuming someone has multiple desktop and IPs, create a standard protocol to delegate slave IPs to download certain websites to offset load or obfuscate traffic, and send back the scraped URL, tags and files back to the master IP

- Possibly create a standard export format using USB for moving files from slave to master in case connection is down

Video and audio support (not much to ask using youtube-dl)

- >>3665 (audio and video metadata)

- Youtube/"Alt-tubes", Soundcloud/Bandcamp subscription and downloader (Bonus: Torrent2Hydrus)

- More MIMEs for different formats (since they all play in VLC/MPV)

API building

- >>10203 (standard thumbnail sizes, requested by some Mobile UI devs)

- >>10168 and >>10199 and >>10200 and >>10201 (API support for mobile)

- Better IPFS >>7405 (fixing IPFS "nocopy") or Dat Protocol with >>10364

Others (ML related)

- Illustration to Vector support for machine learning based image tagging

- >>10047 (image converter support for Waifu2x and DeepCreamPie)

Others (download related)

- MOAR manga sites >>8971

- Further support for yiff.party, fantia.jp and enty.jp (you know the kind of websites these three are)

Others (discovery related)

- artist gallery/favorites from dA/Pixiv => tag/description pattern => recommended keywords

- artists favorites download from dA pixiv => favorited artists pattern => recommended other artists

Others (pet peeves)

- >>10102 (PTR clean up and standardization)

- >>10062 (misc. UI changes and hinting)

- >>7596 (image square comments and scraping)

- >>10450 (proper documentation on how the code works)

>>10510

I think this is what I am most excited about myself. I hold a small candle for increasingly clever machines, even though they aren't amazing yet. The big tech companies, software libraries, and GPU drivers are all gearing up for this tech, so I figure it is about time. I am also enthusiastic about the large tag collection we have generated. We have one of the largest combined meme/elf_tiddy databases ever assembled, so I think we can do some interesting training.

>>10497

No, my mind works this way

I deleted something? why was it deleted?

Open link in new window

Holy shit, why in the fuck did I ever delete that?

redownload.

Or in a more realistic sense to what has been deleted so far

I see its deleted, I check it

Oh it's a meme of low quality

And I piss the time it takes to check it away.

Or

I see a good 20 images are deleted, what could this have been

Open links

OH FUCK YOU its that god damn baby fur sonic thing that some asshole decided would be funny to post here

Or

currently I went through 300 images in dup detector and about 1000 in program before I found out this was going to be an issue for me, so I have about 1300 images that are good enough to be in the archive, that are not in the archive, that I stumble on every now and then, and because of the 1000 I got rid of before I knew it was a problem for me, I have no idea if they were duplicates or if they were mistakes.

Ok I went away for a bit and came back so train of thought has left me behind, point is, the current way hydrus works makes me second guess why things were deleted, along with if I bring a duplicate back in, it will never show that duplicate in a filter again. as it's considered a known pair.

Notes like this will be helpful for more than one application, but mine is at the very least most relevant to my use case.

>- Reduce ui jitter and hanging by improving ui-db async code

Voting for this in the hope that it might make Fatal IO Error 11 go away, or less common.

>- Add Ugoira support (including optional mp4/webm conversion)

>- Add import any file support (giving it 'unknown' mime)

Well, I was the guy pushing for native ugoira support, so of course this needs to be here.

If ugoira gets low priority or takes a while to implement, it would be nice to get unknown mime support asap, so that we can at least hoard ugoiras already, in case an artist's autism kicks in and he deletes all his pixiv posts.

Unknown mime would also be great for all the psd files I have.

>>10429

>- Increase thumbnail size limit (currently 200x200)

You could let the user specify thumbnail size, would also eliminate the need for 2 thumbnails per file.

Generally UI improvements will benefit everyone so i think the focus should be on that first.

Anything that improves the usability of the program like tag sorting, shortcut/interface customization, boundboxing, undos, etc. should be near the top of the list.

Audio, CBZs, pdfs and what not already have very good programs so these should be low priority. Same goes for general file support. Focus on making Hydrus better at what it was intended - image management and viewing. The rest can come later.

>-Reduce ui jitter and hanging by improving ui-db async code

I have to keep the PTR sync paused or else, when it starts, Hydrus stops responding even for days. During the time it hangs it keeps reading and writing to disk as shown in the task manager, but with the ui freezed I have no way to stop the sync safely and I'm forced to terminate the Hydrus process thus losing whatever progress it might have done.

After having the the ui not freeze looking at a

>>10102 (PTR clean up and standardization)

like already mentioned a few time would be nice.

>- Explore a prototype for neural net auto-tagging

>- Add an interface for waifu2x and other file converters/processors

I would also like to see an option, that the user can check, that lets Hydrus automaticaly convert and/or compress archived images (I'm only talking lossless here).

Like passing all png images through pngoptimizer (http://psydk.org/pngoptimizer).

OR searching, pretty please with a cherry on top.

Building custom queries with OR would drastically enhance the way I can review my collection.

Overall, what it would let me do is build queries based on more subjective criteria than tags alone can offer. I could make a search for artists who do a particularly cute moe style, or I could come up with a group of hot readhead anime characters, which I've always wanted to do.

If you added OR searching, it would definitely become the number one thing I did with Hydrus. This is my dream.

It does about everything I want currently after editing some of the scripts to support my autism (thanks for the help in email), for me the only major extra things I'd love is:

- Add Ugoira support (including optional mp4/webm conversion)

[because I save way too much shitty art, and there's countless ugoiras not preconverted on a booru]

- Add OR file search logic

[i.e. search mystery character with blonde hair, or perhaps its tagged as light brown or even brown/orange]

>>10528

>>9142 here, I'm still willing to build a library / API for i2v if it would help. I can also help design a system that would allow for training custom models, which is a much more involved problem, but would probably produce better results. Again, no pressure if you're not ready.>>10528

I am renaming the "Reduce ui jitter and hanging by improving ui-db async code" to "Reduce crashes and ui jitter and hanging by improving ui-db async code" to represent the linux stability side of this job.

I am renaming "Decrease tag sibling/parent jank and improve tag censorship" to "Improve tag siblings/parents and tag censorship" to represent general sibling/parent improvements beyond making it less shit.

I am adding "Cleanup code and improve programming practises".

>>10532

As an aside, as zips are supported in a limited way, ugoiras that have straight zip URLs, like how I think the default danbooru downloader gets them, are just imported as blind zips right now. Part of adding ugoira will be writing some further file parsing to do 'does this zip look like an ugoira m8?' and then reparsing all zips in the db retroactively.

>>10539

For now, I recommend you go options->maintenance and processing and turn off idle time work completely. Only do big work on shutdown, and limit it to, say, 10 minutes. This will make repo processing much more manageable.

Yeah, I'd love to have (optional) png optimisation. In the coming years, as the client slowly moves away from hashes being important, these local/personal improvement tools will make more and more sense.

>>10541

I can add FLIF in easy short work as soon as PIL or OpenCV add support, which I don't think they have done yet. Or, if someone can point me to a good, non-meme pypi FLIF library that can do some version of GetResolutionAndOtherMetashit( path ) and numpy_array = GetRGBPixels( image ).

As long as someone else does the decoding work, it is only about twenty lines of work on my end.

>>10544

Thank you. We'll see how this vote shakes out, but either way, if you are still keen to do work on something like this, I'd love to outsource the expert pain-in-the-ass part so I can focus on building a workflow in the hydrus ui. I'm still a sperg about collaborating, but any sort of library that made parts of this easy-peasy would be very welcome.

I guess we are probably talking two(?) components:

1) Given a model, what tags are suggested for this image?

2) Given tagged images and maybe some human interaction, how to make a model?

Although I presume we are also talking about some shared interface layer and whatever else is needed.

Since we already have i2v model, if you made a library that did the grunt work of 1, I could probably integrate that into a new column in the tag suggestions stuff in regular weekly work. 2 would need to be in 'big work' and more emails/posts back and forth to figure out what workflow and calls the library would need.

I don't know much about this, so any thoughts you have on making this stuff real are welcome.

>>10548

>[…] just imported as blind zips right now.

Can you please enable ugoira download for the pixiv downloader? That way we could already start hoarding properly.

All the current ugoiras have the animation.json file, which starts with the key "ugokuIllustData", so that would be the 100% accurate way for recognition.

Though some people might have older ugoiras that only have the 6-digit numbered jpgs or pngs in the zip file. I guess those might be confused with zip files containing comics, so it might be good to have a way to manually change the handling (animation/book) for those. Since old ugoiras don't have any frame duration information included in the zip, being able to set the frame rate manually would be good in that case.

>>10552

This is a tough one. I have just looked at the problem again. The link we are currently using to get pixiv metadata is this:

https://www.pixiv.net/touch/ajax/illust/details?illust_id=71528360

(for page https://www.pixiv.net/member_illust.php?mode=medium&illust_id=71528360, which is a recent post)

It provides a JSON-less zip:

https://i.pximg.net/img-zip-ugoira/img/2018/11/06/05/48/45/71528360_ugoira600x600.zip

With the frame timings embedded in the API JSON. My new downloader isn't clever enough to synthesise new files from multiple sources of data, so grabbing the zip and inserting some frame timing json up would require a more significant add-on, which I would expect to write in adding ugoira support. It isn't something I can do quick.

As it happens, I was looking at how danbooru do ugoiras, and the couple ugoira zips I downloaded from them didn't have frame timing JSON in the zip either. I wonder if they are just pulling the zip file and using some flat 25ms or something for their webm conversion?

Am I talking rubbish here? Do some pixiv zip links have the animation.json in them, and I just missed them? Do pixiv ugoira pages link to different zips anywhere, and the API is just using different stuff?

>>10560

hey goy. is there support for choosing UI font and fontsize? if not then will you add?

ty

>>10551

Alright, I'll try to infodump what each of those would require.

i2v is a multilabel classifier – you can give it an image and it will give you the confidence for a bunch of tags (1539 of them, examples 'yuu-gi-ou zexal', 'tokyo ghoul', 'kin-iro mosaic', 'safe')

The other kind of model is a binary classifier – it only gives you one tag at a time.

Either way, you feed it an image and get back a number from 0 to 1 for each tag, and you get to decide what's the cutoff.

The model itself is stored in a large-ish file for the weights. For example, the weight file for i2v is 180 MB and doesn't compress much. This isn't tiny, but it's on the small side compared to some more powerful models. Loading the model takes about 0.8s on my machine, classifying one image takes about 0.33s.

The steps to build a model from scratch are:

>Decide on the architecture

This includes describing the various layers, and deciding how many tags you want to look for.

>Gather training data

The amount of data you need depends on how "simple" the tag you want to find is, and how similar are images with / without the tag. A few hundred images is probably enough to train some easy tags, a few thousand should be able to handle harder ones.

>Run training

This involves letting your computer run full blast for a bit while it does a bunch of linear algebra on the images. GPUs make this much faster. It depends on the amount of data we use, but I'd expect most models worth training to take an hour of GPU, or maybe 10 hours of CPU (very rough estimate).

There are tricks you can do to let everyone help out training a single massive model, but that's a technical and logistical nightmare.

There's a trick you can do called "transfer learning" which lets you piggyback off a model you already have. It might be possible to use this to add tags to i2v that aren't in the basic list. This would produce a small model (that still require the larger one to work) and would take less time to train, but it's limited to things that are similar to what i2v was trained on originally.

>>10551

For case 1, I've got a pretty basic file that runs i2v. Loading code, tag list, weight file are at

>https://github.com/antonpaquin/Hydrus-Autotagging/blob/master/illust2vec-flask/illust2vec.py

>https://github.com/antonpaquin/Hydrus-Autotagging/blob/master/illust2vec-flask/tag_list.json

>https://github.com/antonpaquin/Hydrus-Autotagging/releases/download/0.1/illust2vec.h5

This will take a PIL image in and give you a dict of {"tag": score} out. This is probably enough to power the first component, and you can probably reverse engineer enough to not have to use my code at all.

One possible way to handle case 2: I could build a thing that takes N images with a tag, and N images without the tag, and builds a classifier for that. There's a lot of potential for change here, but I think that's the simplest form.

>>10429

Maybe an overhaul on tutorals in the help section on the hydrusnetwork site would be my only request, there's a lot to learn about the various features in hydrus that just isn't there at the moment. I don't know, maybe let other people contribute their own tutorals if you're too busy and all.

>>10560

That's a bummer.

I used the Px Downloader add-on to download the ones that include the json: https://rndomhack.com/2016/01/15/px-downloader/

I wonder if it actually re-packs them? I made sure to disable ugoira conversion in it's settings, which is why I was sure it wouldn't change the original file.

I will dig around some more and see if I can find more info.

>- Add import any file support (giving it 'unknown' mime)

Absence of this (and the ability to store original file name) is the main reason why I haven't considered moving to Hydrus just yet.

>>10569

you do know that you can add any arbitrary namespace right? so filename:<name> is possible. people are doing this.

Improve the client's local booru, atleast tag search

Prototype a client api for external scripts/programs to access

>>10457

Is there any way to keep tags out of the PTR if you're using it. Or is there any way to make sure you aren't committing to it?

>>10572

you have to set up the ptr if you want it, hydrus does not come with it preinstalled, so just don't install it. Also if you do set it up you have to approve tag uploads, so you can be sure your tags stay your own.

>>10541

I keep all my thumbs on an nvme ssd along with the database, much rather have it this way then the hdd getting hammered looking for a few hundred images to generate thumbs for. that said if flif compresses the thumbs better than jpeg, that would be greatly appreciated.

——————-

Ok hdev I remembered something that was brought up a while ago that needs to be improved.

the duplicate detector has to have either a mode or a setting that lets you see already known pairs. one problem we discovered a while back was if you import a duplicate, do the dup detector and then it somehow gets re imported, you will never be told the duplicate is back.

Then there is also something I asked about a while ago with duplicates, one being a 'contender mode' and one being 'prefered alternate'

contender more is simple.

you have an image, you determined it is the better of two images, this one and all its other potential dups get taken out of normal duplicate processing and pushed to a second one, because you now have a known better image. this way you could quick filter all the contender images, getting rid of all the images that are lower resolution or file size, and only needing to go though the ones that are potentially better rather then all potential candidates.

Im not so much thinking this will be used in the first go around with dup processing, but every subsequent one, its a good bet that you will use it to weed out the junk.

Now the reason a filter in and of itself is not good, is simply because resolution or file size of an unknown image is not a good way to determine if the file is good. some faggot on 4chan hated a thread so he bloated every image out and made them fuzzy and un desirable for weeks/months trying to kill a thread, and his work comes up time and time again in dups for images, just looking at file size or resolution would save that shit and remove the good image. Contender mode would get rid of that because there is already a known good version of the image, and you are only looking at higher resolution version, or higher file size versions.

and finally 'prefer alternate'

It doesnt need to be a mapped choice, it just needs to be a choice.

I have several artists who I like who decided to make 20 10mb images that are all the same just small changes, and of them I may want to keep 2 or 3 images, so a perfer alternate option would allow me to mark one for deletion while knowing I don't have it, but I do have a similar one I liked more. this is kind of a useless thing for most people but would be helpful for me along with the "Add an optional system to record why files are being deleted" system.

>>10541

Didn't FLIF died or something? I think they stopped working on it.

>>10592

No, the faggots are at gitter talking about how to get it to the mainstream when BPG/HEIF beat them fair and square for compatibility.

The moment when you read the help about hydrus being able to control and manage files without importing but still doesn't understand a word

How you do this , I mean how you tell hydrus to manage those files WITHOUT importing ?

also how you can make hydrus make a subscribtion or follow a tag on image boards on general to automatically import new images from internet ?

>>10596

>[…]BPG/HEIF beat them fair and square for compatibility.

Maybe compatibility for hardware decoding, but not software compatibility, which is far more important.

No big websites will want to support these formats, because they're containers for HEVC intra frames, which is a licensing/patent nightmare.

Google developed their own codecs to avoid patent fees, so they will probably not want these formats supported in Chrome.

Same for Mozilla, who are pushing for the AV1 codec, so they would probably add support for AVIF long before they cave in to support HEVC based image formats.

FLIF meanwhile doesn't require licensing and doesn't appear to cause any patent conflicts so far. So I wouldn't call it dead yet.

>>10561

At the moment, it should pull whatever your OS defaults are, I think for both font and size. I don't set anything specifically atm, afaik.

I am not a big fan of themes and making things pretty (as you can probably tell!), so I struggle to revisit ui to neaten it up once I get bare functionality going. I am not against the idea of adding font customisation, but I think I would have to do a bunch of code and ui cleanup first.

>>10601

> FLIF meanwhile doesn't require licensing and doesn't appear to cause any patent conflicts so far.

This time not licensing will make it loose competitive edge, soon HEVC will be standard, and Google forcing WebP, Firefox forcing APNG like they normally do… FLIF is toast.

Requesting coverage for most sites in https://theporndude.com/hentai-porn-sites (some sites require JS like Hitomi)

>>10601

>>10603

For reference remember #gitter_FLIF-hub=2FFLIF:matrix.org

>>10562

Thank you, this is great. I've copied it into my ML masterjob.

>>10565

Yeah, if other people would like to write their own tutorials for anything in text or html, I am very happy to link to it or host it on the github.

>>10572

Not really. I'd like to add some tag filters to exclude bad tags at the server end ('banned artist' and 'url:' garbage) and allow for 'I only want 'creator:' tags' at the client end. And some repo recycling/cleaning to clear out some of the cluttered master records and reduce dbshit.

Could you make python3 a choice? just for the python features and the poor weird path people?

>>10575

Thank you, this is interesting. There's a lot I would like to do with the duplicate filter and system generally, especially ui to show/browse/review found duplicate relationships. First I will have to clean up the db side of things. I want to move from the current pair-hell to ordered groups that will allow for neater 'this is the best one' actions. This new structure will also work for siblings and parents, btw, which is not a dissimilar problem to deal with.

>>10598

You cannot manage files without importing. Sorry for my bad wording!

My new downloader help is here:

https://hydrusnetwork.github.io/hydrus/help/getting_started_downloading.html

The subscription help is out of date, but I hope to improve it in the coming weeks.

If you can't understand what I've written, let me know and I'll see if I can reword it. Feel free to email me or grab me on the discord if you want to work one on one.

>>10608

I hope to convert to python3 over this holiday. I will stop putting out releases starting on the 12th December and hope to have it done in four weeks. I will start working on the result of this thread's poll first thing in the new year.

>>10566

>>10560

Okay, it seems very likely that PX Downloader re-packs the zip. It seems to be closed source, so I couldn't confirm, but I had a close look at what pixiv fetches when playing ugoira and a file with included animation.json doesn't exist there.

The URL you get from that meta data doesn't get you the high quality version.

I found that there is a second .json specifically for ugoira meta data.

Example ugoira:

https://www.pixiv.net/member_illust.php?mode=medium&illust_id=48731415

Ugoira meta:

https://www.pixiv.net/ajax/illust/48731415/ugoira_meta

This one lists 2 URLs, "src" and "originalSrc", to get the bigger version.

Though of course my findings don't make archiving these easier. What would be the best way to preserve the originals? I asked for preserving the original files because I assumed that pixiv fixed their format to include meta data in the zip, but that's not the case after all.

If we re-pack the zip file to include meta data, we get the problem of changed hashes that I wanted to prevent by archiving originals.

One idea I had was to mux jpgs into an mkv file as mjpg. That way the frame timings can be saved and the images are not re-encoded.

ffmpeg -framerate 30 -i %06d.jpg -codec copy mjpg.mkv

mkvmerge -o ugoira.mkv -d 0 -A –timestamps "0:timestamps.txt" mjpg.mkv

timestamps.txt would contain time stamps for each frame as the absolute time elapsed BEFORE each frame, while ugoira uses a relative pause AFTER each frame.

E.g. these ugoira timings:

>{"file":"000001.jpg","delay":30},

>{"file":"000002.jpg","delay":30},

>{"file":"000003.jpg","delay":30}

would become these mkv timestamps:

># timestamp format v2

>0

>30

>60

I made a proof of concept python 3 script that converts all frames in a folder to an mkv file with correct variable frame rate. All frames need to be unpacked and the "ugoira_meta.json" needs to be saved to the same folder, because the script generates the timestamp file from that.

It won't let me attach the script, so I put it here: https://pastebin.com/kdaH6CqE

It won't let me attach the sample mkv file either, so I uploaded it here: http://tstorage.info/1rqrc4o43gqu

For identifying ugoira files, an idea I found was to use ffmpeg to generate frame hashes for the individual jpgs:

ffmpeg -i %06d.jpg -f framemd5 -

These hashes actually stay the same even if the container format changes. The jpgs in the original zip file and the muxed mkv mjpg frames will have identical hashes.

What do you think about this solution? This way we could get proper video files without re-encoding anything, and get consistent hashes to identify files.

I used to want to be able to add any generic file the most, but out of using hydrus daily what I want the most now is to be able to force-check a picture that failed to import properly with all the tags it should have. Automatic tagging is promising, but ultimately unnecessary for me since I just scrape files most of the time. If anything id use autotagging as a backup system to a failed tag import

The ability to tag images with consecutive numbers outside of the import files dialog. It would make tagging comics/doujinshi downloaded using the downloaders/watchers much, much, easier.

I am likely to make the poll today, with the release. I may unsticky and lock this thread to move convo over there, but I am not sure.

>>10622

That mkv jpg solution looks great! Thank you for figuring out the variable frame rate stuff and putting it a script together. I have copied this to my ugoira notes for when I get to this.

>>10623

Let me know if I misunderstand here, but you can probably do this now by running the problem file through a program like HxD to figure out its sha256 hash and then searching in hydrus in 'all known files'/'public tag repo' search domain using system:hash=abcd… . That said, if a file cannot import to hydrus, it likely doesn't have any tags in hydrus–or do you mean like 'what tags it has on the site I meant to get it from'?

In either case, I'd be interested in examples of files that look fine but won't import. Please feel free to submit the files themselves or URLs to them!

>>10639

Thanks–I put this on my 'see if you can sneak this in' list a little while ago, and it just didn't happen. I am adding it to the list here as "Add an incremental number tagging dialog for thumbnails".

>>10648

I greatly prefer doing transformations like this with our own CPU/GPU cycles, so I would likely start such a system by talking to local executables and then extend it to work with http POST queries depending on demand.

>>10650

The poll is up! Please go >>10654 to vote!