/hydrus/ -/big data general/

Name
Email
Subject	REC STOP
Comment *
File	Select/drop/paste files here
Password	(Randomized for file and post deletion; you may also set your own.)
Archive	Archive [500 char limit]
* = required field	[▶Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field)
Allowed file types:jpg, jpeg, gif, png, webp,webm, mp4, mov, swf, pdf Max filesize is16 MB. Max image dimensions are15000 x15000. You may upload5 per post.

File: 9615483d4b836e0⋯.jpg (94.1 KB,1024x768,4:3,1601788051705.jpg)

/big data general/ Anonymous 10/05/20 (Mon) 05:17:17 ecf9e3 No.14802

or, "i've got xxxxxxx thousand images in my inbox, what the fuck do i do?"

i'm currently having this problem myself, so i thought it would be nice to have a thread to collect information about various ways to automate sorting/tagging/etc, such as:

* image deduplication scripts:

>https://github.com/knjcode/imgdupes

i'm presently also writing my own as well which should work somewhat better for large datasets, i'll post it here when it's usable

* datasets, useful for running against aforementioned dedupe scripts

>danbooru2019, contains all danbooru pictures + metadata up to early 2019: >https://www.gwern.net/Danbooru2019

i remember there being large dumps of other booru metadata on here years ago, but i can't find them anymore

* AI/neural network software for automated tagging, classifiers and etc:

https://github.com/KichangKim/DeepDanbooru

https://github.com/imamar94/ramrem-classifier

things i couldn't find but would find extremely useful:

>AI anime/photograph classifier

this would be very helpful, especially with deduplicating from danbooru2019

>subject classifier

if there was a NN that could tell me if the subject of a picture was a person or something else, this would be insanely useful, especially for the next item

>SFW/NSFW classifier

about half of my collection is porn. i'm planning on deleting basically all of it, so just being able to get that out of the way quickly would cut my workload in half

____________________________

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 10/05/20 (Mon) 18:17:29 e9829a No.14806

Looks kind of like a ML-general so I'll treat it as such. Got a bunch of stuff an Anon asked me to dump somewhere. No general order, some of it will be more useful than others. I'll try put the actually useful stuff first. It sounded like the OP requested an image comparison tool based on heat-mapping. I forward that request, having forgotten the name of the tool(s). EDIT: Forgot some stuff, formatting.

>http://www.visipics.info/index.php?title=Main_Page

Still probably the best and easiest GUI deduper. Works fine even with Hydrus folders, Hydrus might not like externally deleting pics.

>https://gitgud.io/koto/hydrus-dd/ see https://github.com/KichangKim/DeepDanbooru/

If you want automatic tagging, get it. Latest model is greatly improved. DD site seems even more accurate. Might have to do with model conversion. Run

hydrus-dd evaluate-api-search --service myTagService

in CLI to automatically tag everything in that tag service (default is my tags).

>https://github.com/imbaguanxin/Recognize-Your-Waifu & https://github.com/swrd1337/waifu-recognition & https://github.com/bakwc/PornDetector & https://github.com/hhatto/nude.py & https://github.com/notAI-tech/NudeNet

Classifiers. I personally don't care about these. Nudenet is the easiest to include in a project.

>https://thisxdoesnotexist.com & https://github.com/justinpinkney/awesome-pretrained-stylegan

Masterlist for NVIDIA stylegan projects you can test online.

>https://www.gwern.net/About

Gwern's site is cool if you're into this shit.

>https://github.com/keptan/superCuteGrab

Looks cool, but you need a saucenao account and it's generally a pain to build.

>https://github.com/soruly/awesome-acg

Anime&manga big data.

>https://github.com/josephmisiti/awesome-machine-learning

Bigass list of doing ML in a lot of programming languages.

>https://github.com/BlueAmulet/ESRGAN & https://github.com/ptrsuder/IEU.Winforms/releases & https://upscale.wiki/wiki/Model_Database

Enhanced Super Resolution GAN, GUI for effectively using it and scaling models for everything. What waifu2x wishes it was.

>https://gitgud.io/prkc/hydrus-companion

Obligatory shill for prkc if you're still using hydrus without the companion you're wrong.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 01/07/21 (Thu) 16:30:46 8c0dfe No.15077

>>14802

>i'm currently having this problem myself, so i thought it would be nice to have a thread to collect information about various ways to automate sorting/tagging/etc

It would be cool for hydrus to automatically tag anything imported, or downloaded with tags like hasaudio if it has audio. Or even just all files, seeing I see no reason for it to fill this info in for missing files. It can't really get it wrong. Same with medium:video, etc. These are stuff it can easily recognize from the file alone. Should help the inbox sorting a little.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 01/07/21 (Thu) 22:06:46 000000 No.15082

>>15077

It already does tag stuff based on the file, see the system tags. There is 'system:has audio' and 'system:no audio', and then there's system:filetype which lets you choose either the type (image, video, etc) or exact MIME (i.e. mp4) of files to seach.

There are plenty of others too, like dimensions, time imported, hashes, filesize, etc.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 01/08/21 (Fri) 10:22:34 8c0dfe No.15084

>>15082

>It already does tag stuff based on the file, see the system tags.

I was suggesting it automatically applies these tags when it detects them.

For example if you import a file that has audio, it knows that so it just applies the tag.

Unless I'm missing something, it does not seem to apply automatically.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 01/11/21 (Mon) 14:22:19 000000 No.15099

>>15084

Is that not what it does? Unless you mean applying a non-system tag like "video with audio" or something, it does do that automatically.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 01/11/21 (Mon) 15:21:08 8c0dfe No.15101

>>15099

Just tried adding a new url with a bunch of videos to the watch, some have audio some don't.

None of them have been autotagged with any tags, except for the few that have a hash matched.

So it does not seem as if it is auto-applying tags for stuff that are videos, and stuff that has audio.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 01/13/21 (Wed) 01:30:17 42c7cc No.15107

>>15101

do a search for "system:has audio" you absolute retard

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.