/hydrus/ - Hydrus Network

>>13156

Wanna pitch in on how things can be done, judging by this thread? Of course the more variations the merrier

>>13156

>>13219

Not the other guy but what about a collaborative parent and sibling database? I mentioned this once to the dev awhile back when he first added the option to import and export parent and siblings.The idea is something where everyone can contribute to and add to their client database. For parent tags, my biggest issue is trying to connect each and every character to a series or a sub series to a parent series. If there was a parent database that already did this then this could save a lot of time for some people.

A sibling database though might be a bit tricky to pull off as not everyone is going to agree on a certain term used for a tag and certain people can be pretty autistic about their tags. Over the years of using Hydrus, I've noticed how some booru and sites create useless tags that clog up my client. Some would have misspellings, other times I would have tags like "buttjob", "assjob", "butt job", "ass job", "hotdogging" or tags like character:[first name] [last name] and character:[last name] [first name] when they all mean the same thing. With a collaborative sibling database, not only could we eliminate stuff like this by merging everything that means the same but, also pick and choose which term you like best and have hydrus remember that from now on even when you download a new database. This would also be great for sites with different language tags like Pixiv.

Something like this though would also mean you'd need a site other then github and 8chan to pull this off as you'd also need a place to decide and discus a tagging schema that everyone can look through and follow.

>>13284

> Not the other guy but what about a collaborative parent and sibling database? I mentioned this once to the dev awhile back when he first added the option to import and export parent and siblings.

> The idea is something where everyone can contribute to and add to their client database. For parent tags, my biggest issue is trying to connect each and every character to a series or a sub series to a parent series. If there was a parent database that already did this then this could save a lot of time for some people.

> A sibling database though might be a bit tricky to pull off as not everyone is going to agree on a certain term used for a tag and certain people can be pretty autistic about their tags. Over the years of using Hydrus, I've noticed how some booru and sites create useless tags that clog up my client

> With a collaborative sibling database, not only could we eliminate stuff like this by merging everything that means the same but, also pick and choose which term you like best and have hydrus remember that from now on even when you download a new database. This would also be great for sites with different language tags like Pixiv

I would say that the first step of solving that issue is to have a wiki and a forum/chat dedicated to sorting those issues out. We urgently need a platform to debate such issues.

Here is a basic idea: tag relationship diagrams like [core word] [derivatives] [misspellings] [disambiguation] and [{foreign language}] for siblings, and [subsets] [supersets] for parent/child.

From these we create a page for each tag, from that we can start and sketch out a taxonomy that makes everything clear, like how Wikipedia manage their pages.

If I am allowed to pick a repo, I would choose https://github.com/Requarks/wiki for Javascript-based or maybe a Python-based wiki if the whole of Javascript is a no-go (but it definitely needs page crosslinks, chats/forums, moderation.permissions, branching/forking, archives, pPost too long. Click here to view the full text.

>>13287

example https://github.com/edgewall/trac https://github.com/moinwiki/moin

See also https://wiki.python.org/moin/PythonWikiEngines https://www.progracoding.com/10-best-wiki-cms/

>>13175

just looked at the old version and saw the reply

for multiple similar images at once, the only real use I can think of it is I have a full comic/set and I want to remove the images, I have images that hit an 'unappealing' tag, and I want to get every one of them at once, or in my case, I want that tails comic wiped from wasting space. at least for me, haveing them sort is kind of a moot point.

however if you want them to sort, I think adding a queue system to duplicate filter and having files open in a duplicate filter window would work. this way you search one files duplicates, it posts results and goes onto the next files results and duplicates. this would allow you to also have some access to duplicate tools.

as for the other one, I don't know how simple it is, but the database knows that the files were thrown away, so a 'remove deleted from view' would probably do the job, unless you want to expand the 'remove from view' options to be a bit more complex and have their own sub menu.

It could also be handled from a highlight all function, this would add another step to removing deleted from view, but it would also allow something like 'highlight all 'jpeg'' so I could move them to a different window. it may not have a whole lot of use case for search results, but in the case of publishing a subscription, I think it would have a fair amount of use.

I use hydrus on an external drive and the other day my drive came unplugged accidentally while using hydrus and the program closed as you'd expect it would. Though when I reconnect the drive and try to reopen hydrus I get to "initializing managers tag siblings" then hydrus crashes. Any advice? I have a backup but its a couple weeks old and I'd prefer not to use it. I still have my db folder and client file and everything. Here's the command I use to launch hydrus just for reference. I'm on osx btw.

open -a /Applications/Hydrus\ Network.app –args -d=/Volumes/Drive/hydurs/db

>>13178

I am glad you like the sound of it. The program does not phone home anywhere, but if you are thinking about using the program and are truly skeptical, I recommend you block it with your firewall to start with. It shouldn't try to make any connection unless you tell it to through a booru downloader etc…

If you aren't a user yet, your enthusiasm may dim a bit when you stumble over some of the unprofessional design or buggy code, though! Everything is breaking all the time.

I am a typical Anon and am trying to make the sort of software I want to use. Privacy and user control is very important to me, so it is baked into the program. I love the imageboard community and enjoy giving back to it through my work. If you do use the program, let me know how you overall find it after a few weeks, once you are comfortable with it. Feedback from new users is always useful.

>>13179

Great, thanks for letting me know. I'm not anywhere near ready to start work on this, but next step for the whole downloader system will be some sort of subscription + version checking system for the various scripts, like you'd see in Nexus Mod Manager, so script authors can update people more easily when sites change like this.

>>13181

Thanks, I will keep thinking about this.

>>13187

Hmm, that is odd. When it crashes, does it throw up any error window at all, or just close itself? Does the close look 'clean', or does it just die in one frame?

My first guess is the disconnection caught the db while a transaction was flushing to disk and so there is some bad data in there, although usually that stuff is recovered from.

If you go to /Volumes/Drive/hydurs/db and check your latest client.log file and scroll to the bottom, is there any traceback error there for what happens after the 'booting db' phrase? A db error generally shouldn't cause a crash, so it is also odd that you aren't getting an UI popup of some sort saying 'hey, the db had a problem on boot'. If it is erroring silently and quitting the program neatly, there could be some error in there, but if it is a full-on actual crash, there won't be any kind of tidy-up statements.

If there is no error, let's check your db files. Here's a help file that is a slight pain in the neck to get to as an OS X user:

https://github.com/hydrusnetwork/hydrus/blob/master/db/help%20my%20db%20is%20broke.txt

It will walk you through some integrity checks to make sure your db is all ok. I think there is an OS X sqlite3 executable in your App, if you know how to navigate that, or you can get the tools here:

https://sqlite.org/2019/sqlite-tools-osx-x86-3290000.zip

Let me know what you find. I am somewhat confident we can recover this db, but we might need to reset somethinPost too long. Click here to view the full text.

Thank you based dev.

Is there any way to search for specific duplicate types?

I like to replace files with the best version available, but I fucked up and left a lot of "this is worse" versions around, and I'm not sure if I'm retarded or new versions prevented generic ">0 this is worse" searches.

>>13144

Searching for multiple 'similar files' from right-click is probably doable. Sorting them the other end is probably a hassle, but we'll leave that for later. I'm supposed to be extending system:hash to take multiples soon, so I can probably do them together.

>>13145

That's a good idea. I would say hit right-click->select->(something) where 'something' would be like 'remote' files or similar and then hit remove in a second action, but I can't remember if there is a good option to select those files. I'll make sure that select menu is comprehensive and see if I can do a quick submenu that that does remove. Maybe if no files are selected, the remove menu instead branches to a similar select-type submenu. I often do 'select->archived … remove', so it would be nice to do that kind of stuff in one action.

>>13152

Not precisely. I don't have the help written yet to explain this nicely, but duplicate files are now stored in a valueless group that has a single best 'king' file.

In the coming weeks, I will add something like 'system:is_king'. I think if you combine something like [ 'system:num_duplicates: duplicates>0', 'system:is_not_king' ], that will let you search these non-king (i.e. worse) files while excluding the good ones, and you can then deleted them en masse.

Now I have this simpler duplicates tech, I expect it to slowly appear in more locations. A nice plan is to have the main media viewer say 'hey, this file has a worse quality dupe', a little like how danbooru shows this info in a file page, and have a quick way to flick to that without interrupting the main 'list' of files the media viewer is looking at. See if the new search predicates help you do what you want here, and let me know what else would work for you.

>>13153

> and figured out an advanced-users-only extention to the IPFS service to permit the experimental 'nocopy' pin mode

Please dumb it down to the masses such that IPFS can be used by any pleb.

HACK THE PLANET!

is it possible to delete whole filename: from PTR?

>>13154

It'd be nice if we can figure out an easy-go mode for IPFS. We'll see how this works for the advanced users, and I expect I'll figure a read-only easy mode downloader for multihashes using http gateways, and we'll iterate off that.

>>13157

Not easily yet, but once I am done with the current duplicates work, I'll be adding serverside and clientside tag filters to tag repositories so both ends can prohibit types of tags automatically.

>>12602

I always forget to close shit. Depending on your computer that script might fuck some files up, here's the fix.

import sys 


if(len(sys.argv) < 2):
	print("script.py file.txt")
	sys.exit(0)
	
	
fileName = sys.argv[1]
file = open(fileName)

gel = open(fileName+"-gelbooru.txt", 'w+')
e621 = open(fileName+"-e621.txt", 'w+')
r34xxx = open(fileName+"-rule34xxx.txt", 'w+')
dan = open(fileName+"-danbooru.txt", 'w+')
tbib = open(fileName+"-tbib.txt", 'w+')
xb = open(fileName+"-xbooru.txt", 'w+')
safe = open(fileName+"-safebooru.txt", 'w+')

for line in file:
	gel.write("https://gelbooru.com/index.php?page=post&s=list&tags=md5%3a"+line)
	e621.write("https://e621.net/post/index/1/md5:"+line)
	dan.write("https://danbooru.donmai.us/posts?page=1&tags=md5:"+line)
	r34xxx.write("https://rule34.xxx/index.php?page=post&s=list&tags=md5%3a"+line)
	tbib.write("https://tbib.org/index.php?page=post&s=list&tags=md5%3a"+line)
	xb.write("https://xbooru.com/index.php?page=post&s=list&tags=md5%3a"+line)
	safe.write("https://safebooru.org/index.ph

>>12604

Thanks, this is pretty useful. Sankaku has an md5 lookup as well:

sankaku = open(fileName+"-sankaku.txt", 'w+')
sankaku.write("https://chan.sankakucomplex.com/?tags=md5%3A"+line)
sankaku.close()

>>12604

>>12631

>>12602

I not able to get the md5 unless I open each image in the hydrus media viewer first, in which case there is no easy way to select them all and get it for all of them.

>>8396

Can someone bundle a torrent client with Hydrus+IPFS?

Just found this https://unix.stackexchange.com/questions/44247/how-to-copy-directories-with-preserving-hardlinks

(People in >>>/animu/81926 recommended Hard Links rather than soft links, so yeah)

>>13035

I think pixiv updated their site again and now the downloader stopped working.

>>13094

Yeah modified is fine, thanks.

e621 did an oopsie with Cloudflare!

>>13100

Should be a fix coming in 358 later today! They switched up their API, and a user helpfully provided a new parser. Let me know if it still gives you trouble.

>>13102

Can you say a bit more about the error you got here? Was it the gallery page step, the file page, or the files themselves? I just did a quick test here on my dev machine and it worked ok here. I have seen it kick in site-wide for just an hour or so, which I guess are DDoS protection or similar, but I have also recently had several reports like this, where it works ok for one user but not another. I presume it is an automatic CF-side rule kicking in, based on IP or some other profile. Any chance you are on a VPN? If so, if you switch server, does that fix it?

I have a job to better detect the error page returned and giving a better 'CF problem' error and maybe 'ignored' status or something that can maybe try again several days later once I eventually have that download retry-later tech in. My guess is this problem will only increase in future.

>>13103

Yeah CF got hit cause of muhChina, so it definitely is an issue for the meanwhile.

>>13025

>>13023

>>13022

You can import cookies, better help available on the discord

>>13013

Thanks. Yes, there will be a fix to seeing same alts again. Might be in this week as I do the main potential data, or in next couple of weeks if it proves complicated.

>>13019

>>13020

>>13021

Yeah, I'm going to try to fudge something like this. I'm not totally sure on all the edge cases yet, but I will do something. Maybe matching the smallest x or y to be the same virtual zoom height, with some clever exception for portrait vs landscape pairs. and offsetting drag delta by the (0,0) point on both files. I haven't gone into this exact code, so I can't speak cleverly yet, but it is in my big job todo. Thank you for your feedback.

>>13022

Yeah, hit network->downloaders->manage logins. There should be a gelb entry there by default. Add some credentials, set is 'active', and do a 'do login now' test. (it is worth doing a manual download as well there and then to test it worked for real–this stuff can be unreliable due to all sorts of factors). After then, if your login expires (I think gelb login lasts a year?), hydrus will try to login again automatically without you having to do anything. More info here, at the bottom:

https://hydrusnetwork.github.io/hydrus/help/getting_started_downloading.html

>>13026

>>13028

If you have help->advanced mode on, then clicking 'custom action' in the dupe filter should let you select 'alternate' as the dupe type, which then gives you the dupe merge options and a choice to delete A, B, or both. If you do it from the thumbnails, you should get the dupe merge options too.

I agree that having some sort of 'favourites' custom merge actions will be useful for alternate processing. I think I will do it when I get around to making a proper alternate processing workflow. I am trying to shy away from significant alternate processing in this primarily duplicates data structure overhaul, since it is a black hole of work that I could get lost in for months.

I took away the 'set as not related' from thumbnails because in data terms, this duplicate type is very inefficient if not applied to files that already have potential status. I have left it to the dupe filter and 'show some random pairs' quick workflow. I am still thinking about bringing it back for thumbnails, but at the moment probably won't. It really is a 'set as false positive' rather than 'dismiss any potential info' as it has been previously intepreted. I'm still thinking about it.

I'd like you to pay more attention to github

>>12943

Unfortunately, having downloaders know about deletions after the fact is tricky. Completely downloader items only know file hash and whether it was new/already in/deleted at time of import. They aren't plugged into the content update pipeline and only cause additional file metadata to be loaded in the thumbgrid when thumbs are called to be displayed.

Are those 154k/139k counts are related to this duplicate search domain question? And the ~150k count seems low compared to 4M? Assuming you have done your full search at 0 distance or whatever, that sounds correct–it is only searching the files that have potential matches found in the duplicates system. Having 1/15 to 1/10 duplicate files is typical, and since you have 2.6M eligible files, it sounds correct to me.

There's no good way to do a complete wipe. The old system had some hacked-together commands, but it never worked well. This new system, once it is complete in a few weeks, will have logically consistent 'dissolve' commands that will be able to nuke everything. You can do this if you like, but I suspect your duplicates are fine.

Let me know if I misunderstood. And please bear in mind I am actively overhauling this system right now. Potentials will move over in a couple of weeks, going from pairs to groups, which may fundamentally change how this queue is presented in ui anyway.

>>12945

Not yet. You can hit the 'skip' button, which I think is up arrow shortcut by default or you can hit the double-right arrow icon button, but the same pair is likely to come up in a future filter.

Best solution for now to put it off indefinitely is to exclude it in the duplicate file search domain. You could try adding a 'don't dupe filter' local tag or something and exclude that, or through a system tag if that makes sense.

>>12967

Thank you.

I appreciate the frequent updates.

>>12977

Thanks for clarifying. Yeah, I'll stick the trash icon on 'remote' files that were known to once be in the client. I think thumbnails know this already, so I can just draw the icon on.

My thing with these game controllers is having the side joystick. Feels so good to WASD with just my thumb, I can't see myself going back to a regular keyboard for vidya. I may bite the bullet of a Razer. I did some searching a while ago, and I think you need to 'initialise' your Razer controller with the cloud account, then you set your shortcuts, and can then set it to 'offline' mode. You can't edit the shortcuts completely locally and it is seen as a default device until you do it the first time. When my G13 finally packs up, I'll just have to bite the bullet. I'm ok with having a cloud bullshit driver if it is only ten minutes before I firewall it, even if I find it atomically stupid.

I understand your count issue better. I am not sure about the million files. Normally, a handful of files will be discarded if they have no useful search data–such as completely black/single-colour images–but that does not account for so many. As I do more of the current duplicates work, I will be integrating the 'phash regen' that that maintenance stuff refers to to the new file maintenance manager. If your client has somehow forgotten or missed that big chunk of files, it will be easier at that point to requeue them for generation. I'll have some UI for it under the database menu. At that point, I recommend you do something like 'regen missing search data for: system:filetype=png/jpg'.

Adding skip is a bit tricky just because it adds complexity in storage, workflow, and UI. I haven't reworked the potential data storage yet so I can't yet say cleverly if this would be ok or an utter pain in the neck. I will keep it in mind, but I suspect I will not add it.

>>12719

The majority of people view the weak with disgust, that's the hard truth. It's kind of cruel that society pretends otherwise.

>>12678

>>12677

>>12719

Sorry, I got caught up in the duplicates work last week and haven't checked the catalog for new threads in a bit. Assuming you aren't giggle-posting, it sounds like you are having a truly hard time. I sympathise deeply, but I am afraid I do not believe that happiness or self-esteem comes from getting love from others. I love you as a fellow Anon, but I am skeptical that repeating that sentiment would validate you.

The only one who can make your brain work right is you. As long as you believe you need affirmation from others to feel correct, you will be open to further abuse.

I strongly recommend you start getting exercise, if you do not already. Go for short walks every day. If you can scrape together a few bucks, get these books:

https://en.wikipedia.org/wiki/The_Feeling_Good_Handbook

https://en.wikipedia.org/wiki/Meditations

They are on Amazon and also in the library. If they ask, just say you are getting Feeling Good for a friend.

>>12831

Sure, let's give it a go. 64GB/week. You can see it in your review services if you refresh account right now. If it goes funky for some odd bandwidth-cycling reason, I can always reset it.

Ok hdev, still going though video files for removal, and this is the 8th fucking video that is either the same or damn near the same as the others.

So this got me thinking on if I had the ability, how would I check duplicates for video. while i'm not coming up with good answers, I do have ideas.

first, could take an image every second, or so, and use them to fuzzy match, and if something fuzzy matches it goes a bit more process intensive to see if it is a real match. This would allow for both exact duplicates, and clips from larger files to be found. given what the dup finder can do with literal garbage, I think this could work if its even an option in the first place. depending on how big files are it would pull, it could bloat things out, but ultimately I think it would be an overall good.

>>12841

Yeah, I do plan to add videos to the duplicate system, and I originally designed it to eventually support them. The recent file maintenance system was a step forward in prepping for the CPU work we'll need to do to retroactively crunch the data on this in a reasonable way.

I plan to do something like what you propose. The duplicates system currently works on comparing still images' shapes with each other, and it allows for multiple still image 'phashes' per file, so my task is selecting a good number of useful frames from videos that will match with others.

If it is reasonably possible, I would like do something more clever than just picking one frame per x time units or frames. This would line up right if our two vids were very exact conversions or resizes, but some of the codec changes drop a frame at the start or do 29.97fps vs 30fps bullshit that would desync our comparison. My original duplicates system did add vids by using the first frame, but so many have a black/flat colour first frame that it lead to a billion false positive dupes. Vids are no longer included, and I also drop anything that looks too much like a flat colour image from the system entirely. If I could instead find the x most 'interesting' frames of a video, then 2-second gif clips of 20-second webms would have a higher chance of being matched, and 30/60fps conversions would too.

I don't know, though. That is probably beyond me to do well, or maybe I can hack something that is good enough. I could do something like generating a phash for every frame in the vid and then have them compete with each other to remove similar-looking frames/phashes until the 20 most unique were left. It might pick up a bunch of false positives again with, say, a black screen with a bit of white text on (like an 'intro' title card) though.

Still, I am almost ready to do this now, and dupe work is proceeding, including more efficient storage of potential dupes, so maybe the answer here is to get a simple system in and then iterate on it.

>>12846

when I say literal garbage, I mean the image is drastically different, to the point I cant even see how it thought they were dups, but those are from the asshole who specifically fucked with dup detection when creating trash images, the dup detector is able to have enough wiggle room that even if the images aren't lining up perfectly, it may spit out something useable, and I used 1 second just because no mater what I watch 1 second isnt enough time for a 100% scene change, it should pick up some duplicates from that.

on the title card, you could make a generic here is a black frame with text, and have a few variations of it, this could be used as a compare to X image and trace it if it does, so it would automatically know that everything with it will bee seen as a duplicate.

If you are able to, try to get in contact with the people from what anime is this, and see how they did theirs, it may give some ideas.

>>12846

Could always check how Video Comparer works. It's the best video dupe finder software I've used.