/hydrus/ - Hydrus Network

Thank you for all your work Hydrus dev!

If I could have one far-fetched wish it would be a Regex find/replace for tags. Specifically so I can fix two common problems I have with pulling public tags.

Example of what I mean…

Find: (character:.*)(\s\(.*\))

Replace: $0

Reason: Change tags like "character:mario (Super Mario)" to "character:mario" along with all other screwed up character: tags because people don't seem to understand how to use series: tags.

Find: \s

Replace: _

Reason: Replace whitespace characters with underscores for all tags

For some reason, people who use boorus love including the "series:" namespace in the character namespace. While I do want the character: namespace when I import from a search as it saves me from having to add that information myself, or looking up every character an image has for images with many, many (talking like 40+) characters displayed. If it was an uncommon problem I'd just fix it by hand by searching for character tags and fixing them one-by-one… I have 12,000+ characters to fix.

welp… a fairly large happening happened, all I can say is oof. thankfully this means my archive wont gain a an absolute fuckload from finding new artists anymore but still.

>>13304

>>13305

I WAS ABOUT TO CALL YOU OUT ON THIS VAGUE WORDING SECRET CLUB ELITIST BULLSHIT BUT YOU'RE TALKING ABOUT SADPANDA

HOLY FUCKING BALLS

>>13306

In a similar vein, all the big dick posts on /a/ of people violently sourcing via an out of context exhentai link

no name

no artist

nothing

extremely big dick play on their part that's super epic, man

>>13305

thankfully i'm a bit of a data horder, I have lived though enough happenings to never trust shit I like to be somewhere tomorrow, thankfully I downloaded everything that I could when I found it and when that wasn't an option I downloaded from a different source, the only thing I couldn't get this way was image archives, and without new people shoved in my face, a good chunk of why I have fuck all for hdd space in the archive is gone. then I sit here looking at everyone else freak out about the good shit being gone… 'anyone have X' someone posts a link, I look it up in my archive, there it is already. fucking 47556 archive files and now that most of the shit that interests me is gone from that site, I acquire them FAR slower… the main thing I lossed from all this is a great aggregator for porn.

also, many groups gave up their blogs and released directly to ex, so even going backwards that way is a challenge… as for a competitor… if anything comes of talks, you are still looking at needing serves that can handle something along the lines of 200tb, and that's without the distribution. the current thought process is a torrent like thing that has a centralized base of gate keepers who will allow content. this is a check so no cp that would kill the project gets uploaded and potentially tainting the project, along with being distributed so hopefully no attack like this could affect it, the worst that would happen is the main authenticator goes down and no new things can get added. you would still at least need one database of all the files for incase shit happens, but a system like this could potentially lay foundation to a service that can never truly die. the only problem is that current thoughts would leave you open to attacks like torrents do, as in a copyright holder fucks you. typically they don't go after downloaders, just people who share…

As for a short term… I don't know who everyone is going to go to, there are a few that would work but they have nowhere near the infrastructure to handle what e-hentai did. my main hope is that we can get an aggregator up and running before 2020, something like baka manga but for doujin and porn, they do do that too but its far from ideal considering the vast amount that ex hentai handled

>>13306

lol, I figured that anyone on a chan site would have heard of this shit going down, its possible that this is a short term thing, as it seems like the reason for it going down is suspect, its all on if the owner will fight for a bit or not.

>>13310

>lol, I figured that anyone on a chan site would have heard of this shit going down

I actually stopped using any *chan besides bumming answers if I need them on whatever. I didn't fuck with how everything is just social posturing, at least on 4chan, never used any other. Like the out of context ex links. If you call it out, someone just tells you you're socially unacceptable in as emotional of a way for themselves as possible. There's nothing to respect there except the infrastructure of it being a *chan site… which the users completely forfeit to turn everything to as near an upvote/downvote system as possible. Then having the audacity to use "reddit" as a counterargument. Literally just the word "reddit" and nothing else in response to the socially unacceptable person.

>>13311

you know this is a chan site too right?

>>13314

>I actually stopped using any *chan besides bumming answers if I need them on whatever

>>13314

Oh I guess you might mean the "never used any other" quote. I'm not familiar with 8chan but I figured I could illiterate retard my way into asking a question here if I needed to. I've never just communicated with anyone on anything besides 4chan before.

>>13297

Thanks, that is a good idea. I don't think adding a 'this is better' green/red weight to that difference is guaranteed useful, but it would be neat to just have a 'this image is 720p, the other is not' statement for quick reference.

What are useful resolutions to highlight? Just the regular 720p, 1080p, 4k? I don't want to go down a rabbit hole of highlighting old 1280x1024 wallpapers, but those three standard 16:9 seem sensible to start with.

>>13303

Yeah, I'd definitely like something like this. The main problem with adding cleverer tag sibling and tag replace systems is I first have to flesh out the underlying manager and database 'shape' before I can start adding UI to interact with it. Tag siblings are in a similar 'rats nest' situation as duplicates recently were, and they sorely need a significant overhaul to improve logic and speed. After this dupes work, I'd like to hack-in a way to do namespace siblings, and while I am in there, I will push on this, but I think the bigger job of the complete overhaul will have to be a 'big job' in the next poll that comes up.

>>13293

So I updated hydrus and now I'm getting a lot of "connection failed retrying in 60 seconds" errors when downloading from pixiv, and it's really annoying. Even if I'm running a single artist gallery downloader with the default bandwidth options I keep getting this error every 3 or 4 images.

I guess its because of this from version 356

>- the network engine now waits significantly longer–60s–on connection errors before trying again, and with every failed attempt will wait n times longer again. when in this waiting state, a manual user cancel command cancels it out faster

Before updating I constantly got connection errors with pixiv, but hydrus used to retry immediately.

Is it possible to change or disable this waiting time or is it hardcoded in the program now?

>>13336

Thank you for this report. You can't change it currently, but I will definitely add an option for it and an override command from the network job control's cog icon button.

>>13318

Its not necessarily resolutions, specifically

lets say I wanted to do something tall

typically when I start I would go with 20000x10000

and when im done, it may end up being 5000x2500 or 2500x1250

however if someone fuck with the image, and scales it down in a retarded way, or wants to scale it enough to push it into spam territory that 5000x2500 may turn into 4998x2499, something like this, I would prefer the cleaner number 5000x2500

for checking things… if you don't want to have hard resolutions, then going for ratios would probably be a better bet. this would help point out when something may have been altered from source. like if one of them is a perfect 4:3 or 16:9 but the other one is a pixel or so off.

it may also be worth pointing out some websites resolutions, I know a few would cap a horizontal resolution at 1200, so it may be worth pointing that out as a possibility, hell, it may be possible to look at source and question if that is the version you want to keep if its known for compressing/altering images.

while a hard 'this is better' may not be the best way to go about it, a highlighted 'you may want this one more' would be nice.

>>13325

Yea, working smaller chunks would be nice, as for the exact first, my problem was a set of around 2000 files where many of the files were duplicates/alternatives with some things changed but nothing getting caught, especially with artist archives and such, smaller searches like this with speculative and a defined set of images would likely get me every single image to ok, but due to the time it took and the locking of the program (for what would likely end up being a 24~ Hour period of time) I abandoned that for the time being.

Lowercase everything = get the fuck off my computer.

Seriously git gud @ software design. The fact that you even need a getting started guide is fucking pathetic. No one is gonna use it cause it sucks balls.

>>13361

ok

>>13349

That resolution stuff is an interesting thought. I think trying to solve the problem of what is a useful resolution is beyond the time I have left in this current big job, and I should leave it to human eyes for now. I can add some common resolutions for now, though, and I can detect odd-numbers vs even-numbers easy, and we can iterate on that later if we see other easy answers. I'll make a job for nice resolution ratios, but it may come later.

is there a way to filter 'pixel for pixel' duplicates?

would love to be able to get those hammered out and could do that very fast too.

>>13373

Not yet–this data is currently generated on the fly in the duplicate filter. But I think it is very worth caching it in the db at some point. I can search it incredibly quickly to find pixel dupes and get started on some auto-dupe-resolve rules like 'if two files have the same pixels and one is a png and the other is not, delete the png' so we can finally and automatically clear out Clipboard.png trash dupes. It would be fast enough that I could even add a hook for it on import!

This would be optional and probably default on. More complicated stuff like 'if two jpegs are the same, keep the one with smaller/larger filesize' would be default off and up to the user.

>>13378

I would set both rules to delete the larger file size, it's very rare, but I have come across some jpegs were saving as png did lower file size.