/hydrus/ - Hydrus Network

Does this have the twitter.com/user/media fix?

>>10183

Regarding this, I must be blind, I can't find any mention of it. Still thank you for trying to address it and not taking it the wrong way, I just felt a bit iffed after I changed all my watcher tabs and then noticed that a limit was set that I didn't specify.

>>10180

yea client is busy, even without noticeable hangs it happens, though far less likely.

>>10177

Personally have quite a few of these, but because they were mostly acquired through artist grabs, they are mostly findable without hassle.

I'm waiting on hdev to do something so a deleted image can be notated in a way that I can easily see why something was removed.

If I have to hotkey things like

Low quality

Have better

Prefer alternate

and label images, so fucking be it. but being able to see why I got rid of something needs to happen before I can move forward with culling duplicates, and moving on to general file processing seeing as I have half my db in the dup finder (a fuck up won't let the whole thing be in it at the moment) but still have nearly 180k images, and at least 170k before once combined those are well over 300k images, and even then that's assuming no new dups are found.

but that's just me, if a set is worth keeping everything in i'm not really going to put it in hydrus, i would keep it seperate.

______

On a side note, hdev, just tried the forget hash and urls, didn't work for for grabbing all of

first_seen_at.gt:3 days ago, upvotes.gte:150

From derpibooru, however cache rests seem to.

ill have to let it sit for a while so I can get a bigger pool of potential images,

>>10193

yea, back on their derpibooru again, the search im doing is also very weird.

first_seen_at.gt:3 days ago

that one is explanatory, only images uploaded with in the last 3 days

and

upvotes.gte:150

only images that get 150 upvotes, so I think I figured out what's happening, going to open a search and then do a gallery grab to confirm,

Ok, tell me if this sounds reasonable… ok, never mind, im fuckin retarded, the problem is right there on >>10147

"Found 5 previously seen urls, so assuming we caught up"

Now on a normal search this would be the case and it works good for the most part as anything that has a few already known images would most likely be caught up, however this search is capable of dumping images from days ago in as well as brand new images, so unless a fuckload of brand new ones come in, it will stop the search.

currently the oldest image in this search JUST passed the 150 threshold by 1 image, granted due to its content its an image I would rather not have, but that's a different issue altogether that will be solvable once a universal login system is in place.

Now the ignore image has and urls while it works, the sub still stops once it sees 5 already in db images, and it will do this unless I reset the cache. knowing what is the issue now, the way to fix it would be an 'ignore assumptions' option, as the program is quite literally telling me it's assuming it caught up because of other images it sees, and the nature of this search makes it so every page can have both new and old images.

I honestly cant think of much use for this outside of derpi, and even then this specific search, at the moment, there were a few places that would just dump porn images I use to go to, I think ftp servers or something like that, that would randomly get new images, if you could subscribe to a place like that through a simple downloader 'all images embedded' filter or something, it could be useful there, but yea.

>>10198

You select one of your downloads and click "highlight" or double click on any of your downloads in the top left under "gallery downloader". Its that way to reduce too much strain on hydrus where instead of having multiple tabs with thousands of images slowing it down, it will just not load up everything until you highlight it. If that makes sense.

>>10202

It also resets the sort order if you accidently change it or do something to the files, so clearing and rehighlighting will fix an accident.

————————–

That said, hdev, I had an idea.

So im downloading a gallery of artists stuff, while I would like a button for 'just show new' I had an idea, would it be possible to have new images be a displayable under items? possibly a menu option like deleted was, being able to quickly see how many items were found and then also be able to see how many are new would be a greatly appreciated feature for quickly parsing how much new content I aquire.

>>10191

I found the api call did not reliably give a new min_position value in the json, which I need for the next 'gallery page' url. Maybe I was missing something, but my old parser didn't transfer 100% and I didn't have time to put into figuring it out, so I dropped it for now. If you can get it to work, please let me know and I'll fold the new parser or whatever into an update.

>>10192

End of >>10179

np, I'll repost:

I am sorry you have had trouble here. I don't want to set the minimum time period too low because the gains are limited and it wastes a lot of CPU and bandwidth both clientside and serverside. It can take a decent fraction of a second to parse and process a thread's json, so if you had perhaps 100 good-size threads all checking every, say, ten seconds, you could easily accidentally fall into a situation where your GUI hangs due to all the threads doing redundant json parsing over and over.

At the moment, for dynamic checking, the minimum check time ('never check faster than') should be 30 seconds. If it is stuck on 1 minute for you, try just lowering it to 0, and the 30s should fill in automatically–let me know if it doesn't. I feel this is a good throttle to stop hydrus being overwhelmed. If you set the min time to 30s and 'intended new files per check' to 1, then if your /b/ thread is getting at least one new file per 30s, the client will automatically speed up its checks to check every 30s. And then, when the thread slows down, it will throttle back (up to your max check time, whether than is 10 mins or 24 hours) and save you a lot of waste.

Are these /b/ threads on 4chan? I haven't been there in a long time–what's a typical thread age? Is the difference between 5 and 30 seconds very important, or is it ok? Or are the images posted in spikes, so the dynamic checking missing the ends of threads because it thought it had slowed down? I am willing to reduce the min time further for advanced users, or alter how the file velocity is calculated, but only if it actually makes technical sense.

>>10204

That's an interesting idea. That UI is a bit too crushed atm, so I don't really want to add any new columns or data to them until I update the underlying ListCtrl to have some kind of right-click select columns or something. I could then figure out some kind of cached new_files count to optionally show.

That said, maybe I could optionally split the current '193/202' into a successful/already in db thing, like 56S137A/202, although this may be too autistic to actually parse.

Forgive me if you know this and your question is based around simply showing this in the number, but you can set it to only show new thumbnails under file import options. If you set it to only 'present' new files, then when you double-click, it will only show new, not anything that was 'already in db'. Your subscriptions should default to present/publish like this, and downloader kinosseurs set this for all their import contexts to improve workflow. Check options->importing to alter the default presentation options.

>>10206

Not sure what the twitter parser is doing differently, but the resulting images don't match ones I've downloaded directly. The urls match as :orig files but the image sizes don't.

>>10209

Correction: Newer stuff matches, older stuff that I've had sitting around for a while does not. I've also noticed this with Pixiv's from time to time.

>>10207

Hey, thanks for your answer! Yes, they are 4chan /b/ threads, and some of them quickly disappear. It's hard to determine the average thread age, I think it really depends on the content: Loli threads for example tend to disappear very quickly, especially if an overzealous mod is active, while more socially acceptable pornography can easily stay around for multiple hours.

To be honest, the threads where I think the 10 seconds check interval makes sense mostly, are the ones being pruned by overzealous mods (against site-specific rules, not to bring too much drama into this) and if the board is very active (for example during the recent birthday), where threads disappeared very quickly.

However, it's no issue for me to patch this into Hydrus myself (in ClientGUITime.py), which I have done so far and it works. 30 seconds check interval should theoretically be decent for most things, but I personally prefer to make really sure I got the entire thread, hence checking to the maximum allowed limit (every 10 seconds) seemed appealing to me.

>>10208

Yea, I know I can set it to only show new images, however that autistic part of my brain always runs back to the 'well what else was in the thread' and refuses to let me do that, mostly this part of my thinking refuses to let me change it to new only.

However thinking of it, a new images indicator, that said the worst one I have right now is

189 - 1D1F

now you could have it

189/60 - 1D1F

189 - 60N - 1D1F

189 - 60 - 1D1F

189 - 60N1D1F

personally I like the first second and third option, as it separates the got it done from the didnt get it done

I also like 2 and 3 more because it tells the total images and the new images in a less cluttered way

I also like 3 the most of them because if new was its own box you wouldn't need the N

and to top it off, you already kind of sort out new from old the only problem is you don't say it till its highlighted

How cluttered it is is an issue, but thats why I also suggested it could be an opt in option in the settings, for most things just seeing how many new images there are is enough, and I can just sort by date added and everything is good

I've been wondering about some features for a while.

Has it been considered before to add support for albums? I know that files can be opened in separate file pages by right-clicking on them, or can even be dragged into other file pages. The problem is that their order within the page can't be changed, and the page needs to be kept open.

My idea would be a file page that allows files to be manually ordered. It should then be possible to save the content of the window as an album file, which gets it's own hash and could have it's own tags, possibly with an option to automatically add the tags of files it contains.

The big reason I'm asking for this feature is, even though it's awesome how Hydrus can automatically list all the files I want to see, sometimes I want to group and sort them in a way that can't be logically expressed in tags and ratings.

E.g. I want to have some innocent pics in the list first, continue with some swimsuit pics, some softcore pics, gradually proceed to the hardcore stuff and add the creampie pics last.

Another reason is that it's a bit of a hassle to use Hydrus for reading manga. First I search for it by the artist's name, then scroll down the tag list to find the "title:" tag, then right-click on the tag and open it in a new page, then change the sorting to x-x-x-volume-chapter-page. But even then it can happen that I have an edited page, e.g. an inofficial fan colored page of some manga. I would probably filter to include or exclude some tag to exclude that page, but it's all a few steps too many to just read some manga. I know that collections can simplify this, but I don't like using them because I can't get a quick overview of all pages if they're automatically grouped.

But yeah, the first example is the true reason.

>>10211

>>10209

Can you give me some example URLs for this, so I can try my end?

>>10214

Ok, thank you. Let me know how this works for you. I am still a little afraid of this going wrong if I enable it for lots of people, but I could see enabling it maybe with a warning for advanced mode users only if it works well.

>>10215

Thanks, this is a much better way of showing it. I'll try to fit in a checkbox to display this.

>>10220

Thank you for this feedback. Grouping files has been a challenge from the start. I am not very happy with how collections and page sorting work for several reasons, but I definitely would like good solutions here. There are many reasons to combine and order files in certain ways.

I have two main ideas:

1) reworking the thumbnail grid so it can handle metadata-based grouping. Say by creator tag or star rating–it could split the 600 thumbs in front of you into 20 groups of 30 with an actual band of blank pixels in between each group. The groups could have 'title' info in the blank area, and be quickly selectable as groups and maybe be collapsible (perhaps to a single thumbnail, a la current collections) and so on. I could see this replacing collections entirely.

2) Supporting cbr/cbz formats thoroughly. I don't like how page tags have worked out–they are just too finicky and fairly pointless in content and for searching. Knowing the page is '7' is not as important as knowing it comes after the one with '6' and before the one with '8', and trying to edit mistakes is hellish. cbr and cbz don't care about this–they just have filename order. Treating chapters/volumes/3-page-shorts as single files also makes more sense from processing and viewing workflows. You rarely want to read pages 220, 17, and 56 of a volume–you want to read them in order, and if you want to bookmark, that is better handled in a hydrus-level metadata that would work great on single file objects.

Anyway, that's my current thinking. Better handling of 'alternates' in the duplicate system is also strongly related to this. I'd love to have danbooru-like 'file hints' when a file has parents/siblings, and generally extend the media viewer to be ok with a file having multiple pages or views, maybe having a two-dimensional navigation system (like left/right going from thumb to thumb, and up/down otherwise going between known file alternates or cbr/cbz pages).

>>10225

…And naturally they display properly here. Go figure.

>>10222

Thanks for your reply, I'll let you know. So far it hasn't been an issue, but I'll keep an eye on it.

Can I suggest something else, while I'm at it? I've got quite a few thread watcher tabs, a quick filter would be super useful. I imagine this just as me starting to type in some tab, and it filters the displayed tabs at the top.

So for example I could type "b/rule34" and it would show me all tabs that I renamed with that being part of the name. Currently searching for the correct tab when I want to add a new thread URL to my watcher is kind of annoying, I have to scroll through a rather long list.

>>10223

Smarter grouping and support for the comic book formats would be amazing and solve the manga problem.

But what's your thought about the user made albums that I suggested? Even with better sorting it's nice to be able to make custom image sets.

Ideally it should be possible to add an image to as many albums as you want, so I don't think it's a good idea to use the image's tags, or other meta data to achieve this, since this seems like it could get messy. I think the image's tags shouldn't "know" about the albums at all. Thinking of them as playlists is probably a good comparison.

Thinking further, albums could even be automatically generated, as a way to archive the context of any multi-image post that's downloaded.

E.g. many artists mainly post on twitter, but every so often make a pixiv post with all the twitter images. The images are part of that post, but also part of the previous twitter posts. Currently that context is mostly lost.

Then you might find a thread on an image board that you want to archive. All images posted are part of that thread, but also exist independently from it. You already have half of the images in Hydrus, downloaded e.g. from pixiv, so they don't get archived again. The remaining images get downloaded from the image board and have a time stamp and tags from a repository added to them. If you then remember that one awesome pic from that thread, you will need to guess it's tags to find it again. Having an album of that thread it would be much easier to find it again.

>>10222

nice to know, sadly no release today but cant wait till the 17th for it.

>>10214

with 4chan the two most heavily trafficked boards are /b/ and /v/ with one being 12-17 minute before bump off and 20-30 minutes for bump off. typically /b/ at least doesn't have thread pruning without reason, granted other boards have fairly shit mods who will hit everything no matter what. I find hydruses incremental checker good enough, with a 1 day max and 3 minute minimum, /b/ unless a happening happens, 3 minute static will catch everything. in the event you think you missed something, you can always go to the b archive, honestly even during birthday I don't think /b/ was moving that fast.

on this note hdev, if you give us a fuck it filter, could it be a checkbox/dialouge box? something I can press, import and it unchecks? just a painless way to activate it would be good, as there are times I know a thread will get pruned but don't want to babysit it.

>>10224

Ah, I understand the problem better now. Yeah, over long time-frames, some sites' storage is unreliable on a byte-for-byte basis. I have seen several different CDNs give different versions of the 'same file' over time and even at the same time. Usually the larger the site (and hence the more sophisticated CDN), the more likely they are to do this. Cloudflare is a significant source of this, and is used by more and more sites–as far as I can tell, they dynamically and non-deterministically serve 'optimised' versions of static images depending on which cache server you happen to hit.

My guess here is that pixiv ran an optimisation pass on their storage in the past couple of years, saving that 7KB. I downloaded the same file from pixiv in my browser just now and got the 8a05 version.

This problem is a big pain in the balls for us. I don't have a huge number of options here but to slowly improve the duplicate system in future. Ideally, we will eventually automate how dupes are merged, so you won't even notice this stuff, but we aren't there yet.

>>10225

Thanks, these are great. I get the same issue. I don't have time right now to get into it, but I will run these through ffmpeg manually and see what kind of crazy header data is coming back.

>>10235

>This problem is a big pain in the balls for us. I don't have a huge number of options here but to slowly improve the duplicate system in future. Ideally, we will eventually automate how dupes are merged, so you won't even notice this stuff, but we aren't there yet.

Personally, I'd rather it wasn't automated. Considering that even when booru images have their source listed, they might not be the the best versions, I prefer to use whatever I myself can pick up & verify. I've seen many cases of twitter images not be the :orig files, tumblrs not using raw (when that still worked) or even _1280. Add in that images get revisioned all the time (or even multiple revisions, which might not get listed) and unsourced images coming from who knows where, and it becomes questionable as to which hash/file is the "true" one. If everybody agrees to use the same image posted on the booru then tagging via PTR would be simple, but even the boorus don't agree on which hash is accurate.

It's a bit of a pickle since any exif changes the hash, and particularly for un-sauced chan images where I would have no idea if the hash matches the "original" or not. If there was a way to objectively figure out which hash is the "best" version, and the PTR could sync appropriate tags between versions then that would narrow things a lot. Unfortunately, I kind of doubt it's anywhere near that cut and dry.

>>10186

Hydrus doesn't work on x32/x86?

Got a question

Of the big image dump places where artist/artists go you got boorus, and most of the big ones, you got hentai foundry, inkbunny, you got log in needed ones like pivix, but furaffinity is missing, there any reason or is it just down to you want to get a better login system working?

I ask because, and I am making assumptions here, an artist I found and like posts there exclusively apparently, and apparently because they are barely furry, no one who regularly goes there posts their work anywhere else. they got 700-800 images, but the most any booru can pop up with is 100 and it's a bit annoying considering how absolutely garbage that site is for trying to get anything off of

>>10236

Yeah, I wouldn't rush into any system like this, and I'd almost certainly start with it turned off by default. I intend to introduce really easy merge-decisions first, and only with very high confidence values. Things like relatively-pixel-perfect dupes with like 1024xXXX resolution vs 4300xYYYY, or thumbnail 150xXXX vs 2300xYYYY. Figuring out extremely similar jpgs vs pngs also seems possibly doable–I'd love to automatically clear out the bloated redundant Clipboardimage.png crap I get from imageboards for their reasonable jpg originals.

Figuring out good decisions with watermark dupes and other more difficult scenarios would have to start as 'hints' to guide your human confirmation and potentially go further as we explore neural net tech.

I do not have confidence that I can hardcode 'this image is better than this image' beyond very simple examples.

My overall long-term feeling is that we are going to move away from quantitative hash-maps and towards qualitative knowledge about files. We'll stop sharing so much 'file x has tag y' and more 'this shape is a dog', but I cannot speak with any great amount of expertise yet.

>>10238

Not the releases I build and put out. I am now x64 on all platforms (and a hydrus client that is doing a lot of modern 1080p webm shit will spike through 2GB memory limit no prob). You can try to run from source, but I suspect it would be more trouble than it is worth.

https://hydrusnetwork.github.io/hydrus/help/running_from_source.html

If you try it, let me know how you get on!

>>10245

Yeah, the Hentai Foundry 'login' (which is really just a simulated click-through and 'allow all' filter set) and the Pixiv login were prototype hardcoded systems. They were enough of a hassle to maintain that I didn't go further. I am finally knuckling down and making the new system completely user-editable, like the new downloader stuff.

I never wrote an FA parser because, afaik, you can't get anything without logging in, right? I understand some of the advanced users have been playing with making one and using the cookies.txt import system to log-in hydrus as their browsers, but I don't know much about it. I think Inkbunny and DeviantArt hide nsfw behind login, which are other situations we will fix in time. Aside: This new system will be roughly aware of what a login script gives access to, and will present that in the ui.

I try to be completely content neutral in my hydrus work–furry isn't my taste, but I don't care if you like it. FA is a big site and has been requested plenty, so I am sure I will fold in a default parser and login script for it as the new login system becomes real. Let me know how it works for you.

>>10253

personally for me with how much monster girl shit gets made and put out, and how monsters get lumped in with furry more often than not, so long as the porn is good and not autistic, i'm sure we all know the difference there, I have no problem with it. artist im thinking of specifically right now has no one of fur affinity posting their content outside of it because it's not furry for the most part, so every other place that would normally have their shit just has a small old subsection of it and everywhere else they post themselves is a case of they got sick of posting there.

ill be sure to try it out when it gets put in. but i'm curious, how will hydrus handle captchas? I know this is an issue with some places where they force a captcha regardless of how you log in.

>>10259

This iteration won't support captchas, but it isn't actually that difficult to do. I supported it several aeons ago when hydrus had a 4chan dumper. Basically, you just fetch a jpg and a challenge key (some text that identifies the problem, and times out after x seconds) from the captcha server, present it to the user, and send back the answer and the same key in the same login form you put in the user/pass. Although some of the tech has moved on since then, so if it is anything like 'select all the road signs', I guess it is a more complicated proposition.

For now, any complicated login will have to rely on the advanced cookies.txt import and just copy cookies from your active logged-in browser (I'll be doing some more work on this in the coming weeks, to make it more compatible).

Thank you based dev.

Thank you based dev.

>>10225

Right, I just looked into this, and ffmpeg was giving some funny framerates:

mp4: 0.77 fps 20tbr

webm: 2.5fps 50 tbr

tbr is some kind of fps estimate ffmpeg is pulling from the codec or something. iirc, it stands for:

T: Standards

B: Engineers

R: Are fucked in the head m8, big time

I am not totally sure, but it looks like there is some variable frame rate going on when I look at them in MPC, so I guess it is even more complicated. In any case, I have rejiggered my parsing code to be less trustworthy of these values when they differ like this, and now the videos' frames are counted manually–I now get 7 and 13 respectively, with correct durations.

So these should work better for you in tomorrow's release! As hopefully should the other various low-framerate low-frame-count vids that seem to render all in a rush and then hang on the last frame for ten secs. In v326, turn on help->advanced mode and then right-click these videos and hit reparse files and regen thumbs. They should fix up, at least a bit.

Let me know if you run into any more trouble like this.

This week I started getting this error with pixiv subscriptions in version 325:

Exception

The subscription Pixiv encountered several errors when downloading files, so it abandoned its sync.

Traceback (most recent call last):

File "include/ClientImportSubscriptions.py", line 1125, in Sync

File "include/ClientImportSubscriptions.py", line 479, in _WorkOnFiles

Exception: The subscription Pixiv encountered several errors when downloading files, so it abandoned its sync.

It downloads around 10 to 30 pics, then throws that exception. Is there any way to display the post IDs of failed downloads? I retried a few times, with one artist, and it didn't download any new pics on retry, then threw the exception again. So I assume it fails on the same posts again.

>>10275

Pixiv are in the midst of changing some of their layout again. Please try again in v326, which will automatically update you to a hopefully more stable solution.