/hydrus/ - Hydrus Network

cant wait to see some of the changes to duplicates, if it does what I think, I may be able to do everything in program without a magnifier program assisting me, will still probably use the magnifier, but the need to use it being gone would be great for when I want to watch a video on the side, as the program at 8x magnification can completely kill performance for some reason.

>>13283

Unfortunately the slow/bad rendering of vids and images at high zooms will continue for a bit more. I have a plan to improve it–moving to tile-based rendering, where off-screen tiles are not scaled/rendered–but it will be a 'big job'. If you are at 100% zoom on a file, using a magnifier is a great way to highlight pixel differences in the meantime.

Let me know how these dupe changes go for you. I am basically done with duplicate overhaul now, pretty much just have some help to write.

>>13290

Hydrus dev I still can't fetch "pixiv work" tags after deleting the "pixiv" entries you told me to, and doing the fill in gaps thing >>13245 >>13257

Is there anything more I can do? I didn't change anything about my subscriptions, but I did look at them and saw that everything except the pixiv options for subscriptions is now duplicated, so it's like "nijie" and "nijie (1)" or something now.

>>13292

Hey, I am sorry, I think I read your post wrong the first time. I thought you couldn't search by tag, rather than actually getting certain tags. The pixiv parser is supposed to pull some tags, but I don't know enough about it to know exactly what should be being pulled. I will play with this and address your post more properly later. Can you tell me what a 'pixiv work' tag is–is it like the normal tags for the image, like 'touhou', or something else?

>>13294

Hi, thank you for helping me. It's ok you misunderstood the first time.

What I mean by "pixiv work" tag is the "id" of a pixiv submission. "Saucenao" straight calls it "Pixiv ID" when showing a result from pixiv, but the tag for it Hydrus produces is the namespace "pixiv work".

"Pixiv work" tags don't actually preserve the "page" format of pixiv submissions at a 1:1 ratio; instead they start at page 1 rather than page 0 (so for example, where the filename pixiv itself produces is "73378380_p0", Hydrus handles it as "pixiv work:73378380" and a "page:1" tag).

I was able to get "pixiv work" tags by opening a "gallery" tab in Hydrus and selecting "pixiv artist lookup" in the box under what you enter, then I'd enter the pixiv member id (I don't know what pixiv actually calls the IDs I enter there). Also subscriptions would fetch "pixiv work" tags just fine.

I don't know why I stopped getting them. I don't remember changing anything, except possibly updating. I only noticed I stopped getting them randomly, when searching for a "pixiv work" tag didn't return anything, when I knew it should have.

>>13290

oh sure, the program does not handle videos being large, but that's not what I meant, the magnification program I use when I have it going sometimes causes youtube to drop frames like a motherfucker, so me having 1/2 my screen hydrus, 1/2 my screen a video and my 13 inch display the magnification, the youtube will look drop I think 60000 og 70000 frames, that was the last time it was really bad. not sure what the cause is for it to go from watchable to horrible,

>>13294

Hydrus dev, I know you're busy and have your own life, but I was hoping you could prioritize giving me feedback on at least if it's just me with the issue of not fetching "pixiv work" tags anymore.

Since I still fetch every other tag just fine, I just searched "r-18" and "-pixiv work:*" at the same time to check the damage, and the earliest instance of Hydrus failing to fetch "pixiv work" tags for me was 20 days and 10 hours ago as of my typing this. I only first told you about it seven days ago because I was hoping it wasn't just me, and it would be noticed and fixed in the next release, or something, without me saying anything myself.

I probably didn't update Hydrus as soon as the release came out, so maybe 20 days ago I updated which broke it. But I was just wondering if it was just me with the issue.

>>13317

Thank you Hydrus dev. It's ok for the delay. It's just that I was hoping someone else would come forward with having the issue so I wouldn't have to myself. I can't really confidently meet social expectations, so this was kinda worst case scenario for me short of me being bullied for stalking your replies across threads or something.

I haven't done the "url downloader" suggestion yet (which I definitely will, since I didn't know you could fetch tags without redownloading images until you said as much), but I looked at the parser thanks to your instructions, and I can see that the new one has the same name as the default ("pixiv file page api parser (1)"), but with the "pixiv work" tags the default stopped fetching. So I think my issue is fixed now, thank you.

But this new parser you linked fetches a new tag (namespace) "pixiv id", which fetches the ID of the pixiv account itself. It's a fair name for that function, but I already have legacy tags using the "pixiv id" namespace from when I used "PixivUtil2" to rip from pixiv. Ideally I'd just be able to use this parser's "pixiv id" namespace stock, but in practice I can't, since I'm already using the namespace. I think I'll rename the namespace to "pixiv member id" if possible, which was a custom namespace I was using for pixiv account IDs.

Also every parser is duped except the pixiv parsers because at first I deleted only the pixiv parsers, then re-added defaults for everything. And now the new pixiv parser that can fetch "pixiv work" tags has the name of a dupe. I have to clean that up but otherwise I assume everything's fine on my end. Thanks again.

>>13329

>but I already have legacy tags using the "pixiv id" namespace from when I used "PixivUtil2" to rip from pixiv. Ideally I'd just be able to use this parser's "pixiv id" namespace stock, but in practice I can't, since I'm already using the namespace.

In hindsight this excerpt/ entire paragraph is just unrealistic, since everyone but me will use "pixiv id" namespace for the stock function, but I was planning to use a different namespace, which probably isn't feasible since any update to the parser will revert to the default.

Is there any way to rename instances of a namespace without forfeiting the entire namespace? This doesn't really work for me otherwise; I now just have two types of tags under the same namespace.

>>13331

I just checked and I have 139,198 files in my database with a "pixiv id" tag (namespace) I used for PixivUtil2 filenames.

I can theoretically re-download them all and re-import under a different namespace… but it'd just take a lot of time and my Hydrus would be unusable for the entire import time. Assuming I even have the hard drive space for this.

>>13332

This is stupid but my laptop only has 4GB ram, and normally I'm able to keep my browser and Hydrus open at the same time, and Hydrus is often frozen but still usable sometimes. But in trying to load a tab with 139,198 images at once, Hydrus remains frozen for 40+ minutes at a time for me, and I had to sit there watching it in my peripheral vision to catch it for the seconds in a row it was unfrozen.

I managed to load the images, highlight them all, press f3 to tag, and paste the desired tag in from a text document where I manually typed the tag when Hydrus was frozen. I even seemingly successfully pressed accept, because a subscription window partially loaded, signifying I returned to the main client window. But on this partially-loaded subscription window on the main client window, it's been frozen for like 10 hours now.

My measly 4GB ram never dips below 95% usage wih only Hydrus open in this state, with Hydrus using from 3 mil to 3.2 mil. So I was just wondering if there is a way to tag without loading images… like if I could perform the same search with arbitrary exclusions if possible, but without the burden of the client loading it all, or any of it in one tab.

>>13317

>Then I suggest you open a big search for all files with a pixiv url but no id tag and then ctrl+a and right-click->known urls->copy->pixiv file page urls, then open a new 'url downloader' page and set the tag import options to get tags even if urls recognised and file and in db, and then paste all those urls in. It should catch you up with the missing tags without trying to download the actual files again.

I just did this, and the number it shows is 0/12,025, when my search of "r-18" and "-pixiv work:* only showed 8,499 results. So I was worried it would only fix the r-18 tagged images, but it must've gotten everything instead. So I think it works, thanks.

>>13329

Great. Although the new downloader system is amazing compared to the old rubbish I had, it is not always pleasant to work with with the duplicate objects and so on. The next iteration here will have versioning or something and auto-updates.

>>13331

>>13332

I hope to add namespace siblings in some tag repository work that will start in the next 3-5 weeks! This has long been requested.

>>13348

Yeah, hydrus performance bombs out at about 30-50k thumbs in a single page. And manage tags doesn't work well for selections above about 8k thumbs. I am afraid the code just can't handle this stuff yet. Next step here is to improve manage tags so it does big operations asynchronously with a nice cancel button, and then yeah, I'd like some sort of faster tech for various large operations overall. The main problem for manage tags is when you try to add a tag, it needs to check all the files in the selection if they already have the tag in order to present you with 'pend to 3 files/petition from 2 files' kind of decisions. It does not scale well (and I think the final 'approve' stop is O(n²)), so for 10k files it just takes an unreasonable amount of time. As well as code optimisations, some CPU-faster entry actions (like 'woah lad, there's a lot of files here, let's make some add-only decisions beforehand') will come at some point.

>>13354

Great!

>>13367

Thanks for taking the time to read and respond to my random feedback on trying to add "pixiv work" tags to those that missed them the first time around.

I wanted to say too though that actually I was mistaken- the "url downloader" page only added "pixiv work" tags to the images tagged "r-18".

I was able to tell as much because I found a western artist who used to tag his images "r-18g", which is the gore tag, rather than mere "r-18" to mean adult. So every one of his images before he started using the "r-18" tag still didn't have a "pixiv work" tag.

I think I may not have a meaningful way to add "pixiv work" tags to every image that failed to get it the first time, in the end. Because even if I manage to do the "url downloader" for as many popular tags as I can, there will always be images that weren't tagged correctly.

But the only reason anything even has an "r-18" tag is because I used a "gallery" tab and put in a pixiv member id. I thought I could use the fact I've done that to fill in anything missing a "pixiv work" tag somehow.