/hydrus/ - Hydrus Network

Thank you very much for the pixiv update!

Regarding Twitter, would it be more beneficial to run that through the media page, or even to fans a secondary downloader for that?

I know I'm a bit late to the party, but I'm not a big fan of the new download system in 316.

I just updated to this version from 304 because I was having issues with Sankaku (big shock, right?). Every booru has a file limit - Sometimes, like with Gelbooru, it's in excess of 300 pages, some have a smaller one like Sankaku's pitiful 500 images.

If you want to grab an entire tag, the best way I've found is by adding a tag that takes you to within the image limit, starting a query, and then using a dash to remove that tag from future results. Repeat until tag is grabbed.

This is great for Gelbooru, but many other boorus like Danbooru and Sankaku have limits on the number of tags you can use at once. For these sites, when I get an entire tag I have to download random tags until I end up with a total number of files equal to the total number of files in the actual query I'm trying to get all of the images for. The old downloader refused to show an images on a single page if it was already grabbed by another query, so this was a great workaround of the booru system.

Is there some way to do a similar workaround here? I checked the 316 release post and it said that there was no way to use the old-style downloader.

>>10089

Tried twitter downloader and it works fine, but it seems like Twitter recently changed their image url you encounter when you right click the image to get the url, though previous one still works as well. It's like this:

https://pbs.twimg.com/media/DQbc6kRUIAALHnV.jpg (previous image url, resized by twitter)

https://pbs.twimg.com/media/DQbc6kRUIAALHnV.jpg:orig (previous image url, untouched)

https://pbs.twimg.com/media/DQbc6kRUIAALHnV?format=jpg (new image url)

I don't know if you can access to the untouched images in new format but I hope this doesn't turn into Tumblr's decision with RAW image urls.

Thank you as always, based dev.

Bit of a continue from last thread

I have this

first_seen_at.gt:3 days ago, upvotes.gte:150

set up on a derpy subscription, I check it every 6 hours as its something that should have a rolling amount of images coming in and getting on the list.

problem is I do a gallery search for the same query I come up with about 60 more images every day

Is there a way to set a subscription to just work as though its a brand new search instead of trying to remember where it last was? I'm not 100% on how subs work, but it seems it's trying to resume from somewhere and it's not getting everything, Is there a way to make it treat the subscription as new each time it checks, not remembering pages but remembering file urls?

With the new downloading system done, where does this leave the file lookup scripts? This function has been half broken for ages. Surely it could be updated to use the parser scripts used by the downloaders.

>>10090

I am sorry, I don't understand the question about twitter–can you explain a bit more? It should be fine to download twitter in any regular gallery download page.

>>10091

I am afraid it all works on separate downloaders now. You might have some success changing the 'presentation' options under file import options, which you can set defaults for under options->importing. Tell it only to present 'new files', and your additional queries will not display those already imported. I have all my imports work like this and find it very helpful for advanced workflows.

>>10092

Thanks. Hydrus should be getting the :orig version. I don't know if this is truly untouched (the files are suspiciously small), but I appreciate they have anything. You can check you are getting the right :orig url by right-clicking the file in hydrus and looking at 'known urls'. It should have the tweet and the raw file url.

>>10116

Please try running client_debug.exe in the install_dir folder. It might be because you are on a 32-bit system, which is presently unfixable, or it could be a different library import error.

>>10118

client_debug.exe runs for a moment and then quits without resolving the issue. Does it print its text to a file somewhere?

Also, I'm on 64-bit Windows 7.

>>10099

Unfortunately not. Subs are designed to work on lists that expand statically, with new files coming first. They work best for artist feeds (i.e. like you are subscribing to the artist). They'll always try to 'catch up' with where they were last sync, so if the file results aren't the same feed or aren't coming in descending post order, it'll likely hit its periodic limit and/or complain.

I think you should be able to get this to work though, if you don't mind a popup every day or whatever. Can you expand on how you think it is resuming from the wrong point? Could your derpi domain currently be bandwidth-blocked right now? Subs I think have a 256MB limit per day, typically, so if these are all big pngs, you might be getting choked there–i.e. your sub is fetching the 60 new URLs per day, but it still needs to pursue stuff from days ago, and can never catch up.

Check your sub in network->data->review bandwidth. You might like to go into its entry there and give it more daily bandwidth, if that seems to be the problem.

Then make sure its checker timings are to check once a day, statically, with a periodic file limit that makes sense for the query you are hitting.

Let me know if you figure it out. I didn't design it for this, but if a small tweak on my end can make it work better, I'd love to know.

>>10120

Thanks. Twisted is my network library. It runs the server and can run an optional booru off the client. It doesn't do anything by default for the client, but I think it has to establish a loopback interface as I import it so, afaik, it can talk to itself. It looks like that is what could be failing here–do you have a super paranoid firewall setting that stops programs from opening loopback ports?

>>10108

Yeah, I hope to brush this up sometime. It was the prototype for the modern parsing system, but it has since got left behind. I don't know when I will fit it in–can you say what you would most like me to do to it?

>>10110

Thanks, this is very interesting. It seems bonkers that subsequent regens would re-break it–that suggests the thumbnail generator is non-deterministic. I wonder if instead the thumbs are regenning right, but the thumbnail cache is getting mixed up trying to delete the old ones.

Can you post that Ciri(?)+Monster webm here, or point me to the url, so I can try it my end? That appears to be the only one that actually reverted. Maybe the two lara ones were just delayed somehow in updating, although you can post them too if it is convenient.

>>10111

Thank you for this report. I will check this and fix it this week.

>>10114

The whole routine for "ignore retweets, only download images" can be entirely circumvented by going to https://twitter.com/user/media and that gets you only files they uploaded.

>>10127

Thanks, I hadn't thought of looking at that. It uses a slightly different api call, but as far as I can tell, the json is in the same format. I will see about rolling out a more efficient twitter searcher this week!

>>10139

Thanks. I am afraid I was unable to reproduce this problem, either on that webm or doing repeated regens on larger multiple selections. I have never seen the colour shift and 'quarter displacing' that you are seeing here before, so I am wondering if instead your machine has a combination of factors that could somehow randomly produce this.

Do you ever see the 'quartered' frames when you just look at these videos in hydrus? If so, are they ever something special, like the first or last frames of the videos?

Can you check help->about and tell me what your ffmpeg version is? Mine, the one that should come with any windows hydrus build, is 4.0.

Under options->media, what's your 'video thumbnail %' at? I tried 25, 35, 50, had no problems–are you at 0 or 100, or anything else special?

If you regen some still images, do you ever get this same problem? How about gifs?

If you turn on help->debug->profile modes->pubsub profile mode (warning, don't leave this on for too long, it is spammy) and then regen a video's thumb, is there a critical pattern of [ modal_message, clear_thumbs, new_thumbs, new_file_info ] buried in the mess of set_status_bar_dirty?

>>10115

Thanks, I'll try playing around with that.

>>10146

>Do you ever see the 'quartered' frames when you just look at these videos in hydrus?

I tried viewing some of the ones with bad thumbnails and they have no problem playing in hydrus and show no visual errors.

>Can you check help->about and tell me what your ffmpeg version is?

It's 4.0

>Under options->media, what's your 'video thumbnail %' at?

35

>If you regen some still images, do you ever get this same problem? How about gifs?

Nope

>If you turn on help->debug->profile modes->pubsub profile mode…

Ok, I had a tab with 40 webms with about 5 of them having broken thumbnails. Turned on the profile mode, ran a regen on all of them, turned the profile mode off. One of them (the 24th) generated with a bad thumbnail, the rest were fine. https://pastebin.com/VFVbHHwr

If I regen one webm at a time it's very rare that it ends up being a bad thumbnail but it has happened a couple of times.

Lol, thought this was funny

So i'm going through all the threads from when the tornado took power out and hydrus would shit itself on to many things to do at once, I had a good 870 threads to go through that weren't 404 but did 404 during the program not working time.

So far, I have gone though 30830 images, of those only 1355 were missed so far, and there are still another 42000 images to parse. once this gets to /gif/ images that failure rate is going to go up dramatically.

>>10123

You could start with making sure it works with every booru because right now it doesn't with most of them

You should really allow to check threads faster than once per minute, even if you personally disagree with that. Currently a single check per minute is the fastest you can get (as far as I can tell).

Even then official 4chan API documents (github) states that updating a thread every 5 seconds is okay. Please at least allow this, there's no need to make it the default.

>>10160

Nevermind, you don't even allow that, you _force_ it to at least 3 minutes for static checks. Could you please stop doing that? Let people configure the update interval to how they want. This is especially important on fast-moving boards like /b/ where threads quickly 404.

few releases ago, I asked about a way to lock the tabs in place, so they don't jump around, the issue seems to be getting worse for me where I click on one tab, it jumps out of the page of pages, and is generally a pain in the ass to put back inside. just a menu tab to lock it down would be nice, that way I can unlock when I want to re organize, but other then that, they stay put.

>>10147

Thanks for this feedback. I will keep this in mind.

>>10151

Thanks. That pastebin looks all correct, so I am not sure what is going on here. Now the downloader overhaul is done, I expect to revisit the larger-scale en masse reparsing maintenance stuff, which will do this regen stuff in idle time. Please let me know how that works for you and if you discover anything new.

>>10155

There's some more here, although I didn't make them, so I don't know how well they work atm:

https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Scripts%20-%20Tags

>>10160

>>10161

I am sorry you have had trouble here. I don't want to set the minimum time period too low because the gains are limited and it wastes a lot of CPU and bandwidth both clientside and serverside. It can take a decent fraction of a second to parse and process a thread's json, so if you had perhaps 100 good-size threads all checking every, say, ten seconds, you could easily accidentally fall into a situation where your GUI hangs due to all the threads doing redundant json parsing over and over.

At the moment, for dynamic checking, the minimum check time ('never check faster than') should be 30 seconds. If it is stuck on 1 minute for you, try just lowering it to 0, and the 30s should fill in automatically–let me know if it doesn't. I feel this is a good throttle to stop hydrus being overwhelmed. If you set the min time to 30s and 'intended new files per check' to 1, then if your /b/ thread is getting at least one new file per 30s, the client will automatically speed up its checks to check every 30s. And then, when the thread slows down, it will throttle back (up to your max check time, whether than is 10 mins or 24 hours) and save you a lot of waste.

Are these /b/ threads on 4chan? I haven't been there in a long time–what's a typical thread age? Is the difference between 5 and 30 seconds very important, or is it ok? Or are the images posted in spikes, so the dynamic checking missing the ends of threads because it thought it had slowed down? I am willing to reduce the min time further for advanced users, or alter how the file velocity is calculated, but only if it actually makes technical sense.

>>10173

Thank you for this report. Do you have a busy client? I have noticed this happening during little gui hangs. I have a plan to fix it, and if that doesn't work, I'll add a way to turn off tab DnDs, or only enable with a shift+click or something.

>>10177

Just hang on to them for now and set them as 'alternate' duplicates when convenient. Delete anything you don't like. My duplicates system typically recognises these kinds of related images, but there is no workflow for arranging them into something more intelligent like parent-child or sibling or multi-page relationships. When I next reiterate over the duplicates system, I will add this and we'll be able to process all the 'alternates' into something better.

>>10177

>>10181

More info and examples on duplicate handling here:

https://hydrusnetwork.github.io/hydrus/help/duplicates.html

>>10185

Thanks. This 'resource unavailable x server' stuff is related to wx, my ui library, which still has some crash bugs for linux. It is related to some conflict between my package and your OS (although I build on Ubuntu 16.04, so I don't know what's precisely going on there). Running from source, if you are prepared to put a bit of time into figuring it out, seems to clear them up completely:

https://hydrusnetwork.github.io/hydrus/help/running_from_source.html

If you decide to try it, let me know if you run into any trouble. Some users on the discord do this and can help as well.

Those missing thumbs are unusual. That's how the client displays non-local content (including deleted stuff), but it doesn't usually appear in downloaders, which typically filter out deleted content. Is this a regular download page, and the hydrus thumbs are appearing naturally as the other files stream in, or is it a more unusual operation like going into the file import status and right-clicking->open in new page?

>>10187

Thanks, I might try running from source. How does this impact the performance?

Oh, it might not be such a big problem after all. It's the import page that automatically opened to show new downloaded images. I must have kept the tab open.

When I open a regular search page for the artist it doesn't show the deleted entries anymore. I guess the import page has a fixed list of entries it displays?

>>10187

I've had this issue ever since the big UI update/rewrite. Running from source does not help, but it doesn't seem to hurt anything either. I'm running Linux Mint 18.2, which uses Ubuntu 16.04 for its package base.

On an unrelated note, ever since I updated from v304 to v324, my console logs an error on startup. It says, "Could not import lz4." AFAICT, everything works, but I don't know what lz4 does so I don't know if it's just because lz4 code is never called during my normal use of Hydrus.

>>10189

I get that error too. But lz4 is just some obscure type of compression that's not used in any file formats I know, and it doesn't seem to cause problems, so I didn't bother reporting it.

>>10188

Should be about the same, as the frozen exe is just spinning up a python instance and running the same code, afaik. Boot should be a bit faster since it doesn't have to mess around with that bootstrap environment shit.

That's interesting that that page kept the files through the trashed->full delete transition. They should be removed automatically.. That suggests the underlying file context of those pages is 'everything' rather than 'local'. Or maybe they are loading from db subsequently in a non-filtered way. I'll check this, thank you for reporting it!

>>10189

>>10190

Thanks. If you figure out any new/unusual AssertionErrors getting put to your log (not the GTK-critical stuff, which doesn't give me enough info, but a proper python/wx error trace), please let me know. There are still some instabilities in the linux code, but I don't always get a good traceback in my environment.

lz4 isn't a big deal. I've been unable to package it properly for a few months now due to a PyInstaller update. I burned some hours on it and couldn't figure it out, so I will revisit after the python3 update.

>>10205

Okay, I will have a look at running the source then.