/hydrus/ - Hydrus Network

Ok, before I install the new version I have an issue that may become a problem down the line depending on how people do things.

with thread watchers, it's literally who gives a shit, they are largely a mishmash of images

however with galleries, I went with one artist and did an all the usual suspects search, which resulted in 8 different pages up all with their own files.

would there be a way to combine them into a 'multi' source gallery that is highlightable?

I ask because one of the workflows that hydrus allows is seeing a person on exhentai in an artist round up, and when they have no torrent, or the torrent is out of date by quite a bit, I can go to booros and other places to download from the source, however, if an artist have a file list of 400 on the exhentai but from e621 they have 200 and some others they have a few hundred more, its hard to be able to see at a glance if you have everything of theirs or at least a number that suggests that you got it all

Ok now downloading it and playing around with derpi, I see no immediate issue, however there likely will be something you are going to want to sort out with it.

derpi has a setting I abuse the hell out of with gifs, because gifs on that site are obnoxiously large a lot of the time, where it will serve you a webm instead of a gif, barring a few exceptions, I would rather have the webm of a gif it means it goes from 20mb down to 3 with little noticeable difference.

If you go this route, and if or would probably be for the best

if x frames go gif, or if x size go gif because there are images that break if you go webm over gif when they are to small. I don't know if this is to much for a parser to do though

>>9611

To add to this one, derpi has a fun system that has this tag

first_seen_at.gt:3 days ago

with a

sf=score

sd=desc

together looks like

first_seen_at.gt%3A3+days+ago&sd=desc&sf=score

which sorts everything added in the last 3 days, sorts them by score they get and then in descending order.

What I want to do at some point is make a subscription to this, how I would do this im not sure, but as the 'first_seen_at.gt:3 days ago' search has EVERYTHING seen 3 days ago it currently totals 2548 images, Is there a way to limit the search depth to either a page number or to a 'find image amount' number?

aside from wanting to bulk download some artists and tags on derpi, getting every image as they come in with a relatively high score in a short time frame pretty much allows me to not need to keep visiting the site on a frequent basis.

been using the gallery a bit more today, and I have found an issue. i currently have 44 things on galleries, and when I add a new thing, its a bit of a pain in the ass to find what I added. as there is no clear cut straight to the top label like there is with time on watchers. would it be possible to add a time stamp to the gallery window for when it was added, or even just a number to the left/right like there is on the log? this way you could sort by most recent in a very clear way.

Right now the best way is to sort by status, but the moment its done it falls into the fold and is hard to find unless you remember you what you were looking for, at least in my case i'm going through quite a few at once getting rid of a queue, I can only imagine this gets worse and worse as time goes on

also, is there a way to have it recheck the source without re entering the names?

going through hentai foundry a bit more now, it seems like the new gallery mode has it fail to check often enough I have to keep resetting login to get it to do anything. with that said, it is possible to have the program periodically cull gammery mode where there are absolutely no results? I would like non results to stay long enough for me to know I got a non result, but remove itself if I did after a time, possibly a user toggleable function.

on that note is it possible to easily copy queries or re submit the gallery for search? there are a few artists t where the name you search them by, lets say on tumblr, is different from their creator name, if I can right click and copy the search name that would be helpful, especially when dealing with large amounts of sources.

>>9609

You should try tbib. It's a booru that syncs from a couple others, so instead of searching each booru you could do one search on that.

>>9617

what boorus do they host?

also, https://jlullaby-stuff.tumblr.com/

Not sure if tumblr is broken right now or if parser is broken, I only had 2 successes because the sources were recognized in hydrus, all the rest errored.

just noticed something, is there no way to log into hentai foundry through hydrus? I just noticed there was a person I was trying to download and half their images weren't coming, I logged out and saw only 7 of their images were visible opposed to 14

Well… If my last few posts are any indication you know exactly what I'm doing. Is there any way in the gallery downloader or in the multithread watcher to have some kind of indication telling you what your bandwidth is and what it allows? Honestly interested in seeing where the bottleneck is on this, if it's the program itself bottlenecking, if it's the servers bottlenecking because they can't respond fast enough, or if it's just some setting that's bottlenecking it.

Granted this would be a great tool to be a complete ass hole if you wanted to and I can't really deny that I would be increasing the settings right now to get hentai foundry to download faster if I knew exactly how to do it, I just got around to looking at a massive download list from them I have about 469 artists to fully download, and then login system happens 469 artist a fully download a second time.

>>9620

tumblr is broken because the soyboy retards disabled the _raw file pages. again.

>>9623

the program has an option to manage the data caps by site. You should, however, moderate yourself. we can't all download a terabyte of content everyday.

the option for modifying it should be under Network > data

>>9624

>tumblr is broken because the soyboy retards disabled the _raw file pages. again.

Motherfucker. That explains a lot. Does this happen often?

>>9625

https://e621.net/forum/show/235250?page=11

>>9624

seeing what is causing the bottleneck, either requests on hydruses end, or the websites speed in responding to them would be worth seeing if only to more easily be able to see what the bottleneck is.

also probably related to tumblr going straight up its own ass

Exception

The subscription Shortstack encountered several errors when downloading files, so it abandoned its sync.

Traceback (most recent call last):

File "include\ClientImportSubscriptions.py", line 1347, in Sync

self._WorkOnFiles( job_key )

File "include\ClientImportSubscriptions.py", line 525, in _WorkOnFiles

raise Exception( 'The subscription ' + self._name + ' encountered several errors when downloading files, so it abandoned its sync.' )

Exception: The subscription Shortstack encountered several errors when downloading files, so it abandoned its sync.

Please bring back the option to drag&drop files without them being ordered alphabetically by default in the import window

I used to sort by time created before importing in hydrus, but since 315 i can't anymore since it'll default to name order

>>9620

>>9617

>Signup failed. IP was flagged by stopforumspam.

Block VPN nodes from making accounts, very likely blocks Tor nodes too.

Into the trash it goes.

Isn't rule34.xxx syncing from the main ones already? And it doesn't block VPNs.

Some questions/requests for multi-watcher:

Can multi-watcher get a button-per-line to open up a single thread's imported/applied tags individually rather than having to add thread-specific tags to top-level tag import options and swapping them out every time you add a new thread, for the new tag? It would still be good to have an overall tag application for the multi-watcher, but also a more detailed level rather than having to wait for them to be done, remember what tags need to be added for that thread, double-click, Ctrl+A, right-click, etc, etc

If we do get a per-thread tag import options, can we get an option on the master tag import options to just check a box and it'll add subject:[name of thread here] to all of the files? I'm guessing that can't be too hard to do since the multi-watcher always pulls in the thread name anyways. This is like a batch ID and content indicator in one so it would be really useful to have as a default autotag option.

h

>>9630

Also can we have something like this for subscriptions? Multi-subscriptions is great but I can't use it because my tumblr:accountname tags have no way to apply that I know of if they're all under one common tag, unless maybe I can change the tags as I feed new ones in using the same awkward method as multi-watchers? I'm not sure if it works like that and it's sort of error-prone entering tag data that way.

Worse, I can't easily lookup by source because as we all know websites like Tumblr have fucky inconsistent source folders/subdomains.

>>9610

Can you give me an example of one of these derpibooru gif/webm pages and the alternate download link? Or is this a setting inside derpi associated with your login? If I go here:

https://derpibooru.org/1803434

The download links still serve gif, but I notice the actual content embedded in the page has some unusual 'data-uris' values that point to webm/mp4. Or am I just not seeing the 'download as webm' button?

>>9611

Thanks, I will check this. I notice your derpi URLs are getting 'index' as their page_num, rather than 0, 1, 2. Maybe my code is messing up here.

>>9614

Sure, I will add a timestamp.

Stuff like 'restart this search' will have to wait a bit for me to do the new 'searchers' object, as search initialisation is still on the legacy system.

>>9620

>>9624

>>9625

>>9626

Yeah, they done fucked us again. As far as I can tell, it looks like they changed some settings on the AWS bucket(s) and now you need some cookie authentication to see what's inside their CDN. It might be an attempt to stop people like us and e621 from getting the originals, or it might be a standard upgrade that tightened up the old hole. What a pain in the neck, I just wanted to play videogames, etc…

I'll roll out a different parser this week that grabs the 1280px URLs again and then keep my eyes open for a new way in. Please let me know if you discover a new URL format that works.

>>9622

It looks like you have had a hard drive problem that has damaged your database. Please check out 'install_dir/db/help my db is broke.txt' for some background reading. You may have additional problems beyond this missing table.

The missing table here is not very important, but it is not easy for you to fix. I will try to write an auto-healing routine for v318. Please check your hard drive is healthy in the meantime, roll back to a recent backup if you have one, and let me know if v318 lets you boot.

(If you have a backup, recent or not, let me know, as we can do a better manual migration fix here.)

>>9623

You can try help->debug->report modes->network report mode, which will spam you with every job getting scheduled. I'd like to expose the per-downloader bandwidth managers at some point, for advanced mode users.

>>9621

I think the HF login system (which basically clicks-through the "18+" landing page and then sets up a 'view all' filter) may be broken. I would like to replace it with a proper user/pass login once I am done with this overhaul.

>>9628

Please check the new 'cog' icon on the file import window. It should have an option to turn off the sorting, and it should remember what you last set it to. Let me know if it doesn't work for you!

>>9630

>>9631

I am not sure if I understand this. Can you describe the workflow you are going for here further, maybe with an example? Do you want to quickly add like 'ylyl' tag to a thread of results? Atm I recommend waiting for threads to be DEAD/404 before adding tags or other processing metadata, as it is usually easier to just go Ctrl+a->F3 on the final results before you dismiss the thread entry from the larger multi-watcher.

>>9642

Thanks, and with search retry its more of a down the road I let this shit sit for several weeks/months issue

>>9643

Ok, now im currious, would it be possible to have a 1280px and a raw grabber, and have the tumblr gallery do an auto dup detector pass to eliminate the lower quality version, this way the when it works, it gets rid of the lower quality version when it can get a higher quality one… there may be some other way to do it without the need for dup detector, but getting all the links for the 1280px and raw, then discarding one of them when you see the higher quality one is there should be possible.

>>9645

ok good to know, I have most of the artists I have been getting from in a txt file for easy re grabbing, but god damnit its 480~ artists big…

either way this was a long time coming and something I was going to do at some point with gallery multi mode, it just made it very easy to do now.

on that note, hydrus does not like having 20-50 watchers dumped in it at once, it hangs quite a bit when I do that. never really noticed from 4chan as the threads were so small after the initial dump that by the time i grabbed 1 boards images and went to another board to look for new threads, the first operation was usually done.

Just went back to 316 thread

>>9639

I use to have 600k some images open at any given time, after the post I made I went and culled through everything,

Now I have maybe 10k images at most open, a fuck ton of watchers sure, but less then 10k images, the program is eating 6.5gb of ram.

I'm going to go through the images and cull them once the hentai foundry download is done, restart the program, and see whats up. but honestly, even in a worst case scenario, 1000 of the largest thumbnails in t00 only come to 53mb

you know with that math, my hydrus was eating up 15-16gb at times, and that's the equivalent of loading up 150k thumbnails which I know the program was not doing.

I think there is something bloating larger dbs, but I wont be able to comfirm for a while.

I'm probably just retarded but does anyone else have an issue where the gallery parses only the first page of a tag. I've only tried it with gelbooru so far but it only ever gets the first 42 images then stops.

I can't seem to find a way to fix it and looking through all the changes my illiterate ass can't tell if this is now intended or not. Can anyone help or clarify?

Looks like I'm late to get a response from dev, but I'll post this anyway.

Can you add (if it's already available, tell me how I can) add a sort of filter like the archive/delete filter except it either does or does not add a tag?

Say I want to go through several hundred untagged files and add the tag "solo" to images with one character in them. I could activate this filter for the tag "solo" and breeze through all the images, tagging ones that apply.

>>9653

if it's like ratings, you can make a tag hotkey, highlight all of the ones you want, and press the hotkey.

Personally I do this with ratings as ratings are far easier to deal with then tags at the moment.

>>9652

yea, gel is stopping at 42 images, I have 2 searches from them in gallery, both of which stopped at 42 despite having 6 pages + of I think 44 images.

jlullaby

and

deepthroat x-ray

>>9647

Yeah, previously I would wait for threads to finih downloading from archives and then tag them, but it's a bit tedious since I have to name each tab with the tags I want to put on all of the files for the thread, lest I forget, since I can't keep everything open in 4chan X after it's already dead, it lags too much with so many threads in the queue even when it only shows threads per current board.

I was just pre-tagging each thread watcher and had a tab for each thread before multi-watcher, I guess I'll go back to just using one multi-watcher tab per thread. But it would be nice to have the thread subject auto-importable as a subject: namespace tag since it's getting pulled anyways.

>>9646

Thank you very much, didn't see the new icon, it works just as i wanted

So the link that appears underneath my ratings on an image (e.g. Sankaku), is it possible to set those to open in a specific browser rather than my system default?

Thanks

>>9651

>1000 of the largest thumbnails in t00 only come to 53mb

Those are compressed jpgs, most likely thumbnails are stored uncompressed in memory.

>>9676

which would mean some program re writing should be in order

————

that said, hdev, got a question, now that the hentai foundry mass download is done, at least for now, I noticed that hf has 2 popups for each one, a scraps and pictures.

would it be possible to set up a search parameter that would hit everything at once, or at least an all the usual suspects one?

I just came across a few artists who have tumblrs, and because tumblr is fucked, I put their name though all the boorus that should turn something up, I dont see a reason something like this cant be implemented, but would it be user doable?

like a custom gallery, and then with check boxes, all the searchable galleries that hydrus has to offer.

>>9677

I should clerify, this part

>like a custom gallery, and then with check boxes, all the searchable galleries that hydrus has to offer.

Could have a one time search feature, but I mean more a ui element that would make making custom searches easier, you hit boxes and it makes it and saves it for use later though selecting it like the program has now.

>>9650

Hey, it looks like that clone–or the subsequent file arrangement–did not work out well. That's an error that is like "Hey, I just tried to create a new database from scratch but some parts of an existing one were already there." This suggests the clone either wiped out a really important table or maybe client.db got removed from the db dir while you were making the clones and rearranging files? Please double-check you have the right db setup:

client.db

client.caches.db

client.mappings.db

client.master.db

If any of them are like 58KB and look like they were created when you attempted to boot and got that UNIQUE IntegrityError, I think maybe that file was missing and the client attempted to make a fresh file.

The clone operation would ideally have been something like:

- make clone of client.db called client_new.db

- move old client.db out to a safe folder somewhere

- rename client_new.db to client.db

- attempt boot

And replace 'client.db' with 'client.caches.db' or any of the other files you attempted. Did the renaming step get missed?

>>9651

>>9676

>>9677

Thanks. Thumbnail bmps are only loaded once they need to be drawn to screen, in the fading 'waterfall' effect. Old thumbs are unloaded as soon as the limit in options->speed and memory is hit. I think the default there is a couple hundred MB. But every media object behind the thumb does take a little memory, I would very roughly estimate maybe a handful of KB per object for a really heavily tagged file? And there are all the lists and quick-access indices that go with the results. That was probably a decent whack when you had 100Ks of files open.

Now you are pared down on the ui end, I guess you have some db stuff hanging about. For instance, there's a big quick-access list of all inbox file ids open at all times, which for a client with 100,000 in the inbox is usually no big thing, but if you have millions, I guess it would add up a bit, maybe 100MB? A few of those sorts of things in a row might add up to 6GB, but I am not sure.

I looked at adding a memory profiler a few weeks ago, but the one I found that did what I want crashed the whole program as soon as I ran anything on the ui end. I'll keep my eyes open for another. You might like to hit help->debug->data->print garbage and email/post/pastebin the big list it will dump to your log to me. If we discover 10K bitmaps or 10M lists or something, that'll point us where things might be going wrong here.

>>9652

>>9655

Thanks lads, this should be fixed in today's release. It turns out gelb randomly(?) do not include the '>' 'next page' link on some gallery pages, and since the new gallery parser relies on finding this, it was falsely assuming some results were finished. I now have a backup in place that attempts to just add '42' to the last gallery url for gelb.

Please try these queries again in v318 and let me know how they work.

>>9653

>>9654

Yeah, your best bet at the moment is to add some shortcuts to the 'media' set for the tags/ratings you would like to set or flip. Then use those shortcuts while you do a regular archive/delete filter or browse.

At some point I'd love to generalise the archive/delete filter code so you could customise your own filters like

left-click = 'add "sexy" tag' and move on

right-click = do nothing and move on

But I've still got a bunch to do just catching the existing code up to the new shortcut system.

>>9657

If you feel brave, you can dip into the network->downloader definitions->manage parsers dialog and edit the imageboard parsers to provide the subject as a tag. I recommend you duplicate the existing parsers and work on the dupes so you don't accidentally break the originals if you mess up.

>>9664

I was going to say: Yeah, try editing the launch path under options->files and trash. It should work on all urls the program launches.

But then I realised the HyperlinkCtrls there don't obey my option–only the 'open known urls' bits off the right-click menus do. I will make a job to fix this.

>>9665

>>9666

Thank you for this report. That swirl looks like PIL's doing. I don't know if my PIL code is just shit or if they can't handle some palettes, but that's a long-time problem that I eventually fixed by moving to OpenCV as default. Please check options->media–do you have the 'PIL instead of OpenCV' bugfix setting checked for some reason?

I think PIL also steps in as backup if OpenCV throws an error. I wonder if that GStreamer error is related. What PIL (Pillow) and OpenCV versions do you have under help->about?

Yeah, I don't care about nsfw.

EDIT: Yeah, when I turn on the PIL bugfix option, I get the same swirl as you in Windows. PIL is kicking in for some reason.

>>9677

>>9678

Yeah, I'd like to make the new 'searcher' object nestable, so you'll be able to say 'search for artist x', and it'll generate four separate queries on four different boorus. This will only work well, of course, for those boorus that use the exact same search strings for what you are looking for. Seems like gelb and danb would be good mergable candidates for artist names, I think?

The hentai foundry artist search has always secretly been those two streams, so this new system just exposes that, and things are actually a bit simpler as a result now. I still have to think about this nesting as I build the searcher, though. Things like 'should a nestable searcher create multiple subscription queries or throw everything into the same gallery log'. I don't want to accidentally fuck everyone or let new users completely fuck themselves with infinite looping searches or anything.

>>9685

ok I have it and its in, though not 100% sure how to use it

Ok, got it in the parser,

it still returned a gif

Ok, thinking there was a conflict, I deleted the original parser after exporting it,

Now I get there are no parsers

Ok, import the original parser again

No parsers

Delete the new one

No parser

I think I did something stupid and i'm not seeing how to fix it/enable a parser

do I have to restart the program each time or am I just doing something really wrong and dont realize it?

>>9686

The file sizes look fine and I have all 4. I did the cloning/renaming correctly but only did client.db, that's the only one it said in the help txt. Should I do the other 3?