/hydrus/ - Hydrus Network

Ok, 600k image person here, just updated, with this update I should be able to knock around 200k images off the opened image list.

Good

I tried doing that.

A new gallery search, will highlight and unhighlight just fine, works great

The new shit added to the lineup… again, good job, it helps a lot, though I have a 4k monitor and even if I make the preview image portion as small as possible you still have to scroll. I think making this portion of the window tabbed is going to need to happen rather than would be nice if it happened, but its manageable. as I don't need that stuff much of the time.

however already opened galleries will not clear highlight and highlight, the moment you clear them its gone for now.

all the files are there, it remembers everything, you can look at the long, but the files won't display.

I'm going to restart the program now and see if its a quirk of a first time open or if the functionality is currently not working.

>>9536

Nope, still not highlighting.

Im going to leave this as is for now, neither of the searches are things I will miss if this doesn't get fixed/working, but killing off 200k worth of images on a 'maybe this will work in the next version' isn't something i'm willing to do.

that said, are you going to move simple downloader over to a multidownloader?

Not so much anymore, but I use to get a fuck ton of tabs that went through that, and it would be nice if I could consolidate that too.

So looking a bit more, I noticed that other galleries had 'open selected import files to new page' on right clicking them on the file import, so thinking this would be an acceptable work around, I looked at the files, they have nothing there saying they can do this. if I have a small file import from a gallery pre update im going to try one more thing, but I doubt it exists…

Ok noticed another thing, downloaders seem to be going fairly slow.

1 image every 30 or so seconds, and doing everything I can think of to remove any possible limit does not work outside of telling it no, override everything every 5 seconds.

Just added a few threads to download, yea, its 1 file every 30 seconds, but its limited to each watcher, so 20 watchers can get 1 file every 30 seconds, give or take if the file is already in the archive, as those ones seem to make the downloader immediately go for the next one.

Ok, so I missed downloading a thread before it 404ed, something from a few days ago, and I went through the simple downloader, it immediately got to work and went fast (well fast ish for 1 download at a time)

looking at it closer it appears that gallery multi and simple downloader both go relatively fast, using threadwatcher on 8chan seems to also go fast, though gallery watcher for 8chan seems to break with very low traffic 'this thread has been here for year' threads, it seems like simple downloader 8chan setting managed to get everything.

Ok after much dicking around, I set a custom rule for 4cdn.org

100 requests a second and 2gb every 10 seconds. this appears to have solved the issue, however why was there an issue in the first place? I don't understand what was the hang up at all, all my settings globally have impossible to meet requirements unless hydrus starts doing multiple files at once per watcher. problem seems solved, but I have no idea why there was a problem in the first place.

One last thing before I forget, file import options.

I said last time that it wasn't working, however this version it does… kind of.

The first file import the threadwatcer does not give a damn about this.

the second one below that it does care about and you can set it, close then re open for the files.

after some more dicking around mid sentence, I hit apply options and it worked without opening the watcher.

This is going to be personal preference, I have use for being able to see everything, and then only seeing new, opening options and changing it then applying it, and then reverting it back is a pain in the ass. If its possible to have a 'highlight new' option, that would be much appreciated.

Add vim shortcuts and I will suck you're dick.

>>9533

Pretty cool update.

The multi gallery downloader thing makes me wish I could also use it with filesystem paths, it's in most ways more elegant than waiting for the popup dialog to populate the specified folder / files and only then importing.

Editing my subscriptions [which by now has dozens of entries and encompass some hundred thousand downloaded files] seems quite liable to get Hydrus crashed/ OOM killed.

Even if it does not for once [mainly works if no pages are open and nothing else is going on], it takes tens of seconds to load this dialog and most sub-dialogs, or to close them with apply.

And it stops all subscriptions from downloading meanwhile, not only those being edited.

I did:

new page->download->gallery->Gelbooru "afrobull"

The result:

Files get downloaded but no tags applied. I made sure that default tag import options have "get tags" marked.

Using the same with danbooru or sankaku works.

I have this problem since 314 but am scared to downgrade in fear of losing my db.

>>9549

I forgot to mention, I nuked the db folder in hopes that it was my config but it didn't help.

The 1000 file limit wrecks my workflow. On one of my hydrus installs I'll come back after a month and have it catch up on what I've missed and add it to the queue to pull in over a few days.

Arch user here, I just wanted to thank you for this update and the mention that we should use the download page for initial sync. Can you also make the subscriptions work in pages? I think the popups were the reason I got graphical glitches and system freezes in the past. I don't know what wxphoenix is doing with popups but they seem to spam X11 memory/vram until it can't deal with it anymore. I'm kind of on the low end with this laptop and vram is shared with system memory so it could also be shitty hp engineering again.

Now that I don't have subscriptions running for minutes, it doesn't takes my system down anymore, thanks dude.

>>9554

IfI had to take a guess, its because it read the download list and tried to display the images but the images were gone.

I don't mind this personally, as its better than something not working in a several thousand selected file move

>>9537

>>9536

>>9540

>>9541

>>9543

>>9544

>>9548

I am not sure what is going on with your files not showing with highlight, but in the conversion, some legacy stuff may have been lost. The highlight consults the file import cache directly every time it loads (and it obeys file import options 'publish' rules), whereas the old system just remembered whatever thumbnails were in the panel (so you could do something crazy like drag and drop thumbnails in there from another page and it would remember). So if you maybe culled the files from your file import list, they may now be forgotten, or if your file import options say not to 'publish' archived files or something, then those files may not appear now on subsequent 'highlight' calls.

For your 30s delay, do the affected galleries say 'overriding bandwidth in 30s'? That's a common bandwidth delay that comes into play in certain situations when the client is near the bandwidth limit for a domain. If you open up network->data->bandwidth, do the relevant domains look blocked? Or, on that same panel, if you check the 'default' button's options for a 'downloader instance' entry, does that have anything particularly restrictive. There are a bunch of different bandwidth rules to stop shit running away if you run like 100 queues at once or accidentally set up a queue 100k long, so maybe these are firing strangely or maybe they are working fine and stopping your client going completely bonkers.

Ah, reading your next post, I see it was a bandwidth rule issue. I do not recommend 100 rqs a second. The default limit is 1 rq per sec per domain, which is a nice bottleneck that slows down big stuff and stop accidents that might hang the ui as well. If you had 30 threads running, they'd likely slow down to each, on average, getting one file per 30s. If you start banging servers ten times a second for like two hours straight, you may run into automatic cloudflare-or-whatever filters and get yourself IP banned. If you do a lot of downloading, I recommend letting the client download slower over a longer period.

Yeah, I think I'll add an option for something like 'if I enter a new query/url and none is highlighted, highlight it please'.

This 404 issue does not look good. Thank you for noticing and reporting it. Obviously it should stop checking after 404–and as far as I understand, is usually good and does stop–so I will make sure to check the logic here. Again, I recommend not having high bandwidth rules, as if something goes wrong and the client does do something stupid like checking the same url over and over (or getting lost in a subscription and checking a whole bunch of needless gallery pages), you can get very large numbers like this. Since you are a heavy user, I do not recommend checking every 3mins as a flat rule–instead let the checker check in a range of say 3mins to 24 hours, then if a thread falls off, the checker will only be hitting it once a day until it dies, and if shit goes wrong like this, you'll only tick up a few bad hits rather than thousands. The 'default' buttons in the checker options offer some reasonable rules that will work for most situations. This spammy requesting is likely another cause for the slowdown you saw with more reasonable rules.

I'll also be adding options to let you customise the characters there–you are supposed to see pause/stop characters in unicode in those columns, but I see they aren't rendering right for you.

>>9542

Yeah, I am encountering a similar problem with json APIs, which are requested with like 'offset=50' but then don't give that position as something parsable in the reply, and trying to write some pipeline for 'count how many posts were added and then add that to this number in the url we started with' is just way too complicated to even work right. In your case, even if you can grab that strong, I don't have a 'get next sibling' in the html parser yet. Even if I did, it feels a little dangerous, I am not sure.

So, I think the better solution I am going to go for is adding a 'position index' definition in the gallery url class. So you'll define 'the third path component' or 'the "page" parameter', or in this case 'the "currentPage" param' as the index, and then say 'it increases by 1/24/50 every page', and then if the parser fails to find a link, the system will supply what the url class guesses. This is basically how the old system did it, but I'll just use it as a fallback. I have to think about this a bit more (I may want to let the parser raise a veto to say 'I am certain there are no more pages, don't fallback'), but I think it is what I will do.

>>9545

Hey, I do not know much about vim–can you say what sort of shortcuts you would like?

>>9546

>>9547

Yeah, the original file import dialog is one of the oldest things I wrote, and now I have the file import cache, I wonder if I should pass the mime parsing step to the import, where it is redundantly re-done anyway. This would give us better error preservation for the unparsable files and all sorts.

For your subs, I recommend trying the 'compact' button on the dialog. This should cull old files from your subs and make the whole thing snappier. If your subs are all separate (i.e. 1 query per sub), you may want to 'merge' them as well. Having 10 subs each with 100 queries is easier to handle than 1000 each with 1.

Yeah, the 'pause subs while dialog open' is a necessary evil–it is way to complicated to try to update the dialog with new data while the subs are working and handle the consequences of editing via the gui and then hitting ok on a running sub, so I have to atomise it to doing one task at once.

Let me know if the 'compact' does it. I expect to make it happen automatically in future.

>>9549

>>9550

Thank you for this report. Can you please check each of these locations:

Under network->downloaders->manage default tag import options:

- The 'default for file posts'

- any default for 'gelbooru file page'

A gelb default would override the basic default, but it will only exist if you created it, so you can ignore it if none exists.

If it has 'get tags' checked, does the button aside say 'all tags'? If you click the 'cog' icon, are tags being applied to all files, and not 'only tags that already exist'?

If you 'highlight' the afrobull query and scroll down to look at the highlight panel, do the tag import options there look good (i.e. either set as 'use default' or have the same explicit 'get all tags' settings)?

If it is a help, the tag import options set on the multiple watcher only apply to new queries, so if you set the good tag options after you added 'afrobull', it wouldn't have applied. Clicking the query and then 'set options to queries' will do this, however.

Lastly, if none of that works, please hit network->downloader definitions->manage parsers, find the gelbooru 0.2.5 file page parser, open it, and then put in an example url to fetch from and try the 'test parse' button–do tags come up?

>>9552

Thank you for this report. What kind of limit would work for you, and what kinds of tags are these subs for? Is it stuff like 'blue_eyes' and 'smiling', some generic tags, or is there a different combination of tags I am not thinking of that produces more than 1000 files per month or so? What would be the 'worst' of these firehose tags, in your opinion, and how many does it produce per day?

And would it be reasonable to run this client faster, like once a week, or that not possible for technical or whatever reasons?

>>9554

>>9558

Thanks, yeah, I'd like to do both of these things. There's a bit of code already in to do shuffling watchers and queries between the pages, so I'll just have to figure out a drag and drop or cut/paste solution. But it might be a too-big pain in the neck to do drag and drop quickly, so I might end up having a 'make this page the kind of queries, have it suck up all the queries from other pages' in the meantime. Would a 'consolidate everything here' work for you, or would you like to have multiple pages running at once and shuffle between them?

For the dead files, yeah, the 'publish to page' stuff doesn't filter them out yet. It is the same for the multi-watcher. I am going to revisit the way that queue loads this week, mostly to make it faster, but I'll see if I can ditch these missing files as well.

>>9557

Thank you for this information. Yeah, I've had trouble getting floating stuff to work right in several flavours of non-Windows. Having subs run in pages is interesting. I don't think I have time to write a whole new page for them and figure out all the new back-end that would be needed, but if the popups have been a pain, maybe I can figure out a debug-tier test option to hide sub popups while they run. You can have them 'publish' their files to a page now rather than a popup button, so if you are always letting them run in the background anyway (I certainly am), maybe the popups aren't really needed for advanced users unless something goes wrong.

Btw: if your existing subs are pretty heavy with thousands and thousands of files, try selecting them and hitting the 'compact' button in the edit subs dialog. This will cull old files that aren't needed for file velocity calculations any more and make the whole thing snappier.

I am going to give the outdated subscription help file a pass in the next week or two. It will more explicitly lay out how I mean for the whole system to work now with initial syncs and then subs for catchup. Did you ever get pointed to that, and if not, where would be a good time for me to put a big arrow to it for you? The edit subs dialog has a little help in the top corner, but would it be better if I say tracked the first time a user opens that dialog and pop up a thing saying 'yo, check the help here m8'? Or do you have any other thoughts on this?

>>9570

Thanks, i did everything you asked and every tag option looks fine and is the same as in the parsers that download tags.

I thus did the parser test and no tags showed up in the parsed results while the html preview did contain them. Here is the data:

>network->downloader definitions->manage parsers, find the gelbooru 0.2.5 file page parser, open it, and then put in an example url to fetch from and try the 'test parse' button

I added https://gelbooru.com/index.php?id=4341231&page=post&s=view to example urls and as test url, I fetched it via "fetch test data from url" and "test parse"

"fetch test data from url":

PREVIEW:
<!DOCTYPE html><html>
	<head>
		<meta charset="UTF-8">
		<title>1boy 1girl arm arm at side arm behind back back bare arms bare back bare legs bare shoulders barefoot blonde hair blue eyes blush bracelet breasts cliff closed mouth commentary covering covering ass dress dress lift english commentary feet female floating hair from behind full body green hat hat hiding highres jewelry lasterk legs link long hair looking at viewer looking back medium breasts neck necklace nintendo outdoors pointy ears princess zelda rock semi-transparent shadow sideboob sky standing strapless strapless dress sundress the legend of zelda the legend of zelda: breath of the wild white dress wind wind lift - Image View -  | 4341231 | Gelbooru</title><link rel="stylesheet" type="text/css" media="screen" href="responsive.css?14" title="default" />
		<link rel="stylesheet" type="text/css" media="screen" href="css/jquery-ui.css" title="default" />
		<link rel="stylesheet" type="text/css" href="css/jquery-ui.icon-font.min.css"/> 
		<

"test parse":

*** 1 RESULTS BEGIN ***

downloadable/pursuable url (priority 75): https://gelbooru.com/images/7d/67/7d67a721357cc49258551834444582ed.jpg
downloadable/pursuable url (priority 50): https://simg3.gelbooru.com//images/7d/67/7d67a721357cc49258551834444582ed.jpg
md5 hash: 7d67a721357cc49258551834444582ed
source time: 2018/07/28 15:00:31
associable/source url (priority 0): http://www.pixiv.net/member_illust.php?mode=medium&illust_id=69908210

*** RESULTS END ***

Do you think it serves different pages depending on region/timezone?

It also often stopped downloading after 42 images and I'm not sure how to deal with that.

Thanks for your patience

>>9573

I started a new db with only one danbooru sub with 12 queries after downloading each of them. The freezes ONLY happen when I switch back to the hydrus window, sometimes for a few seconds and sometimes for a long time. It doesn't matter if it is finished syncing or not it seems that it waits patiently for me to get back to the window to give me a nice dose of freezing.

I started hydrus while beginning writing this and it froze for a few seconds when I took a quick peek at it while it was syncing.

Syncing is now finished, I can't see the results of the sync due to the popup glitching out but I think it downloaded 10 pics. It doesn't have any freeze problems for now.

In essence:

Syncing while window in background or if it is active/foreground:

No freezing

Switching active state of the hydrus window when it synced while in background:

Freezes everything, including mouse and desktop for whatever amount it deemed necessary.

Additional note:

There is no system ram shortage on my end and the taskviewer also doesn't show any unusual spikes.

I think it must be vram related due to the graphical glitches that happen. It mosaic-ed my whole desktop one time and I had to completely kill X11 to get back on track (the kernel terminal was fine).

I think a quick and dirty way to try and fix this for shitty laptops like mine is to just not show popups at all during syncing, perhaps replacing it with a busy indicator, and only output the result when nothing is scheduled anymore.

Thanks for your efforts, dude. I really appreciate the work you put into this program.

>>9579

Also

yy to Copy the file to the clipboard.

https://vim.rtorr.com/

>>9579

O open externally

o open a new page

Sorry.

>>9579

Could you write a whole specification for that?

I have a thought

granted with highlight coming to multi watcher and gallery the use is now more limited than before but it may be something worth thinking about.

would it be possible to have a low resource mode? let me put it this way, let's put it this way, my current session is eating 13.5 gb, but while i'm not actively doing things with files, I really only need the function of the watchers I could branch off the program put all this shit with images in a page of pages, save it and thread watchers in another, save it, and just reopen the watchers… now that I say that, I may do that till im able to properly cull images, but putting that aside, would it be possible to have a loading mode that uses the current session, but just doesn't load images… it keeps all the shit necessary to load them remembered, it will append any new tabs or things added, and when you want to get into images, you can load the full image set?

granted a full move to highlight on all modes would kill any need or use this has, but I mention it because it seems like it could be an easy to implement feature.

>>9586

gg To go top would be nice.

>>9576

Thanks. Yeah, I think I am just going to write a simple alternative for popups for non-windows people. Floating windows are just the biggest pain to get working multi-platform. I apologise for the inconvenience.

>>9579

>>9580

>>9581

>>9585

>>9586

>>9589

Thank you for this information. The new shortcuts system I am slowly migrating over to does not support double-shortcuts like ZZ, but I hope to eventually move pretty much all events over to it, so remapping arrow keys for movement should be doable in the future. I am drowning in work atm, but as the downloader overhaul comes to a close I should be able to revisit some of these longer term rewrite/cleanup jobs. Let me know how it works for you as I roll it out.

>>9587

I would like to rewrite session loading to make is less resource intensive and do delayed loading like you suggest here.

But yeah also moving to multi- pages should massively reduce the memory burden for downloaders that would otherwise have like 3000 thumbnails attached. Now the db just has to load the file import status list, which is basically just a URL and a couple of numbers per item, rather than all the tags and so on that go with it.

I have some more ideas here to relieve gigantico-clients, but it will have to wait for the next pass on the downloader overhaul.

>>9579

>>9580

>>9581

>>9586

>>9589

Let me get this straight. You people LIKE vim? Hell, I use it every so often, but only if I need to make a super-minor change and don't feel like waiting for a graphical text editor to load. Or, more likely, it can't load because I FUBARed my system and need to fix it.

Hell, I've always thought that all of those people talking about and comparing text-only editors were either old farts who worked on mainframes in the 1980s or people just joking around.

I'm not arguing against improvements, mind you. I'm not even arguing about priorities. I'm just confused as all hell.

>>9587

Low-resource mode is a great idea. With the old download system, I've long wished to be able to somehow page out only part of the memory Hydrus was using (yes I know paging doesn't work like that, but let me dream goddammit). At one point I actually bought more RAM so Hydrus worked better, especially when also doing something with my computer that's not Hydrus.

>>10150

Shit, I thought this was the new thread.

>>10150

>I'm just confused as all hell.

Why? There is a lot of people who uses vim as their primary text editor, people really really like it, there is even a game to learn Vim, the community is kinda like Emacs (but I think Emacs users are more dedicated to their text editor), there is a lot of programs that use vim or emacs shortcuts, one example is the Terminal, by the default they use the Emacs shortcuts but you can set it to use Vi shortcuts, I prefer to only use the keyboard so most of my programs use the vi keys (I use them even in my web browser) , I try to do the same with hydrus but it ins't completely possible at least yet.