/hydrus/ - Hydrus Network

>>9885

Praise The Lord!

Current Mood.

The time has come.

Ascension into godhood

>>9862

Yeah, zero inbox. I just archive all the imports.

>>9854

Thanks, I didn't notice it before partly because it label is sort of an unclickable line inside the list box.

That label isn't that bad, can't really thing of another one. Maybe take off "services"? I know it's subdivided by them, but might not need to be in the label. It would be clearer (to me) if it were a text line above the box rather than in it.

While I'm here I should tell you that if you have an old import folder and you delete it, then edit the hydrus entry for it and hit Apply without changing the path to something valid, the same popup (or a very similar one, I didn't actually read all the text on each one) pops up four or five times before you're finally allowed to cancel out. Might be intended behavior though, like I said I didn't actually read other than the first two.

I have a low-priority thing to request for the todo also: can you add a potato mode to importing? The full-throttle approach sort of fucks my shit up, if it could import slower with less system resources at a lower CPU priority, that would be helpful.

>>9851

Based, it worked.

ty

>>9866

Thanks. I'll check out the spammy popup on missing location.

I'll be revisiting file import options in the nearish future to do some url- and hash-checking logic options. While I am in there, I'll see if I can add an option for an artificial 'wait x seconds after every import' to throttle things.

A web interface for the server? Hell yes! Hopefully, I will be able to figure out how to set up all that crap. Being able to host my own little Booru from my Hydrus db is something I wanted to do for a while now.

>>9785

>Can you remember what you were doing at the time? Had you recently closed a dialog, or were attempting to? Do you have a 'busy' client with say dozens of pages and downloaders all working at once?

I might be mistaken but I think I was closing a manage tags window. It's not a busy client. Less than 10 pages with few items in them, nothing downloading.

>>9809

I happened to get this error myself this week and think I figured it out. Please let me know if you have any more trouble in v320 on!

I found a cool bug in v318, but not sure what it is or if you already know about it. Here goes:

>I had several multiwatcher tabs open, most idle but one downloading 12 or so threads at once

>I had one simple downloader tab open, downloading from around fifteen threads from an off-site archive

>also syncing tumblr subscriptions at same time

>switch to the dupes tab and run the rebalancing, hadn't done it since before the new tumblr fuckery so I had I think 84 branches to rebalance

>goes for a while, I'm tabbed out reading a thread and messing with jdown and stuff

>tab back in, it's just finished, hit OK, instead of letting it sit a minute and then running dupe discovery, I immediately switch back to the working multiwatcher, but hydrus hangs as it tends to because my laptop sucks, especially after finishing big operations, a message box has popped up under the subscriptions box, but it's blank because the client froze there

>after ten or twenty seconds it resolves itself and refreshes the screen, seems normal, except the subscriptions downloader box and the message box below it are just gone

>figure it was just a message saying it thought it was done syncing for the day, not sure why they closed themselves though

>elapse ten or twenty seconds of normal behavior

>open up tags to enter in new tag for thread I'm about to paste into multiwatcher, enter them and remove last thread's unique tags, hit OK or Apply or whatever, box closes

>client hangs again, this time doesn't recover, but rather than going (Not Responding) it just makes that annoying system bell whenever I try to click on any tab or anything at all

>downloader pops back up and starts, figure it's fine and I can just wait a bit since it's actually downloading

>stays frozen, can't select anything, but downloader still going and there's a rare C++ level error spat out at me instead of Python

>running it in debug, so go to console window and look at it, shows basicallyPost too long. Click here to view the full text.

>>9819

Thank you for this report. This looks the same as >>9782 . The 'modal' popup that comes up during idle maintenance and a couple of other places was being rude to any pre-existing dialogs. The dialog gets hidden but then can't restore when the maintenance job is over, and since the dialog is still modal, ui functionality can't return to the main ui frame. In your case, it was probably the tags window for your new multi-watcher. Maybe some idle job got delayed while the dupe search was going on, arrived late once your db-commit hang finished, and then blatted itself into the ui event loop right when you had that dialog open.

I believe I have this fixed for today, but if you experience it again, particularly with the dupe search stuff–which works on slightly different logic–please let me know. I now know what the error is actually about!

>>9740

some retarded soyboy bitch who couldn't be bothered resizing his files before uploading them to tumblr complained to tumblr that people were getting his patreon content through it because of the _raw URLs, so tumblr removed _raw urls all together, because soyboy bitches reason through groupthink instead of logic.

>>9742

So even the Amazon link fails?

>>9742

would love to know who it was.

>>9735

I don't have that in my version, but I think I remember maybe turning something like that off when I was using it more. There's some auto-backup options somewhere, maybe there are some indexing options near there? Like you can tell it not to check for updates to file structure or something.

That shit was one reason why I started thinking about hydrus, actually. The difficulty of keeping up with a disparate file structure.

>>9737

Yeah, looks like e621 has several examples of these unreliable source urls. I will add options to file import options to change the url check logic here (e.g. '[] don't trust source urls for 'already in db/deleted' checks).

>>9744

Yeah, it is all gone. The amazon links still respond in different ways, suggesting the 'bucket' still exists, but you get variants of 'authorisation missing'. I presume they have basically tightened up their AWS-side back-end CDN to require authentication, so it is back to being a way for their cache servers to grab originals. tumblr are aware of the issue and have strongly suggested it is never coming back, so I imagine if we ever find a new way back in, they will only shut it down again.

This is happening all over the place. My feeling is the future of original source is going to be the various boorus 'inheriting' from the actual original source, be that patreon or some tumblr post that links to HD on imgur or mega or whatever (although if the boorus use CloudFlare, they may be compressing jpegs on the fly! fuck!). In five years, I expect hydrus to have P2P file-sharing components. You can trust that any system I make wouldn't do this shit. But I can also imagine more complicated waifu2x-style systems in future that may nullify most of the issue.

>>9745

A furporn Patreon artist

>>9683

One of the issue is that some accounts are protected. Example: https://twitter.com/uzusio04

>>9493

For #2, dumb-search to smart regex translator? Anyone? Maybe use https://github.com/VerbalExpressions/PythonVerbalExpressions ?

https://www.ebay.com/itm/HP-DL380p-G8-16x-SFF-Intel-XEON-E5-2600-v2-Serverschmiede-Server-Konfigurator/222417019581?hash=item33c91662bd:g:CdgAAOSw11ZZeL8A

https://www.ebay.com/itm/Cisco-UCS-C240-M3-16x-SFF-32GB-4x-8GB-2x-Xeon-E5-2660-2-20-GHz-Raid-Server/222438386802?hash=item33ca5c6c72:g:D0gAAOSwuxFYxqFx

https://www.ebay.com/itm/HP-Proliant-DL380p-G8-Gen8-Server-16x-2-5-SFF-XEON-E5-2600-v1-v2-P420i-Raid-CTO/222751766786?hash=item33dd0a3902:g:TPAAAOSwZq5bqiIR:rk:5:pf:0

https://www.ebay.com/itm/Cisco-UCS-C240-M3-16x-SFF-8GB-2x-4GB-2x-Xeon-E5-2660-2-20-GHz-Raid-Server/222438386145?hash=item33ca5c69e1:g:OYkAAOSwB-1YxqG6:rk:28:pf:0

https://www.ebay.com/itm/Dell-PowerEdge-R720-16x-SFF-XEON-E5-2600-Serverschmiede-Server-Konfigurator/322381837178?hash=item4b0f74737a:g:1zIAAOSwH9lZeL2y:rk:35:pf:0

https://www.ebay.com/itm/Cisco-UCS-C240-M3-16x-SFF-32GB-8x-4GB-2x-Xeon-E5-2660-2-20-GHz-Raid-Server/222438386453?hash=item33ca5c6b15:g:ftwAAOSwsW9YxqGX:rk:46:pf:0

>>9665

>>9666

Thank you for this report. That swirl looks like PIL's doing. I don't know if my PIL code is just shit or if they can't handle some palettes, but that's a long-time problem that I eventually fixed by moving to OpenCV as default. Please check options->media–do you have the 'PIL instead of OpenCV' bugfix setting checked for some reason?

I think PIL also steps in as backup if OpenCV throws an error. I wonder if that GStreamer error is related. What PIL (Pillow) and OpenCV versions do you have under help->about?

Yeah, I don't care about nsfw.

EDIT: Yeah, when I turn on the PIL bugfix option, I get the same swirl as you in Windows. PIL is kicking in for some reason.

>>9677

>>9678

Yeah, I'd like to make the new 'searcher' object nestable, so you'll be able to say 'search for artist x', and it'll generate four separate queries on four different boorus. This will only work well, of course, for those boorus that use the exact same search strings for what you are looking for. Seems like gelb and danb would be good mergable candidates for artist names, I think?

The hentai foundry artist search has always secretly been those two streams, so this new system just exposes that, and things are actually a bit simpler as a result now. I still have to think about this nesting as I build the searcher, though. Things like 'should a nestable searcher create multiple subscription queries or throw everything into the same gallery log'. I don't want to accidentally fuck everyone or let new users completely fuck themselves with infinite looping searches or anything.

>>9685

ok I have it and its in, though not 100% sure how to use it

Ok, got it in the parser,

it still returned a gif

Ok, thinking there was a conflict, I deleted the original parser after exporting it,

Now I get there are no parsers

Ok, import the original parser again

No parsers

Delete the new one

No parser

I think I did something stupid and i'm not seeing how to fix it/enable a parser

do I have to restart the program each time or am I just doing something really wrong and dont realize it?

>>9686

The file sizes look fine and I have all 4. I did the cloning/renaming correctly but only did client.db, that's the only one it said in the help txt. Should I do the other 3?

fantastic, Just get hydrus down to a reasonable image size and then you add derpi… hello 600k+ images again.

cant wait

>>9593

Getting Fiber in next few weeks… looks like I should get a server/NAS for Hydrus!

>>9516

>>9515

>>9517

>>9518

>>9520

>>9521

>>9526

>>9535

Thanks for all this. I am not working on dupe stuff at the moment, but I have written these points down and will keep them in mind. I am overall happy with the first version of dupe stuff but agree that it could really do with a full pass.

>>9519

I don't know much about vim–what sort of shortcuts are you looking for?

>>9531

I was talking to someone else who had this issue a couple of weeks ago and couldn't reproduce it. When I do 4/5/6 on my test machine, I get more each time. The fundamental object that is being tested here is a 8x8 matrix representing some simplified vertical/horizontal waveforms in the image (it is a DCT), so I wonder perhaps if certain classes of image tend to be 'mirrored' in some way on the two axes represented in the matrix, but I am not sure.

I'd be interested if you discover anything new about this. The max potential number here would be 63 or 64 but most of the action is in the 0-8 range, with anything above, say, 12, being completely useless, so I may end up re-doing the shape-based dupe detection to be a 16x16 matrix (i.e. 256 bits) to give us more resolution here, although this would slow search significantly, so it might be better just to put my time into better workflows. The big bottleneck now seems to be human attention (it takes ages to go through pairs by hand), so I wonder if I should attack that first.

>>9532

Yeah, this is highly requested, so I'll roll it in before this whole overhaul is done. There's some stuff here for it already:

https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/NEW%20Download%20System

but I haven't tested it myself and it isn't easy mode yet. You might just want to wait for me to add it naturally.

>>9532

>>9564

Shit, sorry, I read that wrong. Yeah, I'll add a gallery parser and 'searcher' for it when I have that system ready, so it'll just pop into the gallery selector button soon enough!

Thanks, looking forward to it!

Im curious, these don't need to happen soon, just wondering if they are on the docket of things that will happen

is gallery log for multi thread watchers a thing, I ran into a thread where 14 images were deleted, and because I dont delete things currently, I wondered what the hell happened, so I had to load up that thread, ultimately they were images in trash not deleted and they were there for a reason, but opening the thread and trying to find what images did not download was tedious.

and on multi thread watchers, i have one of my multi pages that has 170k images, but 70k are already in db, would there be a way to add a 'highlight new' to the highlight options? I don't think the current file import options works with highlighting, either that or all the threads of 150+ images I just checked with only show new checked all had only new images, which would be shocking.

Like I said, neither of these need priority as my use for them is going to come several months down the road in the case of highlight, or just a nice resource to have in the case of the log, just wondering if they are on the docket of things to do, already in in a way im not seeing, or if they just got added to the ideas pile.

>>9475

Take care of yourself man, get some rest.

some feedback

i looked at the program for half an hour or so, tried to do some things and failed. if you have 150 IQ and are in software design or programming its probably easy to use but for just avg IQ person like myself i found the program to be not very intuitive :(

>>9426

Did you use the help pages? Hydrus has a bit of an initial learning curb, but it functions like any booru so you should be able to do some basics.

>>9405

>>9427

ok, ill study it some more

i dot know what a booru is, i just manually downloaded about 500 memes from 4chan-pol over the last year or so and then i got the program ChanThreadWatch to download some image threads. Was hoping that i could just put about 100 images in a folder and then point Hydrus to that folder and let it go online and find some matches. Looks like it will take a few hours of reading though

>>9425

re: db migration

With the new version I'm not getting the issue anymore, thanks for fixing it.

>>9407

Thank you for this report. The big maintenance routines need to make some large temporary files when they work, so they sometimes need up to, say, 5GB free space. There is a check for this free space for Vacuum jobs but not for repository processing. I will add one.

How big are the four client*.db files in your install_dir/db folder? How much free space is on that drive? How much free space does your system partition have, if that is different (some of the db files are written to your system temp location)?

It may be not doing more maintenance now because it may have successfully 'paused' the repository on the previous failure.

Best solution to your problem is probably to make sure your hydrus drive (and any drive with active files on tbh) is at the bare minimum 5%/10GB free, whichever is larger. Ideally, system drives should be 20% free, but I appreciate this is not always doable. Then check services->review services to see if the repository needs unpausing or whatever to continue work. If it has a more complicated error in its status reports there, please let me know and we'll work on it.

Actually, that "lack of space" is somewhat my fault, I temporarily moved database files from hdd to a ram-disk for them to be processed.

Total size of db files is about 4.5 gb, the disk was 9gb.

Also, public tag repository was indeed paused. After reenabling it, after-exit maitenance was again offered. So, solved ))