/hydrus/ - Hydrus Network

>>10992

—just pasting it here in the new thread too—

here, this is a known link that doesn't pull images up, I tried going incognito to find ones, but apparently it finds the images that way too. for the artist, so I stuck with a known one that wouldn't work.

https://mega.nz/#!KooGTIBC!aRlDdrI1XRA-iMDmkzye7J6Q-C3bTWnb3_74-8-0eDI

Honestly, I think full stopping the download, program wide, till a log in finishes would be a good work around for this. or with the downloaders that these download from, a double tap when logging in, the first page or two of searches could get redone twice, because by then everything should be logged in, and if something didn't catch it would get them in the first two pages.

—————

that said will install the new version right after some downloads finish.

>>11002

Goodbye Youtube-dl, my friend. Hope I can see you when API comes

Deleted my download rules but my Tumblr subscriptions are still stuck on waiting for bandwith for 9 hours…have tried pause/playing them and a bunch of other stuff. Any way to make it just download before I run out of time?

I think I >>10818 screwed up my post back thread version 332. I just realized I deleted my reply to you by mistake trying to correct it so I'm not sure you was able to see it or not.

From your reply >>10856 I did the other stuff and none of it worked. I made the html like you said and looked through it in notepad++ I didn't see anything about "not logged in". I made a pastebin

https://pastebin.com/TtNtZnXi

I'm not really worried about private info since you don't need an email to make an account. You just create a name and password and the site lets you can see everything.

>>11013

Jesus christ anon, what the fuck?

>>11005

Hey, thank you for this video. Please check what you have under network->downloaders->manage logins. I do not think you are logging in with your account, but instead the click-through login. Hit 'change login script' on www.hentai-foundry.com on that dialog and put in your credentials. I actually have some nice new help at the bottom of here on this:

https://hydrusnetwork.github.io/hydrus/help/getting_started_downloading.html

Also, I strongly recommend you clear out your completed download queues. Your downloader there has 400k URLs in it, and the client tends to get slow at about 40k total thumbnails/URLs loaded. That huge list is also going to be a big cause of program lag and other jank. There's the smallest chance it explains the login problem, maybe as some kind of traffic jam, but in either case, please clear out what is finished there–it'll fix a whole bunch.

>>11007

Thanks. This happens sometimes to the popup area, I think mostly due to my jank-ass way of making it appear. It is generally harmless. I'm going to see if python 3 improves things at all, and if not I'll revisit it. I'm also hesitantly entertaining a possible move to Qt for Christmas next year.

>>11008

Yeah, I discovered this week that it can do some video tweets. I really like it and use it every day for all my regular youtube subs. I'd love to one day have it as something the hydrus downloader can throw URLs at and get mp4s back, but the API will also be able to hook the two together. The old youtube solution, which used pafy, could only handle direct video URLs and not streaming DASH stuff.

>>11011

No force mode atm. Check what is 'blocked' under network->data->review bandwidth. Could be some tumblr CDN domain.

>>11003

requesting mr bones track how many files have no tags too.

>>11027

my understanding, at least the last few times I did performance testing, any lag/jank is related to open images, url and downloader wise, it comes down to number of active downloaders

That said, I have been considering of a clean start once I go though the images and having everything there saved as a legacy go through… that will be gone though once I can do something about duplicates, which for me necessitates >>11021… but like I said, I have something around 700gb of space left and push come to shove I can make more space by removing the fuck huge files.

as for the program not logging me in, it does, the 15 image is the click-through, the 68 image is the logged in, though it was still doing a guest log in, so not i'm logged in for 30 days at a time which is good, changing it from click through script to login though seems to have make the program wait longer though.

>>11029

If you go the youtube-dl route, I have a request.

Can you have an option in the downloader for it to NOT use the hydrus archive for saving the file and instead a different directory?

While hydrus is great, its great for stand alone images, and would be fairly shit if I stored video in current hydrus.

Its partly the reason I wouldn't use current hydrus for manga, while I would love the program to work with manga, I would have to rely on the program to work going forward, which I can't do. You as the dev put 334 updates and have been doing it near weekly, I trust that as long as you are alive you will keep supporting the program and make sure it works. but lets take my manga folder into consideration, I have around 2.5tb of manga and h-manga combined, possibly a bit more if I looked on more hdds, if I put these into hydrus, it would hash them, and make them non user sortable. if I just put images in, It would fuck 170 images+ from ever being read in sequence again, and lets put it this way, when it comes to me downloading manga, I would rather re download a manga then take the time to rename it to fit a convention, there is no way in hell I could do that for hashes.

for single images? before I had hydrus, I had around 120-200gb of images, and another 500-1tb of gallery images that were never going to be sorted anyway, so I was no loss to me to hash them, if anything I lost may 30000 images out of my real/drawing/real porn/drawing porn sorting method but I believe enough in hydrus that it was no problem for me even if it failed horribly.

For this reason, I wouldn't want to have youtube-dl save to the gallery, but on the same note, I would love a better ui for youtube dl then the crap that is currently out where the person behind it I think has fucked off and told anyone with issues that he doesn't see the issues as issues so pound sand.

If you end up making a youtube-dl ui option, It would need a few things to it

1) video quality/format choice with a 'choose highest quality' option

2) audio quality/format choice with a 'choose highest quality' option

3) an audio/video only mode

4) at least for me to use it, an out of archive save location that saves based off of videos title

In all honesty thinking of the manga portion again, If you ever do that, I highly suggest putting all the functionality of having a hydrus managed folder layout that is also user readable into the main client but having it walled off, that way a branched version takes no real work to manage as you are already doing everything with the main version anyway.

Hell, You could have 3 folders

inbox / archive / trash

inbox - being where everything gets put in when it first comes to the program,

archive - where everything goes once ok'ed, realistically it's a holdover from current hydrus, but I have had manga where the images are fucked and a v2 was needed

trash - where everything goes if deleted.

in the archive, things would be sorted from

series - (manga name)

or

artist - (artist name)

instead of the current has

possibly splitting adult comics into its own little space with a series and artist separation there too.

In the main non porn, series comes above artist, so if something is sorted into a series that's where the files go, with a possible symbolic link in artist,

for the porn section, artist is where things go, with series being the symbolic link

This way, while the program works, you would do all the reading and browsing in it, and when its fucked, people can VERY easily migrate to something else if they even choose to do that much.

Hell the manga version could also do all video too, sorting it much the same way, with series being something more for tv/movies and artist being something more for youtube channels.

I think I got very off track from what this was originally about.

I have trouble opening the bugs thread, so I'll just dump my issue here.

I still get pretty bad X Window-system crashes when running on my Antergos (Archlinux with some ease of use stuff) system (x86-64 with relatively new packages).

1. For example, when zooming on a large image, I get:

OpenCV(3.4.1) Error: Assertion failed (dsize.area() > 0 || (inv_scale_x > 0 && inv_scale_y > 0)) in resize, file /io/opencv/modules/imgproc/src/resize.cpp, line 4045

2018/12/27 00:04:46: Uncaught exception:
2018/12/27 00:04:46: error
OpenCV(3.4.1) /io/opencv/modules/imgproc/src/resize.cpp:4045: error: (-215) dsize.area() > 0 || (inv_scale_x > 0 && inv_scale_y > 0) in function resize
  File "include/ClientGUICanvas.py", line 5620, in EventPaint
    self._Redraw( dc )
  File "include/ClientGUICanvas.py", line 5583, in _Redraw
    wx_bitmap = self._image_renderer.GetWXBitmap( self._canvas_bmp.GetSize() )
  File "include/ClientRendering.py", line 134, in GetWXBitmap
    wx_numpy_image = ClientImageHandling.ResizeNumpyImage( self._media.GetMime(), self._numpy_image, target_resolution )
  File "include/ClientImageHandling.py", line 353, in ResizeNumpyImage
    return cv2.resize( numpy_image, ( target_x, target_y ), interpolation = interpolation )


The program 'client' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadAlloc (insufficient resources for operation)'.
  (Details: serial 649355 error_code 11 request_code 53 minor_code 0)
  (Note to programmers: normally, X errors are reported asynchronously;
   that is, you will receive the error a while after causing it.
   To debug your program, run it with the --sync command line
   option to change this behavior. You can then get a meaningful
   backtrace from your debugger if you break on the gdk_x_error() function.)

2. Haven't fully nailed down when, but I will sometimes get

client: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.0.

This mainly occurs when entering tags or when doing the archive/delete filter. May be a nasty multiprocessor bug, given it's an IO error.

In both situations I have reasonable memory available (+4GB).

>>11119

Regarding no. 2: I've nailed it down to the "manage tags" window. The filter works nicely, but I cannot add tags while filtering, because that may crash hydrus.

Hey h dev…

So, just got done making a split 4chan thing where I only have the active ones now and I have the inactive ones saved to be dealt with later.

I went from a working 12-14gb down to 1.4gb of use… I still have around 20-30k images open that I have been getting around to… but holy fucking shit, what the hell was eating the ram?

I know that last time I did performance testing, 40~k images what the hard limit on where the program will start stuttering, but I can't for the life of me think of what the fuck was eating 13~ gigs just from having threads in the watchers? at least the way I see it, there would be data on what image it is, where said image is, its tags and shit, all that fun stuff, and I cant come up with a reason that the un highlighted threads have any reason to eat up more then 100mb, much less 13 gb… able to explain this at all?

>>11128

Not dev, but were you archiving from /gif/?

For reasons unknown to me hydrus runs ffmpeg every time it downloads a file. Checking for compatibility, reformatting for consistency, maybe? Could never figure it out, but every /gif/ thread I have a downloader entry or a tab for always seems to scale up mem use far more than the others, and I notice those wild ffmpegs. Maybe dev can explain better if this is the case.

>>11141

I think in total around 1.2 million links, with well over 75% of the files being things I already had, gif was one board, but I watched every board I removed dead watchers for just lighten the load a bit more, more so then getting around 40k images in active use removed. the fact it persisted through restarts was also an issue that makes what you described questionable if that was it.

I mean I would understand it if every now and then a restart would reset everything, but at least as far as i'm aware everything should be text, and even assuming there is a good paragraph of text for each file there is no reason it should have eaten that much ram.

>>11052

Yeah, there's a bunch of these bad progress gauges, including repo processing, where it made sense in the early days or during devving and I am doing 17/25, but in reality a lot of this stuff ends up being 4000/4003. I'll make a cleanup job to go through this and make it more helpful.

>>11119

>>11120

Thank you, I will check manage tags during filtering!

The zoom problem is probably unavoidable for now. The rendering pipeline is still shit for this as it ends up making a whole giant bmp rather than tiles, so zooming to virtual 20,000x20,000 screen real estate actually tries to allocate that much ram, breaking a whole bunch of stuff. It's another thing I have to fix. "Don't zoom big stuff 800%" is the unfortunate solution for now.

>>11128

>>11141

>>11145

I am not sure what the superbloat was there. There's a few indices being built with every import list, so that adds a little bit, but it usually only gets killer when there is one big list, like 1 x 600k, rather than 10 x 60k.

I think I've said before I hope to add a memory profiler in py3. It may be I am accidentally holding on to additional copies of some information somehow. It is possible some of your saved memory here is helped by my new 'media results' cache, but I'm a bit in the dark on the details until I can profile this properly.

I use ffmpeg to figure out video mime and pull a thumbnail, so if the client can't guess a new file is a jpg or whatever straight off the first few header bytes, it gets sent to a new ffmpeg to see what it thinks of it. If you are importing a bunch of webms in a row, you'll be burning a decent whack of CPU. I understand there were once some ok python libraries that wrapped the ffmpeg dll and let you load it properly, but I don't know of a good one now, so I have to rely on some mickey-mouse subprocess piping bullshit to get all this to work (including video rendering, wew!). It shouldn't increase hydrus memory as it'll be a child process, but maybe your task manager counts up total memory usage including child processes or whatever. Please let me know how py3 goes for you here–it hopefully will be better.

>>11186

lol for all the bugfuckkery it just did, it added a grand total of 56 images of 7000 some potential ones to the archive.

>>11187

down to 1.8gb

Thank you, based dev.

ok got a bit of an issue

not able to connect to hentai foundry

and newgrounds im not sure has ever worked.

4chan downloading works, so its not something inherently broken with network,

Could not connect!… (Copy note to see full error)

Traceback (most recent call last):

File "include\ClientImportGallerySeeds.py", line 249, in WorkOnURL

network_job.WaitUntilDone()

File "include\ClientNetworkingJobs.py", line 1091, in WaitUntilDone

raise self._error_exception

ConnectionException: Could not connect!

Ok, I mentioned this issue a while back but it got dismissed, however it cropped up again

when importing files the beginning will start to bloat, where in a fresh directory it will be 0, after a while it will start to add numbers to the front. it's possible that files that cannot be imported add to the beginning, but the issue is once removed, they still persist right now from my image importing I had a clean folder but it has gotten around 140 images/files I can track that that couldn't be imported. but it starts at 185.

is there a way to reset this/allow the importer to forget previously impossible files to import?

>>11201

HF seem to have let their ssl cert expire (but only as of Jan 6th, I think?), I don't know what is going on over there atm. Maybe they blocked hydrus based on User-Agent, but maybe their server is undergoing some other fuckery. I will keep an eye on it and see if their get their https back in order.

I recommend you pause any HF subs/downloads for now.

>>11203

Hey, I am sorry, I do not totally understand what you wrote here. Can you rephrase it or add a screenshot?

I do remember a user reporting an issue with 'import folders' having bad tracking for the files it did not import (for whatever reason)–was this you? If you check the button next to 'review currently cached import paths' on the edit import folder panel, what do the paths there suggest about your problem?

>>11227

>>11226

Thanks. I will make a job to check this code and have it occasionally clear out paths that no longer exist.

I'll also be looking at changing some of these popup progress gauges to show progress more for the 'current job', like 1/17, rather than always showing entire-life progress like 4001/4017, which is always just a green bar with one pixel of grey, no matter how big the most recent sub sync or whatever was.

>>11231

lol asked about that somewhere too because that was a bit annoying seeing 'oh there are 1000 images here, the fuck happened' only to see the next stage of the sub have 975/983 and realized it remembers everything. not worth resetting the checkers for as to me it was a minor annoyance.

I should also say, the weirdness where on saving the session where it would bloat from 12gb to 18+gb has been solved with the removing the dead watchers from active view, however with a little over 100k images in active view mode, I am unable to set it to every 5 minutes as it does start to bloat and lag a bit, no where near the extent it did before, but a bit.