/hydrus/ - Hydrus Network

Based dev, thanks.

Started client and left it open for an hour or so.

Got this:

RuntimeError

wrapped C/C++ object of type AutoCompleteDropdownTagsRead has been deleted

File "include\ClientGUIACDropdown.py", line 500, in EventMove

if self._ShouldShow():

File "include\ClientGUIACDropdown.py", line 223, in _ShouldShow

tlp_active = self.GetTopLevelParent().IsActive() or self._dropdown_window.IsActive()

>>8036

Not sure if this is something you improved upon in this release specifically, but I'm really happy with how Hydrus relinquishes memory usage after rendering large images. Had it spike to 1-2 GB's of memory usage for a pic or two (and it lagged of course), but after viewing and waiting another minute it went back down to a mere 160MB

Image board as in booru I vote derpibooru

this place has a shitload of content that is basically only available there or following artist blogs

anime is taken care of with dan, gel, and pivix

furry is handled by many of the other booru's and if you ever get a furafinity inkbunny parser going,, but for pony alone, this is basically it.

If you mean image board as in 4chan/8chan like places, look at the site and see if page of images works to comb the threads, if it works I honestly think time is better spent elsewhere in the program if the parsers are simple to make.

>>8043

It would be nice

for me it says 300~ image of some number higher,

lets say I have 7 images, it just said 311 of 318

its nice, but something is wrong with it for me

I think I found some false positives. Here's the log excerpt:

2018/02/15 23:21:14: Error when processing https://e621.net/post/show/1279537/absurd_res-cyanlightning-equine-female-feral-frien !
2018/02/15 23:21:14: Traceback (most recent call last):
  File "include\ClientImporting.py", line 4387, in _WorkOnFiles
    ( status, hash ) = HG.client_controller.client_files_manager.ImportFile( file_import_job )
  File "include\ClientCaches.py", line 1008, in ImportFile
    file_import_job.GenerateInfo()
  File "include\ClientImporting.py", line 477, in GenerateInfo
    raise HydrusExceptions.SizeException( 'Image seems to be a Decompression Bomb!' )
SizeException: Image seems to be a Decompression Bomb!

2018/02/15 23:21:33: Error when processing https://e621.net/post/show/1279540/absurd_res-cocky-cyanlightning-equine-feathered_wi !
2018/02/15 23:21:33: Traceback (most recent call last):
  File "include\ClientImporting.py", line 4387, in _WorkOnFiles
    ( status, hash ) = HG.client_controller.client_files_manager.ImportFile( file_import_job )
  File "include\ClientCaches.py", line 1008, in ImportFile
    file_import_job.GenerateInfo()
  File "include\ClientImporting.py", line 477, in GenerateInfo
    raise HydrusExceptions.SizeException( 'Image seems to be a Decompression Bomb!' )
SizeException: Image seems to be a Decompression Bomb!

2018/02/15 23:22:53: Error when processing https://e621.net/post/show/1279564/absurd_res-braeburn_-mlp-cutie_mark-earth_pony-equ !
2018/02/15 23:22:53: Traceback (most recent call last):
  File "include\ClientImporting.py", line 4387, in _WorkOnFiles
    ( status, hash ) = HG.client_controller.client_files_manager.ImportFile( file_import_job )
  File "include\ClientCaches.py", line 1008, in ImportFile
    file_import_job.GenerateInfo()
  File "include\ClientImporting.py", line 477, in GenerateInfo
    raise HydrusExceptions.SizeException( 'Image seems to be a Decompression Bomb!' )
SizeException: Image seems to be a Decompression Bomb!

2018/02/15 23:23:08: Error when processing https://e621.net/post/show/1279565/absurd_res-big_macintosh_-mlp-cutie_mark-earth_pon !
2018/02/15 23:23:08: Traceback (most recent call last):
  File "include\ClientImporting.py", line 4387, in _WorkOnFiles
    ( status, hash ) = HG.client_controller.client_files_manager.ImportFile( file_import_job )
  File "include\ClientCaches.py", line 1008, in ImportFile
    file_import_job.GenerateInfo()
  File "include\ClientImporting.py", line 477, in GenerateInfo
    raise HydrusExceptions.SizeException( 'Image seems to be a Decompression Bomb!' )
SizeException: Image seems to be a Decompression Bomb!

Other than that, v294 is great for me so far. Looking forward to the internal flash renderer comeback though. :)

Just got this… well at some point got this.

RuntimeError

wrapped C/C++ object of type AutoCompleteDropdownTagsRead has been deleted

File "include\ClientGUIACDropdown.py", line 500, in EventMove

if self._ShouldShow():

File "include\ClientGUIACDropdown.py", line 223, in _ShouldShow

tlp_active = self.GetTopLevelParent().IsActive() or self._dropdown_window.IsActive()

>>8051

10007x12493

10440x12032

10000x12882

10000x12779

program at least in my extensive "I download several hundred thousand images with this thing, oh well lets get some more" searches, the program hates stupidly large image sizes.

>>8049

It might be counting all of the non-image files too even though it doesn't import them

Ok when I drag the window I got theses

RuntimeError

wrapped C/C++ object of type AutoCompleteDropdownTagsRead has been deleted

File "include\ClientGUIACDropdown.py", line 500, in EventMove

if self._ShouldShow():

File "include\ClientGUIACDropdown.py", line 223, in _ShouldShow

tlp_active = self.GetTopLevelParent().IsActive() or self._dropdown_window.IsActive()

It seems to just happen till I stop dragging the window.

>>8054

The directory it imports from has 23 files it wont import, so that cant be the reason

Ok pulling some more images from tumblr and ran into a very obvious problem, and something interesting at least to me

so this tumblr, and content not withstanding, https://bcdickchixtt.tumblr.com/ about half the images wouldn't download, the other half would, and the first image, somehow that one downloaded an image that was a good 4 times larger than the largest one I could find on the page.

so, here is the issue as far as I can tell, any image on tumblr that you can open by clicking on to get that gallery mode they have, those will download, however any image that doesn't do that, this won't pull from it.

I'm sure there are other tumblers that will also present this, but I think this one has the most issues of the ones I have come across,

with that said, going to have a copy of every tumblr I pull images from form now on in a file so I can recheck when/if you get around to fixing/improving the tumblr parser.

>>8053

It was one of my extensive searches which aborted, because it stumbled over these images. Otherwise I wouldn't even have noticed the errors.

My library can well live without these images. But maybe one day there'll be an image that I'd be really eager to have and then this error would drive me nuts. :)

>>8057

for the 4x bigger one, did it pull the _raw version rather than the _1280 version?

>>8061

I believe you can force very large images to import.

at least from my experience, large images like that are never things I want to save as they tend to piss away space for no reason, but if you save the image and import from hdd it should go thought I beleive, or there should be something to ignore it, I believe I imported a few large ones this way, but the program just crashed so I have to wait for it to re load. looking at my shit, I have 7 images over 150 mpixles. all of them are either things I have imported myself, or are older than the decompression bomb

>>8064

yea, there a way to get to raw versions easily? never really had an image on tumblr that wouldn't load a far larger one before if it was available.

~~~~~

Ok dev, had one crash when I was trying to search my files, no idea while, program just crashed, after reloading it it is fine, im also not getting errors moving the window around.

>>8068

There's a few on Greasyfork. This is the one I use:

https://greasyfork.org/en/scripts/30833-tumblr-image-url-redirect

I also made a Twitter one similar to it that gets you the :orig files there. You can find that one in this post over here >>7937 . Just save it as a .user.js, figure out how to add it to your user script manager of choice, and baddabing.

When trying to run hydrus from source I get the following error:

python client.pywTraceback (most recent call last):
  File "client.pyw", line 20, in <module>
    from include import ClientController
  File "~/hydrus-source/include/ClientController.py", line 1, in <module>
    import ClientCaches
  File "~/hydrus-source/include/ClientCaches.py", line 5, in <module>
    import ClientSearch
  File "~/hydrus-source/include/ClientSearch.py", line 4, in <module>
    import ClientTags
  File "~/hydrus-source/include/ClientTags.py", line 2, in <module>
    import ClientGUIDialogs
  File "~/hydrus-source/include/ClientGUIDialogs.py", line 6, in <module>
    import ClientDragDrop
  File "~/hydrus-source/include/ClientDragDrop.py", line 1, in <module>
    import ClientGUICommon
  File "~/hydrus-source/include/ClientGUICommon.py", line 679, in <module>
    class CheckboxCollect( wx.ComboCtrl ):
AttributeError: 'module' object has no attribute 'ComboCtrl'

The same error happens with test.py. I just extracted from the .tar.gz. What am I doing wrong?

>>8068

The work around you suggest is fair enough. Guess I've just gotten used to being pampered with the automatic tag importing and the general convenience Hydrus provides to my image hoarding.

I think what mostly annoys me about this is that Hydrus considers this a hard error which counts towards aborting the sync.

I'd prefer Hydrus to continue the sync and tell me later that X files were skipped for reasons A, B, C and potentially offer me choices like "Accept", "Retry now", "Retry later". Or based on what makes sense it could be more specific "Import suspected decompression bombs".

But maybe this is all over the top and an option (per booru?) not to check decompression bombs is enough. Or the check itself could be more refined? No clue what fits the vision for this project best. I'll trust the dev will make a good call. :)

>>8068

just an update for the crash, I think I am back to getting >>8052 error message again, it doesn't happen at launch, it happens several hours after. part of me wants to check my images to see if it crashes again, but another part of me doesn't want the program to force the program to crash

>>8069

Ok got them both

>>8071

I never had it just hard stop, a few days ago i did a 100k image download from gel, I think it found 4 obscenely oversized images, hell my imports were 59081

63303

65040

and 67839 respectively,

they are

https://gelbooru.com/index.php?page=post&s=view&id=1990295

https://gelbooru.com/index.php?page=post&s=view&id=3153655

https://gelbooru.com/index.php?page=post&s=view&id=3461938

https://gelbooru.com/index.php?page=post&s=view&id=3472486

I saved them, then went to manually import and seen they were also blocked, so, options, media, and turn off the decompression bomb, a bit annoying as this setting filters out a lot of crap I dont want, but its leaving out images I do, so yea…

just a forwarning, the ryoko image ate 500mb of ram, but the overwatch ones were 2~gb each when 100%ed

>>8047

>>8052

>>8055

Thanks lads. I get the same thing. It looks like my fix for the crash is sometimes not deleting something. Thankfully this doesn't seem to be ultimately harmful, and the crashes do seem to be gone, but this is annoying. It seems to be initialised by the manage tags dialog or a regular search page with a tag input being closed. Restarting the client will clear it.

Thank you for these reports, I will have this fixed for v295.

>>8048

Great. Yeah, I've put some work into this recently. For videos as well, which had their own memory bloat problems.

>>8049

Thanks. I hope to convert all the existing boorus over to the new system. Derpibooru isn't in the defaults, but I know lots of people would like it, so I'll make sure to add a new default for it when the time rolls around.

For the 311/318 issue, does your import folder have 311-or-so unimportable files in it? That would probably be why they were hanging around without being imported and cleared out. If you go into manage import folders and then click your import folder's file import status button (the little icon button that opens a new window with the list of file imports), you'll probably see a bunch of existing files with 'uninteresting mime' or 'failed' or something?

Or if your import folder is set to 'leave the file alone, do not reattempt it' for several import outcomes, these files may just be those hanging around, I think.

Please let me know if the numbers here are off to what you actually have. There might just be a counting error that is counting them up for you wrong.

Ah, I see in >>8055 that it seems the count is off. How many paths are in your file import status window? What status do they have? Do they seem like the right paths?

>>8051

>>8053

Thank you for this report. If you would really like those files, please try reimporting them with options->media->do not import decompression bombs turned off. Be warned that these files may take more than a gig of memory and up to 30s to import and render every time you look at them!

I have a couple of leads on getting flash back. I want to try it again, and now the crashes seem pretty much cleared out, I can spend a bit more time on this.

>>8057

>>8064

>>8068

Thank you for this report. I currently use the old tumblr API, which has spotty gallery support. For official tumblr galleries it works ok, but anything embedded in an html post isn't as reliable. In the new downloader engine I think I will replace it with a hydbid parser that maybe uses the JSON API to fetch post URLs and then parse the raw html of each post to fetch all the img/video content.

Yeah, any big images (over 1280px wide or whatever) from tumblr are the newly discovered _raw format. Hydrus tries to substitute to raw whenever it can. Files from 2012 or before do not have _raw, iirc.

>>8070

Thank you for this report. If you open a terminal and go:

python
import wx
wx.__version__
exit()

What version does it think you have? Hydrus since v288 uses wxPhoenix (4.x), whereas I used to use 3.0. Getting wx4.0 can be a small pain, but generally much less than getting 3.0. You can also tuck 4.0 into a virtualenv. Please check out the running from source help page here:

http://hydrusnetwork.github.io/hydrus/help/running_from_source.html

>>8071

Thanks–I hadn't realised the decomp bombs were messing with sub sync. I'll see if I can catch them more gracefully.

based on discussion on discord

we want to compile suggestion/ todo/recommendation on github

if anyone interested, can look here

https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/issues

>>8073

I'm >>8070

Thanks, I stopped after reading the word 'Windows' so I didn't see the part about wxPython. I had 3.0 but now I updated and hydrus runs fine again. The install command you linked to was useful.

Something you should probably mention in that guide though is that

pip install package

will not upgrade you to the newest version if you already have an older version, for that you need to use

pip install --upgrade package

I used Hydrus for a couple of hours yesterday, doing a lot of manual tagging. No crashes or problems at all.

For all of these various types of scripts Hydrus is using, downloaders & parsers etc, it would be cool if the creator of the script could add an update URL to them so that they could autoupdate, useful should anything on the sites change that makes the script invalid. It seems to happen occasionally and it would be convenient if that was independent from program updates. Just a thought.

Okay I just came into another problem that's entirely my fault but I'm wondering if there's a way that it can be prevented in the future through software.

So last night I was downloading a bunch of videos and I come across a site that makes the right-click and save the video it won't just save it by clicking the link to the video file itself. Okay I do this it downloads it plays everything's fine, however what I didn't notice was it saved it to my image folder not the normal download folder.

There are very few reasons for me to actually right-click save as something on web browsers. One of which being I want to save the website itself, chrome has for a long time implemented a way to save the website as a single file opposed to a hyperlink with a folder of images, so that's not an issue for the program if it saves in the wrong directory, and video/audio files. However most of the time the video and audio files have a way for me to click on them once a man the site to directly just download them so I don't have to go save as. Knowing all these cases it's going to either put it in the download folder or the file that Hydrus doesn't recognize so it's not going to import. However the file that I had was a 15 minute long MP4 and it saved it into the image folder.

So the circumstances of why this would even happened, would there be a way to put an upper limit on the file size that can be imported into the program through import folders?

I can see this useful and quite a few different areas of the program like downloading from the website, I know one of the boorus had downloaded 150 MB file before, if there was an upper limit on how big the file could be before you need the explicitly tell the program yes I want this file in the program I think that could actually benefit long run.

Personally the way I'm using the program I don't think I'll ever be putting long videos in here are full TV shows or movies, however I could easily sleep clips of him being in their having a check between do I want to import this very large file which is likely a mistake I think would be a good idea.

I just did a quick check in hydrous at least for me there are 24 files over 100 MB large, I believe that each one of the images on there is something I would've never okay doing been imported had I known what was or at least how big it was to begin with. In total of the 24 files 23 of them are video/gifs personally I think I'm to be running them through handbrake and there's no reason a three minute long video that looks like they look should take up over 100 MB but knowing in advance that these files would have downloaded or would have been imported it would've been really nice to have an upper limit and be notified that something big was coming/could be okayed.

>>8083

Thanks, I will add this.

>>8085

Yeah, I think being able to 'subscribe' to an updating parser would be a nice system, however it was implemented. At this stage I just want to get the thing working, though. But once we are up and running, I'll be very open to seeing how often these things need to be changed in practise, and in what ways, and see what ways people feel best about pushing or pulling updates to parsers.

Letting clients suck and run code structures automatically from the mysterious internet may require a bigger trust infrastructure than we currently have. Maybe signing parsers with an author certificate, that sort of thing.

>>8096

Thanks, this is a great idea. I will add a 'maximum file size' to the file import options. There's already min_size and min_resolution, so it would be easy to add the opposite.

The 'big (usually SFM) gif' problem is very much on my mind. It is so pernicious and ridiculous (my current record is a 222MB gif 4000x6000, 16 frames) that I think I may just write a special rule in hydrus to deal with them. I might also write some kind of choice logic into this new parsing/downloading engine I am working on. Something that will generate a checkbox on the gallery downloader ui that says 'ignore big gifs' (default on) so we don't have to deal with this shit. Maybe just a checkbox in options->media like for the decompression bombs.

I have heard that big gif versions of webms are for phonefags who can't deal with high-cpu video codecs for several reasons, although I imagine they aren't too chuffed with 100MB bandwidth themselves. I hate it like I hate 12,000x10,000 'premium original' pngs on patreon.