/hydrus/ - Hydrus Network

honestly the only site that I really want something for is derpibooru as its the only booru that I would want to set active subscriptions for, along with also being the only place that hosts content other boorus don't touch so there is little overlap unless a popularish artist also draws other things.

beyond that, image board wise, im more looking forward to a possible multi page thread watcher then boards themselves, because honestly every board I go to and tested with will respond to page of images grabs, if you made a page of images, If you expanded page of images features to implement a recurring check, this would honestly hit nearly every image board out side of exotic ones.

at least this is the way I see it, I have 72 threads hidden on one of the boards I gather images from on 4chan, because it could take days-weeks for the threads to 404, its not possible to thread watch when watchers are limited to around 160~ pages, so instead I will end up dumping them into page of images, or when they 404 completely, harvest from archives.

having page of images recurring works for most boards because of the small hold over archives on 4chan, it would likely catch everything, if it could act like a watcher it would likely hit the threads till they 404.

watcher would be more useful because it would check if something 404'd or is dead and likely be able to tell even if there were more than one thread.

if page of images could act recurringly, it would need something in the ui to tell it how many images were there on last check, and how many checks since the last image was found.

till one of these happen, I will likely just use threadwatchers for threads where I don't really want to go through the archives to harvest.

at least this is my use case for them.

Got a crash after hydrus running for about half an hour. Had just imported some files through subscriptions, and gone through and edited their tags plus using scripts to grab some tags. Was done, uploaded the tags to PTR, switched to another program for about 5 minutes, came back to a crashed hydrus.

I'm sorry if this isn't the place to ask it, but is there any way I can use a Danbooru gold/plat account with Hydrus? I couldn't find any info regarding this. Searching for only 2 tags is kinda sucky. ;__:

>>7907

Use gelbooru instead, they have the same content afaik anyway.

>>7902

Hey, macOS guy here once again, just wanted to confirm that v292 is working nicely, just like v291 did. :)

Thinking about it, aside from the crashes in v290 I've only ever had one issue and that has been there since at least v21x:

Once hydrus is open, the menu is unresponsive for about 10-15 seconds. And by unresponsive I mean I click and nothing happens. It doesn't just delay the input either, it seems to discard it altogether.

I can't click the system button next to the menu (the Apple logo) either when hydrus is focused after opening it. The application tray icons and all the other stuff (volume, battery, search etc.) on the right side of the menu bar work just fine though.

After this 10-15 second waiting period, the menu behaves normally.

What also works instead of waiting is focusing another application (or switching the visible workspace, basically just unfocusing hydrus) and then refocusing hydrus again. In that case the menu works instantly, I don't have to wait 10-15 seconds (I've tried to do that as fast as I can (~ 500-1000ms) and the menu was fine afterwards).

I hope you understand what I mean, I would clip it but the application I normally use for screen recordings (the built-in Quicktime app) doesn't record the menu bar unfortunately.

It's not really an issue that bothers me, just curious if you have any idea what that could be. Don't worry about fixing it though, I'm sure you have more important stuff that needs attention.

Once again, it's just the macOS build (had the issue on several different machines, MacBook Pros and iMacs), I've never had this issue on my Windows machine.

Crashing still after adding some new tags to stuff.

Faulting application name: client.exe, version: 0.0.0.0, time stamp: 0x00000000
Faulting module name: wxbase30u_vc90_x64.dll, version: 3.0.4.0, time stamp: 0x59bd86ef
Exception code: 0xc0000005
Fault offset: 0x0000000000126340
Faulting process id: 0x954
Faulting application start time: 0x01d39ba84f5ceb7f
Faulting application path: D:\Hydrus Network\client.exe
Faulting module path: D:\HYDRUS~1\wxbase30u_vc90_x64.dll
Report Id: 3babc7f6-07a6-11e8-b332-00224d80bd5e

This might help though, a within the few informational logs in Event Viewer, there's this:

Possible heap corruption detected (exception code 3221225477). Initiating further diagnostics.

But I'd also gotten that during a very weird series of events where it was highly unlikely that heaps were corrupted, so take that as you will.

Had a valueError whilst opening the options, it worked seconds before but I closed it by clicking apply. Two thread watchers should have finished their downloading about 30 seconds before this happened.

ValueError
need more than 3 values to unpack
  File "include\ClientGUIMenus.py", line 141, in event_callable
    callable( *args, **kwargs )
  File "include\ClientGUI.py", line 1940, in _ManageOptions
    panel = ClientGUIScrolledPanelsManagement.ManageOptionsPanel( dlg )
  File "include\ClientGUIScrolledPanelsManagement.py", line 1310, in __init__
    self._listbook.AddPage( 'default file system predicates', 'default file system predicates', self._DefaultFileSystemPredicatesPanel( self._listbook, self._new_options ) )
  File "include\ClientGUIScrolledPanelsManagement.py", line 2006, in __init__
    self._file_system_predicate_age = ClientGUIPredicates.PanelPredicateSystemAgeDelta( self )
  File "include\ClientGUIPredicates.py", line 92, in __init__
    ( sign, years, months, days, hours ) = system_predicates[ 'age' ]

it didn't crash, just added this in a popup at the bottom right.

>>7915

after this I tried to open the options again, same error. Then did the duplicate processing of all the new files and got this 'error'.

error
C:\projects\opencv-python\opencv\modules\core\src\alloc.cpp:55: error: (-4) Failed to allocate 3221028867 bytes in function cv::OutOfMemoryError

  File "include\ClientGUICanvas.py", line 5468, in EventPaint
    self._Redraw( dc )
  File "include\ClientGUICanvas.py", line 5431, in _Redraw
    wx_bitmap = self._image_renderer.GetWXBitmap( self._canvas_bmp.GetSize() )
  File "include\ClientRendering.py", line 120, in GetWXBitmap
    wx_numpy_image = ClientImageHandling.ResizeNumpyImage( self._media.GetMime(), self._numpy_image, target_resolution )
  File "include\ClientImageHandling.py", line 326, in ResizeNumpyImage
    return cv2.resize( numpy_image, ( target_x, target_y ), interpolation = interpolation )

I must add I tried to zoom very deep into a 3000*2000 image to see the difference with a lower filesize same res image that the filter caught. that could be the problem. I have 112MB dedicated VRAM on my intel HD chip, and 2048MB of shared memory (RAM that since I have 12 G ram can be used as VRAM). Could that be the problem? 3221028867 /1024/1024 = 3017MB, so that is too much for my graphics chip.

>>7912

Not to take away from anything you're saying, but you may want to add https://saucenao.com/ to your other methods of searching for images from other images. Tossing your first artifact-ridden thumbnail into it immediately gave me the artist's original Pixiv and Nijie posts as well as a Sankaku upload for good measure.

>>7917

>Not to take away from anything you're saying

They didn't even seem to be saying anything remotely related to hydrus itself. Literally all I got from it is "I've never heard of saucenao and I'm using every resource but the sole one that works for what I'm actually looking for".

>>7921

For the most part, yeah, but they finally got to something at the end about the page of images downloader not handling the page the way they think it should. About 70% of the post was unnecessary while the rest wasn't explained very well.

>>7905

Thank you for your feedback.

I am in the same place with wanting a multi-page thread watcher. I've mostly moved my processing workflow out of the thread watcher pages themselves, so I'm not looking at them for much, so the pages are just taking up space until they are [DEAD].

I intend to have multi-page thread watcher done before this downloader overhaul is finished.

Since you mention it and now I think of it, I am not sure what to do with the page of images downloader. It will be possible to write much better 'simple gallery' parsers in the new system, so it would be nice to expand what it can do, but I also want stuff in the new system to be more user-editable. I will keep thinking about this.

>>7906

>>7911

Thank you for these reports. It seems manage tags may still be doing something, although other users report it is much better if not fixed, so I wonder if there is a combination of things going on. A couple of users think the media viewer may be involved. I will put some more time into it this week.

If either of you have time to test, I would be interested to know if you get any crashes if you completely abstain from using the manage tags dialog. Please do import and browse media as normal–just don't set any tags manually.

>>7907

Not yet, but you will be able to by the time this downloader overhaul is finished. I have the first part of a login engine in place, and once this new parsing system is ok, I will be able to match the two together and create arbitrary login forms for any site, and you should be good to go.

>>7910

Thanks, I am glad it works.

I am not sure what this menu stuff would be other than OS X generally not being chuffed with what I am doing. I'm unsigned hackery-dackery-doo frozen python code running on wx, so there must be ten hoops the program has to jump through before OS X is willing to put pixels on the screen. I wouldn't be surprised if there is some kind of ui restricting in the first few seconds of running, either as a kind of 'don't let user click anything popup/malware-ish' or a 'don't let this program put menus in the system area until I have virus-checked it all' or something similar.

I don't get the same here on my OS X dev machine, if that is any help–either from source or the built app. It might be something to do with your system particularly, maybe some anti-virus or window manager something.

>>7912

As I was just sort of saying to >>7905 , I would like to move the page of images downloader into the new parsing system so any user can more easily grab galleries like this. I am not sure exactly what form that will be atm, but it will be able to do more than current.

Since you are into ripping, please wait for more of the new parsing system to roll out, and then, once I've got some help documentation for it ready, please check it out and see if creating a parser is the sort of thing you would like to do! I hope to have a final system that can solve these 'we have the content, but only three pages deep' problems–at least if there isn't any javascript required.

>>7915

Thank you for this report. I fucked my own shit up this week with the system:age update and invalidated an old option by mistake. I apologise for the inconvenience and will have this cleaned up for next week. I will also add a new test to my weekly test routine to make sure it doesn't happen again.

>>7916

Yeah, unfortunately the current rendering pipeline does not do zooming of big pictures very well. At the moment it blows the whole thing up to a gigantic bmp and scrolls that around, whereas I want to update it to be a tiled system that will only render the bits on screen.

Please refrain from zooming 4k-ish stuff to like 1000%, as the libraries will just dump out.

>If there are other imageboards you would like to be able to thread watch next week, please let me know

Pregchan, please! (e. g. https://pregchan.com/d/index.html)

>>7924

> If either of you have time to test, I would be interested to know if you get any crashes if you completely abstain from using the manage tags dialog. Please do import and browse media as normal–just don't set any tags manually.

not any of those anons, but as a precaution I have not used the tagging dialog and I have not crashed.

> >>7915 (You)

> Thank you for this report. I fucked my own shit up this week with the system:age update and invalidated an old option by mistake. I apologise for the inconvenience and will have this cleaned up for next week. I will also add a new test to my weekly test routine to make sure it doesn't happen again.

> >>7916 (You)

> Yeah, unfortunately the current rendering pipeline does not do zooming of big pictures very well. At the moment it blows the whole thing up to a gigantic bmp and scrolls that around, whereas I want to update it to be a tiled system that will only render the bits on screen.

Thanks a lot, the zooming thing isn't really hard crash so that's nice, thanks for all the hard work my man

>>7924

Managed to hold off from using the tag dialogue, though I applied tags while importing, via the import dialogue.

Wasn't crashing, let it sit for a bit while I did other work, just came back to it having crashed. Exact same error, the wxbase30u_vc90_x64.dll. Every value is exactly the same except for the app start time and report ID (naturally), and the faulting process ID is 0x1464. Not sure if that's one of those static value sorts of things or not though.

>>7924

honestly, here is a thought on thread watchers.

would it be possible to make threadwatchers into a more subscription like method?

something that wouldn't cause a ui issue, so you could have a fuck load of pages, won't load up images and require the images be loaded the whole time.

a menu that you could go into, select a thread to load the images for, and load them, it would tell you when a thread died/404ed and stay in till you decide what to do.

I have no idea how hard this would be, but it would allow threadwatchers to exist in a traditional way and a sub way so you can have the stupid amounts, while keeping almost all the functionality you want out of one.

but again, page of images hits damn near everything regardless of site, so a way to make page of images check sites in a recurring order, either a user set x minutes, or a program handled x images over x time = check x way its handled now would deal with the need for thread watches altogether, along with, if you could set up a page of images for more then one site, give us the multi thread watcher.

either way i'm interested to see the developments made.

been running into 404's a lot with the archives for 4chan recently, desu seems to be unable to keep up with threads, and gookmoot seems to be considering making a paid feature and apparently has been making it harder for archives to function.

I think a multi thread watcher/recurring page of image's thing needs to happen somewhat soon.

opened a log for a 12k download page, resumed 3 images that failed, opened source the deleted image to see what it was.

looked at the 3 images in program, closed the log, program crashed.

>>7926

Thank you for this suggestion. I ran out of time this week to write parsers for non-4chan/8chan, but I will try to get pregchan and a couple of others working for v294. Please let me know how it works for you as I roll it out.

>>7928

There is another manage tags dialog crash fix in the release I am putting out later today. I hope this improves the situation, but please let me know if you still run into trouble.

>>7932

Thank you for this report. I think the fix to the manage tags dialog in today's release will also fix your issue–the filename tagging dialog has some of the same tag autocomplete entries that were not cleaning themselves up right. Please let me know how you get on in v293.

>>7936

I am leaning a bit towards making two systems–a 'simple' on that is basically the current one-thread-per-page one, and an advanced one that is a bit more like what you suggest.

As an advanced user, I don't usually want to browse through my thread watchers' results–I want to leave them until they are [DEAD], and then add a bit of sub-processing metadata and dismiss them into the inbox aether.

A full-on 'manage' dialog like subscriptions feels like overkill for ephemeral like threads, but some kind of summarised list tucked into a cleverer thread watcher panel sounds like the direction to go in. Or maybe a sub dialog off that download page, a bit like how the 'tag import options' now work.

I want to drop a thread url on the client and have it automatically get queued up in the background and only bother me again when it is dead.

>>7944

Thank you for this report. I apologise for the continued inconvenience. I have some more crash-fixes in today's release, for the manage tags dialog and the 'file import path tagging' dialog. If you hit either of these dialogs up to about 60 mins before this crash, this one may be fixed. If not, let me know and I will put some extra time into looking at the file import status frame and so on.