[ / / / / / / / / / / / / / ] [ dir / baaa / choroy / dempart / doomer / hentai / lounge / monarchy ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.
Name
Email
Subject
Comment *
File
Password (Randomized for file and post deletion; you may also set your own.)
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 5 per post.


New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Experienced user with a bit of cash who wants to help out? ---> Patreon

Current to-do list has: 1,617 items

The program is now on Python 3! Check v335 release post if you need to update from before then!

Current big job: OR search


YouTube embed. Click thumbnail to play.

71589f  No.6653

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v271/Hydrus.Network.271.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v271/Hydrus.Network.271.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v271/Hydrus.Network.271.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v271/Hydrus.Network.271.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v271/Hydrus.Network.271.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/master.zip

I had a challenging week due to some IRL stuff. I got some good fixes in for hydrus though!

My Linux build machine died this week. It thankfully happened on Thursday, so I had plenty of time to figure out a replacement. I believe everything is essentially the same as before, but please let me know if you discover any problems.

page tab DnD works better

I have improved the new page tab drag and drop for all platforms. It should now work for Linux and OS X, and on Windows some old look and feel should also be fixed.

OS X seems to need a double-click to start the DnD. I don't know anything about customising Cocoa, but let me know if you experience anything different.

I also had some trouble figuring out OS X's 'page was dropped here' coordinates and ended up fixing it with a fudge. I will revisit it, but if you discover your page DnDs are not ending up where you expect, please experiment with dropping the page ~20 pixels higher or lower than your target, and let me know what you discover. A 100% zoomed screenshot of your tabs would also be helpful!

couple of other things

'page of pages' pages now start with an empty search page!

Some network engine stuff should work better and fail more gracefully!

The 'delete_file' shortcut action should now work properly on media viewers and thumbnail panels!

The file import status window will now sort accurately! I have also added a '#' column to represent the download queue order.

full list

- fixed an issue where the notebook 'motion' event was being consumed, disallowing the regular OS 'highlighting' on tab mouseover

- fixed page tab drag and drop crash for Linux

- fixed page tab drag and drop coordinate calculation for OS X

- page tab drag and drop should now work for all platforms

- new 'page of pages' pages now start with a blank file search page

- fixed several issues with row display in the new listctrl

- added a # index column to the file import status window, so you can resort the 'to be imported' order reliably

- fixed an issue where the 'delete_file' application command was not firing on an appropriately mapped shortcut

- fixed an issue where automatic bandwidth override would only fire if its page was in view. it now occurs at all times

- fixed an issue where the file import status frame, as launched from manage subscriptions, would sometimes be non-responsive (typically for Linux)

- fixed the 'pattern shortcuts' button on the export files dialog, which was misbehaving for certain choices for some window managers

- fixed an unusual error where middle-clicking to create a new page while a menu was open would lock the application from receiving new key/mouse events until program focus was lost, wew lad

- updated some old network code to the new engine

- added some delay to network connection error reattempts so that instant fails due to local network disconnect (e.g. while computer is waking from sleep) do not consume all the reattempts too quickly

- greatly simplified low-time-delta bandwidth testing and reporting, improving essential accuracy and overall feel

- gallery download errors will now print additional info to the log–let's see if we can gather more info on the gelbooru 404s

- wrote a patch for users of 2.X opencv who were running from source, who were unable to boot

- due to hardware failure, had to make a new Linux build computer. some libraries are updated, but build _should_ be the same–let me know if not

- updated to opencv 3.3 for all releases

- misc old menu code updates

next week

I was a bit short on time (and energy) this week. I was able to keep up with messages, so I'm happy with that schedule change, but I still have some IRL obligations blatting my other hydrus work time. It should clear up sometime next week, and in fact, I may discover a bit more time to work on hydrus in the coming future.

Otherwise, next week I'd like to go back to finishing the network engine stuff and putting some time into longer-term issues and other overdue rewrites. There are a bunch of broken unit tests that I should get to before they bite me.

78784b  No.6654

>>6653

Thanks Dev!


bab29e  No.6655

>>6621

Ok, had to deal with some infection shit and am playing catch up, im the person with the crash...

so lets start off with the current tab load

there are currently 142,000~ image open in total, im working on getting 46000 of them closed out, but the process is slow and tedious

There are 37000 that cant be closed till desuarchive is back up as they are download windows with around 6000 failed images, I am able to get the images, but I need the image drive that they were on to be back online.

there are 11 tabs of tabs right now, and 84 tabs open right now if I counted right.

When I ran into the issue of starting thread watches in rapid succession I had significantly less tabs open, probably around 40 less, and around 16000 images less.

one crash was caused by likely having to many tabs open again as the way it crashed reminded me how the program acts when you have to many fuckin tabs,

after that error popped up in the screenshot, the program would crash every 20-30 minutes once innital load happened. I closed out around 10 tabs and it seemed to have solved that issue though it was a new one for me

and just loading thread watchers in in rapid succession, I think it was 7 tabs of them before one finished caused it to crash, this was when I had probably 40~ fewer tabs at least.

Have yet to download the new client to look at it, will get that soon.


c07373  No.6656

>>6653

Hello hydrus dev. Following the discussion in the previous thread regarding the 404s heres what the error log gives me:

2017/08/31 03:15:52: The url http://simg4.gelbooru.com//images/61/9d/619d794355209893a169dd041866b739.png gave the following problem:

2017/08/31 03:15:52:

2017/08/31 03:15:52: Exception:

2017/08/31 03:15:52: NotFoundException: <html>

<head><title>404 Not Found</title></head>

<body bgcolor="white">

<center><h1>404 Not Found</h1></center>

<hr><center>nginx</center>

</body>

</html>

File "threading.py", line 774, in __bootstrap

File "threading.py", line 801, in __bootstrap_inner

File "include\HydrusThreading.py", line 146, in run

self._callable( self._controller )

File "include\ClientDaemons.py", line 302, in DAEMONSynchroniseSubscriptions

subscription.Sync()

File "include\ClientImporting.py", line 3527, in Sync

self._WorkOnFiles( job_key )

File "include\ClientImporting.py", line 3127, in _WorkOnFiles

downloaded_tags = gallery.GetFileAndTags( temp_path, url )

File "include\ClientDownloading.py", line 987, in GetFileAndTags

self._FetchData( file_url, referral_url = url, temp_path = temp_path )

File "include\ClientDownloading.py", line 561, in _FetchData

HydrusData.PrintException( e )

File "include\HydrusData.py", line 896, in PrintException

trace_list = traceback.format_stack()

2017/08/31 03:15:53: Error when processing http://gelbooru.com/index.php?page=post&s=view&id=3850223!

2017/08/31 03:15:53: Traceback (most recent call last):

File "include\ClientImporting.py", line 3127, in _WorkOnFiles

downloaded_tags = gallery.GetFileAndTags( temp_path, url )

File "include\ClientDownloading.py", line 987, in GetFileAndTags

self._FetchData( file_url, referral_url = url, temp_path = temp_path )

File "include\ClientDownloading.py", line 563, in _FetchData

raise e

NotFoundException: <html>

<head><title>404 Not Found</title></head>

<body bgcolor="white">

<center><h1>404 Not Found</h1></center>

<hr><center>nginx</center>

</body>

</html>

It seems its attempting to download a http file where only https is provided.

https://simg4.gelbooru.com//images/61/9d/619d794355209893a169dd041866b739.png

This is the correct link it should be grabbing. If I input the http link in my browser it gives me a 404 or redirects to https. Seems like only some images redirect to https while the others 404, which explains why only some images get grabbed by the client.


1ba4c1  No.6657


% git rev-parse v270
fac67766ebc17c42db10f2346c9f371e4272f1ac
% git rev-parse v271
fac67766ebc17c42db10f2346c9f371e4272f1ac

The v271 tag is attached to the wrong commit


41ea17  No.6659

File: 6b0da0a67cda238⋯.png (139.86 KB, 1280x1024, 5:4, Buttonless.png)

Minor errors on Linux mint - All the shadowing around buttons have become indistingushable from the background, making it really hard to tell what's a clickable button and what's just text.


7d6ee2  No.6662

File: 30f995a973a44d1⋯.png (17.45 KB, 634x98, 317:49, client_2017-08-31_20-53-35.png)

I keep running into an issue with drag and dropping thread watchers, They end up crashing somewhat often even when it's one at a time, but now I have an new problem with them, I had a few added in that were doing initial thread checks, so once the program loaded, it would hit those thread watchers and crash. it seems that every time I open the program it gets a couple more images, I'm hoping that If I cycle it enough it will sort itself out, i'm losing hope of this one though

Now because there are a few theadwatchers that are stuck open and I can't exit them fast enough, and the program doesn't realize I exited them i'm in an error cycle

wxmsw30u_core_vc90_x64.dll

seems to be the issue.

so I am thinking of ways to fix it, or mitigate the error cycle…

Is there a way to add to the start up options a way to save the prior session? That way when sessions crash due to excessive tabs, or a shitty tab, it can be re opened with some effort.


b70135  No.6665

This might be a bit late, but the GUI fix you applied in whatever previous version to make the file import status list so fast and responsive and non-crashy is much appreciated. Good job


7d6ee2  No.6667

Is there a way to open saved sessions one page at a time? if not I think that could be a very useful feature.

also, is there a way in program to set how many lines the mouse wheel moves images? I know some programs use customizeable values, the reason I ask is because 3 lines is what I consider good for nearly every application, even when applications make big moves, most things have smooth scroll or something like it at this point.

However, in the program 3 lines is 3/5ths the page, and without any form of smooth scrolling it can be a bit jarring. is a custom option for the program for how far the mouse wheel goes something that can happen?


744a82  No.6669

File: 446f68118a52bb9⋯.jpg (3.34 MB, 3235x2348, 3235:2348, 446f68118a52bb9f00d96ba5cb….jpg)

>>6655

Thank you for these details. I will try to rejigger the order of events of a session load to be a bit more polite. There's no resource reason why a heavy session load should cause a crash, but perhaps your Window Manager is getting caught up on some thread trying to update something before everything is fully loaded.

I am not sure about the 20-30 minute crash. I am planning on simplifying some behind the scenes update timers soon, which will reduce some overhead due to having many tabs open. This might fix your issue, or at least alleviate it.

>>6656

Thank you, this was extremely helpful. It looks like some longer-term users still have an 'http' version of gelbooru in their booru settings, and this is being combined to build the faulty CDN link. I have an explicit fix for it for next week. I suspect that if you go into services->manage boorus and set your gelbooru entry to be https, your problem will be magically fixed. Otherwise, please let me know how v272 works for you.

>>6657

Thank you. I done fucked up my sync order-of-events as I was putting the release together. Is there an easy way to say 'this release should point at this commit' in the github app or site? Or is it simpler to just say 'get the master this week'?

>>6659

That is pretty odd! I haven't changed anything my end with buttons, afaik--could Mint have changed anything with its window manager? I think I just pull system colours for simple widget colours, and the buttons I don't set anything other than defaults for things like borders.

If you go help->about, does that have an ok/close button, and is it broken in the same way? I think that button is created completely automatically at the wx/system level, so that would help us figure out where the wrong colour/style is coming from.

>>6662

I am sorry you are having this problem. I am not sure why it is occurring, nor why your thread watchers are crashing. Strangely enough, when I search for the "wxWindowNR" error, there aren't many results, but one is >>6275 from a couple of months ago. Was that you? It might be something like the drag-and-drop cannot complete due to some internal gui object count being exceeded by the subsequent 'detach page from here, insert it in here' operation. How many total tabs are we talking--20 or 100+? I understand older versions of Windows have smaller gui object limits, so you might be being nobbled at, say, ~80 pages.

I have a longer-term plan to reduce total gui object count for download pages and hence increase the max number open at any time.

Saving anything before a crash is tricky because I can't detect when one is about to occur, and after it happens, I can't do anything. You will still have the 'last session' session however, so I recommend you do something like:

boot the client, select 'just open a blank page' on the 'you crashed last time' dialog

go services->review bandwidth usage and set a restrictive rule for the 'global' context that is like '1 request per day' this will throttle the thread watchers while we try to save them

go pages->sessions->load->last session and cross your fingers

if it all loads ok and is stable, identify some thread watchers or other pages to kill, to reduce overall count.

shut the client down

try booting it again

try removing the bandwidth rule

You likely don't have time to do this, but you can go help->debug->save 'last session' session to manually save your 'last session' if you expect a crash but want to lock in your changes immediately. If I remember right, last session is otherwise normally updated every five minutes or so.

It is possible that the session load changes I made with >>6655 in mind will improve your situation as well. Please let me know how you get on here, and with v272, and if things are still a problem, I will see what more I can do.

I've also thought of possibly expanding the crash recovery dialog to be like firefox's, where it'll say 'which of these tabs from last session do you want to load'. This might help you eliminate bad thread watcher tabs, but I would ideally rather figure out the underlying problem of why they are crashing in the first place.

>>6665

Great! I am trying to find time every week to copy that new listctrl class to all the other places the old listctrl is used in the program. I expect to make further improvements to the new class, including adding column selection for 'busy' instances like on the manage subscriptions dialog.


744a82  No.6670

>>6667

I think more session controls is worth doing, especially with some of the recent problems. Maybe I should just schedule time to make a proper bit of gui for it and then apply it in several places.

I don't think I have much control over the scroll stuff, but I might be able to do something. As I understand it, your windows/mouse driver sends x scroll events per scroll notch you hit. Somewhere in Windows options I think you can alter this, the default usually being 3.

On my end, I get each of those x scroll events as separate events. It automatically causes the thumbnail window to move a minimal unit with every event, and for me, a minimal unit is 'one thumbnail height'.

Rather than try to wangle some weird 'catch and swallow event 2 and 3' kind of option, I think I could add some sort of 'set minimum vertical scroll distance to be 1/2, 1/3, 1/4 of thumbnail height, but we'd probably have to experiment with it in case that breaks some other rendering calculations.


05a077  No.6672

>>6669

Just tried it. Changing gelbooru to https in the manage boorus window did nothing. It's still trying to grab http instad of https according to the log file. I did delete all my logs and ran with an empty one just to make sure it was spitting out the same error and I wasn't reading it wrong.


7d6ee2  No.6678

File: 1bf7bce9e85f198⋯.png (835.62 KB, 1738x849, 1738:849, chrome_2017-09-03_11-56-22.png)

File: a03c80f1e748cc6⋯.png (740.56 KB, 1254x1053, 418:351, chrome_2017-09-03_11-57-30.png)

File: 4e3d3d7a1909404⋯.png (97.61 KB, 946x663, 946:663, chrome_2017-09-03_12-01-15.png)

File: eb702542d7fcc3e⋯.png (13.83 KB, 1101x192, 367:64, chrome_2017-09-03_12-17-21.png)

>>6669

yea that one was me, that one if i remember right was hitting a hard page limit in the program when that issue was happening.

windows 7 ultimate, and I have had 168 pages open open before the program started acting funny before.

as for having time, yea, I have enough time when this was happening to close out 4-6 tabs once everything was loaded in, the session with lots of crashing was saved as dfgh, however most of the threads in the thread watcher are either expiring or getting close to it so there is no real reason to try to recover it anymore as I loaded up a saved session from when I upgraded to the 271 client.

There are about 6 of those ranging from 'save the session' when I didn't remember what the number was, 'pre 271' when I did, and 'save the fucking session already' so if its not obvious, there was an issue where I could not see any of the sessions I was saving, not sure if its a bug or what was going on there. will keep in mind global settings on a blank page next time something like this happens, fingers crossed it doesn't.

as for loading specific tabs, honestly, that would be helpful, especially in an instance I just toss my hands up and say fuck it 'I did something stupid and just want shit to work' However a combination of being able to go through saved sessions tab by tab would probably be better.

If this is something you can do quickly and would be simple, go for it, if its something that would take real time, then fuck it. If I remember a while back you said/had an idea for thread watches being consolidated into one tab, I have to ask, was that page of pages or are these two separate things? because thread watchers being handled through one page on its own would easily handle most if not all issues I have currently and would likely be a better use of time.

on the topic of old shit

>>1288

I remember this, from what I remember e-hentai and exhentai is different front facing and will have different content. things the front facing e-hentai decided to expunge will typically still be on ex, now the thing I would use it for is this the first image is just a typical old gallery, the only way to get all the files is either pay the currency and download, or go file by file to download it.

on the file to file page I chose one where it was resized at the bottom there is a button for getting the unaltered image

Now this is a very small gallery so its not to much of an issue, however just as an example a large gallery there is no way to download it with points for most people, its 1600+ images and 4gb, which would be a nightmare to go through.

now, through a quick experiment I found out that 200kb is about 1 point I beleive, most of the older galleries or manga are under 200mb from my math each day would be able to regen 4320 points or 864mb. If the downloader was set up to only get 36mb an hour it would be able to keep up with regen

At the very least for me going though larger old books, or larger old galleries it would be a god send to have something that can automatically do this and set up a bandwidth limit for the domain, either being 700-800~mb daily or 36 hourly so I can easily grab some of the older galleries without an absolute hassle.

On that note of e-hentai and exhentai, is there a way to export files with original filenames? I can't think of a use for this outside of downloading an older manga and remaking the rar/zip of it

on mouse wheel, I have an idea, no idea if it would work.

With minimum vertical scroll would there be a way to have a user changeable number and the math is done behind the scene? like, the option says "Lines one unit scrolls [X] thumnails wised to move [X.X]"

This way the program knows how many units one tick will scroll and will convert it to how many thumbnails you want to move.

In all honesty, it doesn't need to be the scroll wheel, that's just the most convenient, a shortcut that would move the thumbnails up or down on its own would as helpful, because using arrows when I have it full screen, the arrow method will select images that only have 1/5 or 1/10th of their thumnail showing but not bring the row up, and loading previews for these can also cause the program to hang up a bit. if there is a way to/shortcut for just scrolling 1 or 2 rows at a time, that would likely replace the scroll wheel for me


4eed50  No.6679

File: b15ccc92526072b⋯.jpg (303.78 KB, 840x1200, 7:10, temp1.jpg)

>>6656

>>6669

I was having this same problem as well trying to download stuff from gelbooru (for the tag zounose). However, I was able to get it to work like so:

1. Change the gelbooru thing to use an https link instead of an http one.

2. Stopping the current download session and deleting all of the results it found.

3. Restarted Hydrus.

4. Opened a new gelbooru download session and re-entered the tag to download.


05a077  No.6680

>>6669

>>6672

I just came back to post this but >>6679 got it in before me. I realized there was a link cache for the subs which I forgot to delete after setting gelbooru to https. I did that and let it slowly accummulate its links again for a few hours and so far no 404 errors. I will assume this issue is fixed.

Thanks hydrus dev!


744a82  No.6686

>>6678

Allowing thread watchers to have multiple urls is something I would like to do as part of the downloader overhaul, in the same way I would like to allow subscriptions to support multiple queries for the same site and check period and tag options and so on. It is part of a general push this iteration to have more logging and separation at the gallery parsing end, rather than tossing all found urls into the file import status window together and then forgetting how they were found.

I see thread watchers having multiple queries as a new thing, perhaps by having the management panel on the left listing the different queries in a list and then despositing the different sets of results in a notebook inside the thumbnail panel on the right. This would mean each page would then support multiple sets of thumbnails.

The main benefits would be increased user workflow and reduced number of ui objects per thread being watched (which affects your problem). The main drawbacks would be a significant increase in code complexity in certain places.

Given the success and relative ease of the page of pages change (which has relieved the workflow angle, since you can dump all your thread watchers in their own nested area so they don't clutter everything else up), I am not sure if I will do this after all, at least for thread watchers. I can and already plan to reduce gui object count per import page by moving the file and tag options to dialogs launched from buttons, rather than the currently weighty expandable panels.

I am still thinking about it. It'd be nice to do, but this current overhaul is already dragging and there are a hundred other things I really need to do first.

Thank you for the info about e-hentai. The new bandwidth engine supports exactly these sorts of rules. I am sure someone (if not me) will figure out a hydrus downloader for it in the new engine. I also intend to support the unusual sadpanda login situation in the new login engine. I will add some default bandwidth rules for those domains as that happens.

We'll see about the scroll position stuff. I have not had luck trying to interfere with scroll events previously–the different OSes handle it all slightly differently, so you get a lot of jittery bugs–so I will try a simple solution first and see how that works out.

>>6679

>>6680

Great, I am glad this got sorted. Note to anyone else who has this: you can do this to fix it now, or you can wait for v272, which will fix it for you.


7d6ee2  No.6687

File: f9960c9a2131cf0⋯.gif (2.35 MB, 664x508, 166:127, 2017-09-04_21-41-08.gif)

File: 126a71fa40105ba⋯.gif (3.13 MB, 652x1070, 326:535, 2017-09-04_21-41-51.gif)

>>6686

If a custom scroll amount is not doable, would smooth scroll be doable? that would help with how jarring 3/5the the page changing can be. Or a shortcut that will scroll down, the main reason I don't see the arrow keys as an option is they select images which can cause the program to hang, if there was a way to scroll down without selecting images, I could use that. Lets also say you don't need to make the shortcut convenient right away, I have an mmo mouse and a macro pad I can use.

now, just for some examples of what arrow scrolling is like, both down and up for me select images off screen, and in down you can see the program hang a bit, the moment the image loads i press down after the initial start and till the last one, if the program didn't hang, that would be a great method to use, the main reason I want the scroll wheel to work is scrolling with it doesn't cause the program to hang at all, and there is likely very little you can do to stop the hanging I am encountering till the program is further along if even then as its likely a hdd being slow issue more than a program is lagging one.

as for thread watchers… i'm honestly a bit torn on how I like them, don't get me wrong they are fucking great at what they do when they don't crash, but i'm thinking of ways to make them better, and I actually think the program is very close to having a method already inside it, going to test it with a thread on /b/ now.

Ok it works the way I thought it did.

page of images when I import a link it grabs all the images from it and goes to town, and when I repaste the link when some new images are posted, it only adds the new images, and one link to another board because that's what 4chan does with its ad area now. Now I only used this on 4chan, 8chan, and 4chan derivative sites, but I just checked with 7chan, it works there too.

Now, instead of the pages just going away after you paste them, if you added a recheck timer to it then you have just effectively expanded thread watcher to cover any site that download page of images works with.

Now in the interest of not haveing just a fuckhuge page of images, images seem to remember where they were acquired from, and where they were attempted to be reacquired from. so instead of having like in thread watcher where all the website info and stuff is on the left hand side, where images are could hold the website, how many images it has, and time to next check, when selected all the stuff of the normal thread watcher displays that links data, with an option to open the thread in a new tab, at which point it would call up all the images that url provided in a new tab. Now because its not a normal thread watcher and instead a page of downloaded images, a 404 may not be as noticeable or detectable, so a small bit of 'if most of the links are no longer there, something fucking happened' warning at which point it turns the link or whatever info you see for threads from black to red, either prompting you to look at it yourself or assume its done and review what it got or just close it.

from what I can tell, to turn page of images into a thread watcher that can handle more than 1 url everything is already there, not the above idea adter 'fuckhuge page of images' but for a basic thread watcher based on it, all that's missing is the automation aspect

this would have two benefits, 1 being all the thread watchers are in one place but also they would largely work one task at a time, instead of the current fuck if I know… 54 thread watchers I currently have going which will step on each others toes




[Return][Go to top][Catalog][Nerve Center][Cancer][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / / / / / ] [ dir / baaa / choroy / dempart / doomer / hentai / lounge / monarchy ]