[ / / / / / / / / / ] [ dir / cute / egy / fur / kind / kpop / miku / waifuist / wooo ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog

Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 12 MB.
Max image dimensions are 10000 x 10000.
You may upload 5 per post.


New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Current to-do list has: 714 items

Current big job: finishing off duplicate search/filtering workflow


YouTube embed. Click thumbnail to play.

c2e469 No.4162

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v226/Hydrus.Network.226.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v226/Hydrus.Network.226.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v226/Hydrus.Network.226.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v226/Hydrus.Network.226.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v226/Hydrus.Network.226.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v226.tar.gz

I had a good week. I fixed and sped up a bunch of stuff, and I got the first bit of the new parser's gui done.

new parser gui

I'm now confident in how I want my new parsing engine to work. The objects are almost there, so now I'm mostly working on gui to manage it all. There's still a bunch to do, but if you expect to create a parser in my new system and would like to check out the general direction I am headed, check out the unfinished dialog at services->manage parsing scripts. I'm trying to emphasise easy in-gui testing and sharing.

I want to eventually move all parsing to this dialog, both html and json, but I'll start with some simple html web file->tag lookup (iqdb or "hey booru, here's an md5, do you have tags for that?" search) for the suggested tags control. I expect to further develop the dialog in the coming weeks (I have four or so more panels to write, and I need to tie in all the button events into real code and so on), and then the prototype should be ready.

otherwise

I sped up tag-heavy search results and fixed some things, including more Windows popup trouble and some bad sibling autocomplete count merging.

full list

- fleshed out a bunch of parsing script engine and gui framework and added an 'under construction' entry to the services menu. feel free to play around with it!

- fixed autocomplete same-service sibling count merging

- sped up how autocomplete counts are put together

- results will build faster due to an improvement in how tag rows are fetched

- results will build faster due to an improvement in how tag strings are fetched

- incidence taglist sorting now secondary sorts by lexicographic in the 'correct' direction (a-z for desc, z-a for asc)

- all listctrls now support ctrl+a to select all

- cleaned up some superfluous and possibly buggy canvas zoom calc when using the open externally button

- the zoom_switch event will now reset the drag coordinates (recentering the image) if the resultant zoom fits into the canvas frame

- the mouse cursor hides itself more quickly on media drag on Windows (so warppointer jitter is hidden)

- the popup message manager will be careful about switching from hide to show while the parent gui is minimised on windows, which should reduce the grey box problem

- the popup message manager will specifically re-layout on a transition from hidden to shown to stop the 'crushed into a corner' layout problem that would sometimes happen here

- popup message manager will now consider a minimsed gui as unfocused for the purposes the of the focus BUGFIX option

- added a new delayed popup message item to the debug menu for testing minimised popup creation

- vacuum maintenance period option is moved, reset to larger default of 30 days

- 'last session' will now always be updated on client close, even if it is not the default session

next week

I'll try to do more parsing gui stuff, mostly, but I'd also like to add the 'db path' command switch I've been talking about with some people that'll make multiple installs and unusual hard drive situations easier to manage.

92c80e No.4164

Keep being based, dev.


50749d No.4165

You want to move

from include import HydrusConstants as HC
outside the try-except in hydrus/client.pyw or the except-block will always fail when trying to access
HC.DB_DIR
and fall into the
except NameError, IOError
.

Is there any way to actively contribute code to the project? You don't seem to use GitHub except to push code.


52f2f9 No.4166

Honestly the search results appear to be a lot slower than previously. Doing a large-ish search for ~100-150k would previously could chug at probably 1-3minutes, at the moment it takes around 5+minutes


50749d No.4167

Also I'm almost sure that this error:

The program 'client.pyw' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadAlloc (insufficient resources for operation)'.
(Details: serial 25260 error_code 11 request_code 53 minor_code 0)
(Note to programmers: normally, X errors are reported asynchronously;
that is, you will receive the error a while after causing it.
To debug your program, run it with the --sync command line
option to change this behavior. You can then get a meaningful
backtrace from your debugger if you break on the gdk_x_error() function.)

Is caused by the popup windows that display information in the GUI during cleanup/optimization processes etc. I'm running this on Arch Linux and it keeps crashing when Hydrus starts optimizing the database while the GUI is open (as opposed to when the client is closed).

I enabled the 2 popup bugfix options, and it seems to have fixed the problem – although I'm almost sure that it would reappear if I focused on the window during the process. Could it be that you are constantly recreating the popup window and the X Windows System vomits at some point because of this? Maybe the created windows' X window resources are not properly cleaned up somehow or the windows themselves are just hidden, exhausting at some point an internal X11 resource?

Something to look at maybe. I have also seen this bug mentioned in the bug tracker and here previously.


df0245 No.4170

>>4165

I mirror the wish to contribute to this project. I've made some small fixes to improve the workflow for video management which I'd like to be able to PR.


50749d No.4171

Also something else I noticed, at least on Linux, I get this when starting hydrus-client:

> WARNING:root:pafy: youtube-dl not found; falling back to internal backend. This is not as well maintained as the youtube-dl backend. To hide this message, set the environmental variable PAFY_BACKEND to "internal".

youtube-dl is definitely on the PATH though. Does hydrus expect it to be in hydrus/bin? Maybe checking both, environment variables (PATH – afaik cross-platform for all systems) and that location might be a good idea?


c5d82a No.4172

>>4162

>but I'd also like to add the 'db path' command switch I've been talking about

Awesome, can't wait.


2ea45a No.4173

Are you planning on having an easy "import all tags" from iqdb, or will it be grabbing tags, then you have to manually double click each tag you want to add?


beb202 No.4175

On Windows getting:

PyAssertionError
C++ assertion "(itemid >= 0 && itemid < SHRT_MAX) || (itemid >= wxID_AUTO_LOWEST && itemid <= wxID_AUTO_HIGHEST)" failed at ..\..\src\common\menucmn.cpp(260) in wxMenuItemBase::wxMenuItemBase(): invalid itemid value
File "include\ClientGUIMedia.py", line 2661, in EventShowMenu
filter_menu.Append( ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetTemporaryId( 'custom_filter' ), 'custom filter' )
File "site-packages\wx-3.0-msw\wx\_core.py", line 12007, in Append

after a while when attempting to right click on any 2 or more selected gallery images and less frequently when right clicking a tag in the selection tags list on the left sidebar.

Only way to fix it is to restart Hydrus and it will work for a little while then give that error again.


2b59c9 No.4180

So is there some kind of API to easily access the various images + tags in the database and modify them exposed by Hydrus?

If there was things like these: https://8ch.net/hydrus/res/1914.html#2323 could entirely be avoided. As I understand currently this requires you to remove every image from your database that has missing tags for some reason, and then reimport them after running that tool. It would be much easier being able to write a command-line script that iterates over your images, filters them based on some predicate and then uses some booru API wrapper to add the missing tags.

Even updating tags that way would work – possibly making it easy to keep the public tag repository up to date with the public *booru tagging.


d23e76 No.4196

>>4180

This is relevant to my interest too, I want to interface with Hydrus to write my own substitute for the booru share option, I just need to be able to get image data and tags from hydrus itself for my own handling, editing is optional for my case, getting filtered results from a hydrus instance would also be nice so I wouldn't have to parse it myself, I just want to make a new frontend for external people to browse a giant subset of my collection(s). Allowing them to search my collection indirectly by asking the frontend which asks hydrus which relays the results, would be very nice.


c2e469 No.4215

File: 3bff476a0aafd3d⋯.jpg (2.86 MB, 2386x3606, 1193:1803, 3bff476a0aafd3df9218b10990….jpg)

>>4165

>>4170

I appreciate your interest, but I'm afraid I enjoy working on hydrus alone. I find collaborative programming stressful, and I don't want to burn out.

That said, if you would still like to put work in, there are plenty of python libraries or python wrappers for existing C libraries I would be interested in seeing be made. A multiplatform audio library (as in 'here's PCM data, now throw it to the speakers') would be great, or better pdf parsing/rendering, or more reliable swf thumbnail generation, an svg renderer, a better python-friendly way of dealing with menu events in wx, and many more. Let me know if you are interested in this sort of thing.

>>4165

I actually put that NameError catch there for that exact purpose. If DB_DIR can't be figured out, then I can't find a place to put the crash log, so I just dump to terminal.

As it happens, I'll be rewriting that system in the near future when I add the db location command switch. Once I'm done, I expect the DB_DIR discovery to be a bit neater.

>>4167

Thank you for this information. The popups are proving a continual headache to get multiplatform and window manager ok. I've looked at this code closely several times over the past few weeks, and most of the problems seem to be related to Show or Hide or Layout occurring when the window has some status the window manager isn't happy with (it is minimised, has no children, whatever). I'm not sure I can account for each manager's idiosyncrasies, so I am starting to think I should just have a BUGFIX option that just shows '(7) messages' on the status bar and launches a separate frame to display them..

Multiplat popup message frame in wx would be another great library I would love to see!

>>4171

I don't know much about this, but I think pafy is looking for youtube_dl python package. I guess you have a built version? It seems to be on pypi:

https://pypi.python.org/pypi/youtube_dl

So if you go:

sudo pip install youtube_dl

You should be sorted.

>>4166

Thank you for this update. Can you tell me some more info about your machine? Is it new or old? Are you off an SSD or an HDD? If an HDD, is it fragged? Do you sync with my PTR? How many files do you have total? What happens if you go help->debug->load whole db into disk cache (it'll take a minute or two to do this) and then rerun a big search?

And what happens if you include system:limit=1000 or so in your query? Are things snappy, or is there additional lag? What about if you add system:number of tags = 0? For the 0 tags query, the initial part will take a while, but the results building part should not use these recent optimisations–is it as slow as the others, or much quicker?

>>4173

First step will be an suggested tags control column in the manage tags dialog of any single file. It'll have a 'fetch tags from (iqdb/wherever)' button. You click the button, and the client asks the site for tags for that file and then presents them in the column so you can double-click them.

Once I am happy with the system, we might be able to expand it into a 'auto-tag all these selected files based on my neato dabooru file-tag lookup script', although we'll have to be careful to avoid creating a system that spams sites unnecessarily.

>>4175

Thank you. I am aware of this bug. It has actually been around for a while, but was being silenced. I haven't seen an easy fix yet, but I intend to rewrite my menu system and expect that to solve the issue. For now, reboot is the fix.

I'm still not sure why the ids are overflowing. Let me know if you discover any particular menus you are creating that speed this error up.

>>4180

>>4196

There is nothing yet. I'd like to eventually add this stuff to the local server as a series of GET and POST requests. If you are brave, you could write a python script to pull this info straight from the db itself, which is just sqlite, although I wouldn't recommend altering the db. I'm happy to give you pointers on all the tables if you want to try this.

Also: HydrusTagArchives can be used to do mass manual tag editing. Write a python script to create your HTA and then use the gui in the client to update/different the content with a particular tag service.


bd8f74 No.4219

>>4215

It's probably much too late to suggest this, but I have a feeling that the client would have worked out much better cross-platform wise if written in something Like PyQt or PySide. Qt is seriously good and has many many features: including exactly what you are searching for: http://stackoverflow.com/questions/2217035/simple-cross-platform-free-audio-library-for-raw-pcm

I know this is a moot point right now, but maybe, just maybe, something to consider in the future. I'm not actually sure how closely coupled your GUI is to your code (although I doubt it is much at all), but if you notice more and more problems cropping up with wx you might consider replacing the frontend at some point in time – or at least prototyping one in Qt and see how you like it.

> A multiplatform audio library (as in 'here's PCM data, now throw it to the speakers') would be great, or better pdf parsing/rendering, or more reliable swf thumbnail generation, an svg renderer, a better python-friendly way of dealing with menu events in wx, and many more. Let me know if you are interested in this sort of thing.

List a few more of these requirements and maybe people can try their hands on a few. Although I assume this would need to be Python 2.


f77b99 No.4222

>>4219

>>4196

>>4180

I can only second these. A language-agnostic API would be ideal. Hydrus could expose such an interface as a local webservice and scripts/applications could interact with that interface. Ultimately this could be a way to write entirely new interfaces in different languages/toolkits for Hydrus too, while Hydrus handles all the complicated backend stuff.

Automatically monitoring threads relevant to certain keywords and adding them to Hydrus' downloads automatically would also be possible.


52f2f9 No.4226

>>4215

Older, i5-2500k 8gb.

SSD for Hydrus/DB/Thumbs, HDD for storage.

I don't sync with your PTR.

~180k total.

Still slow after loading into disk cache.

Limiting to 1000 has the same effect. 0 is instant (as far as i can tell).

The search tallies the total what it will be fetching instantly, but the "Loading x of Total" clicks up at a very consistent rate (256 per ~1ish second). this used to be a couple of thousand, and not consistent at 256.


c2e469 No.4228

>>4165

I worked on this today and I realised what you meant here–I apologise, I misunderstood how the exceptions were propagating. It is all fixed for tomorrow along with the db_dir command switch.


461607 No.4235

File: 78f63ad72f838fa⋯.gif (479.73 KB, 250x188, 125:94, 78f63ad72f838fa0b353e25c24….gif)

My client was crashing whenever I clicked/viewed any .gif thumbnails, so I downgraded to version 225 and that solved the problem

sorry if I'm reporting this too late, but I only noticed it myself not long ago


c2e469 No.4251

File: e44b4535ab1c1f2⋯.jpg (370.85 KB, 1217x1024, 1217:1024, e44b4535ab1c1f21864b921a6a….jpg)

>>4219

I broadly agree. I can't remember why I picked wx in the first place, although I'm quite fond of it now, despite its flaws, and I have a ton of experience in it, particularly in Windows. I'm obviously locked into it now–the amount of work to rewrite all the gui stuff in a new library would just take too long. Perhaps this will change in the future if I further decouple the db from the gui, allowing for many different interfaces.

In the same way, as you mention, I assume I'll have to move to Python 3 at some point. I guess I'll just have to take a month to learn about all the differences and convert everything over. I'm also waiting on wxPython Phoenix to happen.

>>4222

I'd like to move in this direction.

>>4226

Thanks for this follow-up. I may be seeing similar on my own laptop. I've got a job to look into this in more depth.

>>4235

Thank you for this report. Some others have reported similar. Please try the v227 I put out a couple of days ago and let me know if it still breaks for you. Also let me know if there is any difference when the preview window (in the bottom-left) is hidden or shown when you select the gif.


461607 No.4255

File: da179fa14aca27d⋯.gif (1024 KB, 339x300, 113:100, da179fa14aca27da7827a42fc1….gif)

>>4251

v227 is working fine for me, thank you!

>Also let me know if there is any difference when the preview window (in the bottom-left) is hidden or shown when you select the gif.

v226 still froze upon selecting a .gif, even after I hid the preview window under file->options->GUI




[Return][Go to top][Catalog][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / ] [ dir / cute / egy / fur / kind / kpop / miku / waifuist / wooo ]