[ / / / / / / / / / ] [ dir / cute / egy / fur / kind / kpop / miku / waifuist / wooo ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog

Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 12 MB.
Max image dimensions are 10000 x 10000.
You may upload 5 per post.

New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Current to-do list has: 714 items

Current big job: finishing off duplicate search/filtering workflow

YouTube embed. Click thumbnail to play.

eedd83 No.3379

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v217/Hydrus.Network.217.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v217/Hydrus.Network.217.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v217/Hydrus.Network.217.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v217/Hydrus.Network.217.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v217/Hydrus.Network.217.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v217.tar.gz

I had a fantastic week! I fixed some important problems with last week's video renderer update, caught a big source of lag for pages with many thumbnails, and prototyped a 'related tags' tag suggestion control.

A user created a discord server for hydrus discussion. I don't usually do chat, but if you are interested, you can check it out at: https://discord.gg/vy8CUB4

first, new builds

I updated some important libraries, and my Windows python, and my Ubuntu dev machine from 14.04 to 16.04. This should generally mean improvements for everyone, but I had some new hurdles putting the Linux and Windows builds together today. I have it all working for my dev and test machines, but please let me know if you have any trouble. There are no db updates this week, so you should be able to downgrade back to v216 no prob, although I would, as always, recommend that you maintain a weekly backup before you update, just in case.

The Linux build has now grown to 125MB as well, for some reason! I guess 16.04 has larger libraries?

video renderer

Although I was pleased with last week's progress on the video renderer, I was honestly rushed. There were some big mistakes that meant certain situations would bloat with memory or spam laggy threads, which meant the whole thing was five steps forward and four back. I've now given it a full pass and am much happier with it. Small gifs should scale up even to very large screens no problem and multi-minute webms should correctly buffer without any huge hitches. Let me know if you discover any new bugs!

I've added a bit of gui feedback for this–the current frame buffer (i.e. the span of frames that are rendered and ready to put on the screen very quick) is now drawn straight on the scanbar below any animation! Small and short animations will be able to hold the entire image in memory, while larger ones will have a 'chasing' buffer that will keep up with the scan caret. It gives a nice indication of how well the client is currently keeping up rendering things, and if you like to drag the caret back and forth, it lets you know if and when the stream will hitch to regen a new buffer. This sounds more complicated than it is, so just check it out–I really like it.

Also, semi-related: If you set a mime to show the 'open externally' button, it'll now also show its thumbnail, if it has one, above the button. I'd like to do similar for the 'embed' button as well, just to give a preview of what you are launching if you are browsing through a fullscreen media viewer.

thumbnail lag

There's been a quiet bug in for a while that meant doing a large selection change, like hitting ctrl+a, would queue up all the thumbnails selected for renderering. This was fine when there were only 500 files in view, but when there were 10,000 it could be a cause of subtle client slowdown for several minutes as every thumbnail was individually loaded from disk.

I apologise for this–it was, like most of these difficult-to-spot problems, a simple mistake of one similar variable name for another. I've fixed it now, and when the current selection of thumbnails undergoes a visible state change, only those that are visible in the scroll buffer will be checked and redrawn.

suggested tags

I had a bit of extra time to spend on hydrus this week, so I also managed to get a decent first version of a 'related tags' tag suggestion control done. This sits on the manage tags window and does 'files tagged abc are also often tagged xyz', presenting xyz in a list so you can easily double-click them.

So, if you have a file with the two tags 'character:ayanami rei' and 'series:neon genesis evangelion', it will reliably suggest 'blue hair', 'red eyes' 'plugsuit', 'character:ikari shinji' and other useful related tags. This was quite tricky to get right! Once I got it counting and filtering correctly, I was surprised at how good its suggestions sometimes were. I'm enthusiastic about the potential of other similar time-saving workflows in future.

To test this control, you can turn it on by going to file->options->tags and hitting the 'related' checkbox at the bottom. If you are interested in this stuff, please check it out and let me know where it works well and where it fails. There are three numbered buttons that control how long to spend polling the db for suggestions–longer time tends to give more accurate and comprehensive results.

I don't want manage tags to end up huge, so I'm thinking of collapsing the various new suggested tags controls into a notebook control with tabs. I can also see that some people with larger screens might still want several columns in a row, or perhaps stacked on top of each other for a portrait monitor, so I'll also add other layout preferences.

full list

- fixed some high-res video streaming thread scheduling problems with the new video renderer

- fixed a cause of huge memory bloat with greatly upscaled videos

- to improve seek response time, streaming buffer for the video renderer has a much smaller cap

- renderer throttling calculations are more sensible and reliable

- the video renderer discards frames to save time if they happen to still be in its buffer

- the video scanbar now displays the current frame buffer around the caret!

- video canvas now recycles the same frame blit bitmap to save a little time

- wrote a prototype related-tags suggestion 'service' for the suggested tags control

- you can turn it on and set some options for it at options->tags, feedback would be appreciated

- munged increasingly complicated components of the suggested tags control into a clean and proper self-hiding panel

- fixed a very important bug that was failing to filter visible thumbnail fetch on mass select and thus massively slowing down the client on large ctrl+a-like operations

- open externally button now shows the media's thumbnail, if it has one

- open externally and embed buttons now use hand cursor

- the simple path tagging dialog panel now cuts off .jpg extensions from filenames on filename parse

- if the string component of a generated file export path already ends in the correct .jpg extension, a second will not be added

- ipfs unpin will no longer break if the file was already unpinned

- the hydrus server now gives filename (for a file save as dialog) correctly on a content-disposition header (this affects the client's local booru as well)

- the secondary sort can now be a namespace or rating sort

- fixed some potential init problems with some dropdown controls

- an edge case object-missing cache retrieval bug is fixed

- updated openssl on os x, which might have fixed some problems

- updated python on windows, which updated openssl and a bunch of other stuff

- updated sqlite on windows

- updated linux dev machine to ubuntu 16.04, so a variety of packaged libraries are updated

- fixed auto server setup if the client is launched from a windows cmd window

- misc cleanup

next week

I've got some long emails/posts waiting for me, so I'll put some time aside for that, and I'd like to do some general bugfixing and polishing of existing stuff.

5c7ffa No.3382

I like it! Although I'd prefer if related tags wouldn't show tags already on an image, nor remove tags already on an image if you double click it in the related tags frame.


5c7ffa No.3383

On another note, after applying many tags to one image, it takes a while before the main GUI becomes responsive again, with Windows thinking it's stopped responding for a moment. It's definitely slower than v215, which is what I had before.


8fe915 No.3384

Good stuff. I agree with >>3383 though, I've noticed the manage tags window lagging when I paste many tags from image to image.


6d0876 No.3387

>>3383

I've noticed this too, applying many tags to an image with the PTR causes Hydrus to freeze for perhaps a good 2-3 minutes before I can use Hydrus again. Makes individually tagging many files almost not worth it at the moment.


07e05e No.3394

is media player for webm not meant to play audio?

not really fussed


b8e873 No.3397

having a problem under GNU/Linux (Ubuntu 16.04) here, when starting either client or server I get this:

Traceback (most recent call last):
  File "<string>", line 11, in <module>
  File "/usr/local/lib/python2.7/dist-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/loader/pyi_importers.py", line 276, in load_module
  File "/home/hydrus/Desktop/hydrus/build/client/out00-PYZ.pyz/pkg_resources", line 69, in <module>
  File "/home/hydrus/Desktop/hydrus/build/client/out00-PYZ.pyz/pkg_resources.extern", line 60, in load_module
ImportError: The 'packaging.version' package is required; normally this is bundled with this package so if you get this warning, consult the packager of your distribution.

and no, my username is not hydrus, so I'd assume there's some wrong path in the python files; maybe hardcoded some path instead of using 'basedir' (or however it's called in python) ?


1af7a4 No.3399

>>3397

Same here on Gentoo ~amd64.


cfcfc2 No.3406

>>3397

>>3399

Works on Arch's AUR version, maybe take a look at the PKBUILD of the packages

https://aur.archlinux.org/packages/?O=0&K=hydrus

The pkdir is in the right under Pakage Actions.

___

I don't know how to use the suggested tags thingy, sounds pretty cool, I tried looking for help on http://hydrusnetwork.github.io/hydrus/help/index.html, but nothing. Please give me a hand.


1af7a4 No.3410

Found another little bug, client.pyw has missing import declarations on the outer exception handling section. "os" and the aliased "HC" aren't imported.

>>3406

The Arch AUR PKGBUILD is apparently doing a proper installation from source. Patches paths, compiles optimized bytecode, installs Hydrus and dependencies correctly.

That could certainly be helpful in writing native distro packages. But maybe it is not going to help with the error in the official Linux executable release. Seems to be a problem with pyinstaller, which I don't think the Arch AUR uses in any way.


eedd83 No.3416

File: 1470512632721.jpg (253.41 KB, 1280x1720, 32:43, 5b3bdb7068d801d31e46a36892….jpg)

>>3382

Thanks, I think the dupe tags are due to siblings. I'll made a note to filter them out.

>>3383

>>3384

>>3387

Thanks, I'll look at this. I'm fairly certain I know what it causing it, although 2-3 min lag seems too long. Let me know how this issue is in v218.

>>3394

Yeah, no sound yet. I'd love to add it in future. For now, hit 'open externally' on any videos with sound.

>>3397

>>3399

Please try this:

http://www.mediafire.com/download/s7ldwq6d8qvvm3l/Hydrus_Network_217_packaging.version_-_Linux_-_Executable.tar.gz

That 'hydrus' stuff is my linux dev machine's build path. I think PyInstaller, which I use to make the build, creates virtual import paths inside an archive (the out00-PYZ bit) inside the 'client'/'server' exe. The Windows build gives similar when it fails to import.

>>3406

As it is still prototype, I haven't written any help for it yet. Please check file->options->tags for now. You can set some favourites and hit a checkbox to make the new related tags control show on a regular manage tags panel.

>>3410

Thanks for the import catch.

Yeah, this is some PyInstaller stuff. Source runs no prob. I already explicitly included 'packages' in my OP release, but it looks like its submodules aren't detected or something.

When I added .version and some others on my dev machine, requests suddenly broke for reasons I don't completely understand. I've patched it enough to get it to boot on my machine again, but let me know what you get.


7a01af No.3419

File: 1470515738216.png (66.7 KB, 295x148, 295:148, indicators.png)

>>3416

>Yeah, no sound yet. I'd love to add it in future.

In the mean time, how about some indicator for files with audio streams? It's easy to check with ffmpeg/ffprobe.

Actually, how about an indicator for animations in general? Maybe something like Danbooru's? Obviously you'd have to tweak the layout a bit.


1af7a4 No.3420

>>3416

> http://www.mediafire.com/download/s7ldwq6d8qvvm3l/Hydrus_Network_217_packaging.version_-_Linux_-_Executable.tar.gz

Runs for me on Gentoo.

Busted the DB somehow, probably due to my own stupidity, so I guess I'll be running a little "stability test" with the huge client files reimport next.

> Thanks for the import catch.

No problem. Thanks for working out / around the pyinstaller issue!

> requests suddenly broke for reasons I don't completely understand

Packaging tools do weird / hard to diagnose stuff. But yea, it seems fixed here.


b8e873 No.3424

>>3416

>http://www.mediafire.com/download/s7ldwq6d8qvvm3l/Hydrus_Network_217_packaging.version_-_Linux_-_Executable.tar.gz

works for me on Ubuntu (16.04; amd64)


eedd83 No.3426

File: 1470587305046.jpg (382.09 KB, 1400x2108, 350:527, e66f6bc0ed8bd959d7ecbd9a55….jpg)

>>3419

I like this. I think I can add a video/animated icon, but the db isn't aware of audio yet. I'd like to add a 'has sound' metadata, but It'll need a network update, so I'll try to fold it into a bigger network rewrite job.

>>3420

>>3424

I'm glad we got this fixed. I've updated all my build scripts, so future versions should be ok, but obviously let me know if you have any more problems.

>>3420

Do you have any error messages in your client.log about your broken db? I'm up for trying to fix it, if you like.


8fe915 No.3428

>>3416

On the subject of suggested tags; I'm noticing it suggesting characters from other series or series themselves a lot. Ex. I just imported a bunch of Love Live images and the suggested tags does list characters from Love Live, but it also lists characters from K-On and series:K-On. Since those aren't really applicable they just wind up taking space on the suggested tags list (this is mostly for the 3rd search option, but the 2nd catches them some as well). Maybe filter out series tags if the image has a differing series, and remove characters from other series as well? I haven't checked to see how well it handles if you have only a series tag or only a character tag, but I'm assuming it'll suggest the proper series/characters.

Secondly, that bug with gif's not deleting properly while the preview window is rendering them is still present. Turns out the file remains as an orphan, which isn't a big deal but is tedious to search for when your db is sizeable. I guess at this point I should switch to deleting them from a trash search instead of using the "delete from trash now" option in inbox/archive. Not sure if the same problem affects webms/other video formats though.


1af7a4 No.3457

>>3426

>Do you have any error messages in your client.log about your broken db? I'm up for trying to fix it, if you like.

Many thanks for that offer!

However, I got rid of the logs when I ran diagnostics to be able to just see the new errors. Didn't think about this problem when I just rm'd the old log to get a new one.

Well, until now, it is working. Sure, Hydrus grows unresponsive and apparently slower at importing over time (especially with more images in the current import results - removing them periodically helps a bit). But it is still going.


eedd83 No.3472

File: 1471039799554.jpg (341.09 KB, 1277x1730, 1277:1730, 84e7c218de860b82772409254c….jpg)

>>3428

Thanks, yeah, I've noticed some of these unexpected namespaces coming in. The polling algorithm currently treats all the input tags with equal weight, so [long hair, blue eyes, character:asuka] will give plenty of tags that share [long hair, blue eyes] without requiring or emphasizing the asuka part. If one artist does a series of crossovers as well, this can make a spike in the db that the algorithm isn't clever enough to filter out.

Like you suggest, I'll add some namespace weighting options, so you can say 'count series namespace as worth 10 of an unnamespaced tag', or even mandate them.

I had another crack at the animation deleting in v218. Let me know if it is any better. Also, just in case you didn't notice, you can have the client regularly clear out your trash behind the scenes (avoiding this rendering problem) at file->options->files and trash. I designed it to work like an OS's recycle bin, ditching files quietly while still permitting easy undelete.



[Return][Go to top][Catalog][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / ] [ dir / cute / egy / fur / kind / kpop / miku / waifuist / wooo ]