[ / / / / / / / / / ] [ dir / cute / egy / fur / kind / kpop / miku / waifuist / wooo ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog

Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 12 MB.
Max image dimensions are 10000 x 10000.
You may upload 5 per post.

New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Current to-do list has: 714 items

Current big job: finishing off duplicate search/filtering workflow

YouTube embed. Click thumbnail to play.

3b41d8 No.5340

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v247/Hydrus.Network.247.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v247/Hydrus.Network.247.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v247/Hydrus.Network.247.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v247/Hydrus.Network.247.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v247/Hydrus.Network.247.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v247.tar.gz

I had a good week. I fixed a bunch of bugs and made an important visual improvement to image display.

I got adminside petition processing working again this week, so I've been catching up on all the petitions made since v245–there were about 4,000!

flickerless scrolling

From now on, when you scroll through images in the media viewer, the 'container' window that displays the image will be recycled. This means the old one- or two-frame flicker of grey on scrolling (as the new window is generated) is eliminated. You can now scroll through images a lot faster and neater, and you can keep going real fast if you have a powerful computer that can render new ones quick enough. Animations will also recycle, but you'll still get a frame or more of white as the renderer initialises.

I've been meaning to do this for a while, but I bumped it up in preparation for the forthcoming duplicate filter, where you will be flicking back and forth between pairs of images to find differences. Having that be a single-frame transition will be a dealbreaker.

full list

- fixed a problem with deleting more than 256 files at once

- furthermore, deleting from thumbnail view or after a filter will split delete jobs into chunks of 64 files at a time to reduce gui hang from deleting many tag-heavy files

- rewrote canvas media container code to recycle containers, embed windows, animation bars, static image windows, and animation windows. scrolling through all kinds of media is less flickery (less 'grey box' window initialise flicker) and scrolling through static images should be completely flickerless! William Gibson slideshow speed works again!

- this flickerless static image transition will be particularly useful in the forthcoming duplicate image filter!

- got adminside petition processing working again

- petition counts and fetching is now split by content_type and status

- the approve/deny colour hint is more obvious on the petition panel

- petitions now process off the main gui thread and throw up a popup message

- added 'check all' and 'check none' buttons to petition panel

- several serverside petition processing fixes

- generalised and improved dynamic menu check item initialisation and inversion support

- moved the 'get tags even if file already in db' option into a cog button on regular downloader pages

- added a default menu option for 'get tags even if file already in db' to the same cog button

- added this cog button to the edit subscription panel as well

- fixed exporting tags to Hydrus Tag Archives

- fixed exporting 'all known tags' to HTAs

- cleaned some HTA and related code

- fixed namespace based tag censorship

- fixed autocomplete not filtering out current/pending counts if they are set to 'excluded'

- fixed generation of non-expiring new accounts

- fixed some v245->v246 tag improvement code that was replacing invalid tags with the incorrect namespace

- patched a problem with '-:' dirty tag in the v245->v246 update code–I'm not sure what it was doing, but it catches the unusual problem and puts it in the 'invalid tags' category, so let me know if you get trouble with this in future

- wrote unit tests for bytesdictionary and shortcuts serialisable objects

- harmonised and improved how separators are appended to menus

- cleaned up some client db index creation

- cleaned up some client tuple stripping

- misc pylint warnings cleanup

- misc fixes

next week

Fixing more bugs, doing more v245 catchup, and starting a proper duplicates filter.

I looked at allowing custom tag namespace/subtag connectors this week (for having the client render tags like 'series - evangelion' or however you like), and while it is doable, it needs a bit more time than I was expecting. I'd like to focus and get that done.

f151db No.5342

Arigatou senpai


c34fa8 No.5343

>>5340

>William Gibson slideshow speed works again!

Something I've been meaning to ask, why is it named like that?


bb2cae No.5344

Awesome update. That flashing in the media viewer has been bugging me for some time.


4ac144 No.5345

This error happens when updating.

Traceback (most recent call last):
  File "include\ClientController.py", line 1099, in THREADBootEverything
    self.InitModel()
  File "include\ClientController.py", line 520, in InitModel
    HydrusController.HydrusController.InitModel( self )
  File "include\HydrusController.py", line 205, in InitModel
    self._db = self._InitDB()
  File "include\ClientController.py", line 59, in _InitDB
    return ClientDB.DB( self, self._db_dir, 'client', no_wal = self._no_wal )
  File "include\ClientDB.py", line 139, in __init__
    HydrusDB.HydrusDB.__init__( self, controller, db_dir, db_name, no_wal = no_wal )
  File "include\HydrusDB.py", line 226, in __init__
    raise e
Exception: Updating the client db to version 246 caused this error:
Traceback (most recent call last):
  File "include\HydrusDB.py", line 205, in __init__
    self._UpdateDB( version )
  File "include\ClientDB.py", line 8247, in _UpdateDB
    self._c.execute( 'UPDATE subtags SET subtag = ? WHERE subtag_id = ?;', ( clean_subtag, invalid_subtag_id ) )
IntegrityError: UNIQUE constraint failed: subtags.subtag


1667de No.5348

>>5340

I wanted to ask about flickerless scrolling, but you actually implemented it before I could mention it, this is a tweak that bumps this software up from novelty to actual proper levels of useability. Thanks a lot for this update!


b633c8 No.5349

Is nice. Trying it this couple days, like what I see. Has imported most of pics.

Two question, though. First, I'm on Arch Linux. Second, I can't seem to find Remote tab on Manage Service. Third, is there a way to filter low-res pic?


49eca3 No.5350

>>5340

Nice, I couldn't use Hydrus at all last week because the tag cleaning failed, so thanks for fixing that.


f165a4 No.5352

>>5345

I'm also getting the same error.

I'm updating from 245a to 247.


5be7de No.5353

>>5349

For the third question, try system:size and system:dimensions. Also check out system:mime. Don't forget system:limit to avoid freezing your client with lots of files.


e3a42d No.5358

File: 994824c08b90a29⋯.png (9.7 KB, 317x317, 1:1, 994824c08b90a29e41a8c591e8….png)

>>5343

I can't remember if I came up with it or a user did. It is just a joke–in that Gibsonesque cyberpunk movies often have very fast slideshows of weird internet shit, and now, thirty years later, you can actually do it yourself. If you load up a search of 25-50 images (enough to fit into the image cache, although if you like this sort of thing, you can bump it up in the options) of some meme and then load up Gibson mode slideshow and let it cycle once to prep the cache, you should get a pretty smooth and fast slideshow.

>>5345

>>5352

Thank you for this report. Another user mentioned it as well. I believe I have it fixed for v248, but if either of you happen to use the Windows extract release, please try this:

http://www.mediafire.com/file/ma1w5czd6k8qwq3/Hydrus_Network_247_update_fix_test_-_Windows_-_Extract_only.zip

>>5349

A user puts an Arch release together, as my link on these release posts is put together on Ubuntu and apparently does not run well on Arch. I don't know anything about how Arch package stuff works, but I think this is it:

https://aur.archlinux.org/packages/hydrus/

For manage services, I recently completely changed how the guts of the dialog works and haven't caught the help up yet. The local/remote dichotomy is currently replaced with just a single list that lumps everything together. I'm not sure if I like it all mixed up, so I might split it up again. If you want to add my tag repository, the ui looks a little different but uses the same info. Just click the 'add' button on manage tags to get started. I'll be catching the help up in the next few weeks, but please ask again if you can't figure anything out.

And >>5353 is correct–check out the system: predicates for all those sorts of 'meta' searches. They can do a lot.


c3132b No.5359

>>5345

>>5352

>>5358

I messed up this release a little, but it seems to work. If you try it, please try booting it twice. (You'll get an error about 'no transaction active' on the first go, but the update seems to go through ok.) Here's an installer version as well:

http://www.mediafire.com/file/ta7fexbe867qke9/Hydrus_Network_247_update_fix_test_-_Windows_-_Installer.exe


8ad7dd No.5360

>>5358

Yeah, I got it as AUR package. Basically, it is source + details about what dependency should be used for it.

Anyway, the suggestion anon made works reasonably well (thanks!). I want to know though, is it possible to not syncing the tags repo after that? Since downloading that much tags will kill my data quota in a sec.


f06018 No.5361

Oh, almost forgot.

Any plan for implementing parent/child pictures? Or album?


d09bf5 No.5363

File: 719308521f1cf06⋯.png (23.31 KB, 590x625, 118:125, service.png)

>>5360

>>5361

The total download to catch up from nothing to today is about 750MB in 3500 files or so. Every day adds a few MB. In general, that's all the bandwidth syncing with the PTR will need. Also, since this new rewrite, you can now set some clever bandwidth restrictions so it doesn't all download in one go.

I think that dialog will give you a default of 50MB a day, but perhaps you would prefer 8MB a day or 30MB a week, and you should catch up over time. If you can't afford even that much bandwidth, you may wish to stay away from the PTR for now.

The interface is still a little ugly (I'm still rewriting some of the complicated parts), but my screenshot is what you should now be looking at to set up the PTR. Click the 'add' button on the manage services dialog and select 'hydrus tag repository'.

For parent/child, in my current work on better duplicate image management, I have set up the internal database structure to support parent/child images, but I won't add it in this job. I would like to support many kinds of 'image alternates' and other image-to-image connections in a future update, but it will be another big job like this dupe stuff.

I also want to support better 'chapter'/album support, likely by better supporting current standards like cbz/cbr. Although you are free to play around, and I'd love feedback if you do, hydrus is not very good at supporting paged content like comics yet.


9c5c9c No.5365

While you're at it could you add a way for the thread watcher feature to check an unlimited amount of times or until the thread 404s? Being limited to 100 minutes with the lowest interval is really annoying (and also impractical if you consider downloading for example r34 threads from /b/ which have a tendency of 404ing very quickly, causing you to potentially miss images).

Talking about limits, the thread update interval is also unreasonable, at least for 4chan. The official API expects you to update the thread no more often than once every 5 seconds, and the forced second delay between GET requests to the used CDN for the image files also doesn't make sense if you consider modern HTTP pipelining features this might actually increase the load on the server if the connection isn't kept alive in between requests. Besides that. the official embedded JavaScript featurs on 4chan have a way to expand all images at once: you can imagine that there's absolutely no delay between download requests that way – in fact the downloads are probably even happening in a mulit-threaded fashion.

Wanting to keep the scraping within reasonable limits is nice and all but overdoing it and artificially limiting the usage is just annoying, especially if it's not even needed.

Thanks as always for your work.


a29a94 No.5368

File: 896882f6e7e312e⋯.png (43.91 KB, 571x451, 571:451, ss+(2017-03-20+at+05.26.37….png)

Hiya Dev, I get pic related whenever I try to update to v244 and beyond. How should I proceed?

Also, thanks for the great job you've been doing.


16e636 No.5370

>>5363

Ah, it is indeed for paged content! But my aim is something small, like two or four comic strips that's generally too small to be put as CBZ/R.


4ace45 No.5373

File: 9c6691878837125⋯.jpg (216.65 KB, 1600x1200, 4:3, 9c66918788371258a0026888f9….jpg)

>>5365

Thank you for these thoughts. I hope I can improve these issues in the nearish future.

The downloader overhaul, which I will be starting after this current dupe stuff, will also overhaul the thread watcher. Users will be able to create watchers for any sort of page and have greater control over what causes a 'stop checking' event and also run multiple threads with independant timers on the same watcher page. Allowing longer overall check durations (such as 'keep checking every ten minutes for 24 hours') can absolutety be part of that.

But for v248, I've bumped the check limit up from 100 to 65536 and reduced the min time delta from 30 mins down to 5. I do not want to reduce the time more than that until the downloader overhaul is done, as hydrus is still ugly in how it does some http stuff and cannot take advantages of some optimisations these generous APIs expect.

In the same way, the current bandwidth limit of some seconds is a quick bodge I wrote. I would like to apply the new bandwidth controls I created in the recent service overhaul to the network engine, allowing for finer control of bandwidth on a per-domain basis. The new downloader engine will be clever enough to plug into this as well.

Please let me know how this stuff works for you as I roll it out.

>>5368

The v244 update step temporarily needs additional disk space on your system partition (or wherever your temp folder is located, if you are aware you have moved it–usually it is under C:\users\you\appdata\local\temp, so we are talking about C:). This space is approximately 1.2 times the size of your client.mappings.db file in install_dir/db (although a bit extra won't hurt. It may need twice that much if your hydrus is installed on C: as well. For users that sync with the PTR, this is probably about 4GB. If you don't have that much free on C, please free up a little space and try again. If you do have that much space (and on your hydrus drive), let me know as the problem may be more complicated.

Some other users also encountered this problem, so I added additional checks to warn you about this, but they only came in around v246 or so. If you are updating in single steps, and this error comes from a v244 install, then you aren't being warned correctly because they aren't in yet, but if you are updating to v246 or v247 here and never got the warning, please let me know your hard drive space and db size specifically so I can try to figure out what went wrong with my tests.

>>5370

Small stuff is manageable, but make sure you give the little collections a unique title: tag or something so you can easily find, sort,, and collect them. I mainly warn you to forestall you getting your hopes up and painstakingly setting up regexes to parse 17,000 pages of stuff and then finding the endpoint ain't that great yet. Main problem at the moment is that if your pages are off by one or something, there's no easy way to bump them all up–you have to edit them all manually!


cf5f5d No.5374

File: 8778b185ecff50b⋯.png (60.81 KB, 698x416, 349:208, timedeltabutton.png)

>>5365

>>5373

Wait a sec–I don't know what I'm talking about and I think I read your post wrong as well. Did you mean 100 minutes or 100 checks?

The minimum thread check time was already 30 seconds–I read my own code wrong. I'll leave it at that value but still bump the number of checks up to 65536, which at 30 secs is something like three weeks. I hope this will do until I can improve the underlying engine.

Just in case you also can't change the check interval, here's what I see–do you get anything different to my image?


d2cba6 No.5375

Good to know!

Two things. First, Hydrus client keep crashing on me when I'm syncing the tags repo - I have… Client 247, network 18. As for other components:

FFMPEG: 3.2.4

OpenCV: 3.2.0

openssl: OpenSSL 1.0.2k 26 Jan 2017

PIL: 1.1.7

Pillow: 4.0.0

python: 2.7.13

sqlite: 3.17.0

wx: 3.0.2.0 gtk2 (classic)

It doesn't crash instantly, instead it'll sync some amount of tags - between 10-250~ish - before crashing, which usually take five minutes or less. I can just start it again, but the problem persist.

Second… how do I start Hydrus from terminal?


9d2055 No.5376

File: 5cf8e36025b6102⋯.jpg (587.68 KB, 1188x788, 297:197, 5cf8e36025b6102589103ce553….jpg)

>>5375

I'm not sure why this is happening. I am sorry for the inconvenience.

Are you the Anon from earlier in the thread who is running the AUR package from source?

All those library versions you report look about the same as my version. The only difference is your Pillow, which is newer than mine, but that image library isn't used in tag processing. Hydrus crashes–particularly when run from source–are usually down to quirks in some OSs' window managers.

My first suspicion is that your window manager doesn't like something about the popup window that appears when a tag repository processes, but I am not sure where your processing is occurring. Is the processing you are having trouble with happening in the main client (for instance triggered from the 'process now' button in review services), or is it during shutdown? If the crash is happening in one, does it happen in the other?

Does going help->debug->make some popups work ok?

How about if you temporarity give five or six random local tags to several hundred files in one step? This would simulate–clientside–what the tag processing is generally doing, and would help us isolate if this is a db or gui issue.

And what about database->maintain->vacuum/analyze? Choose the 'soft' option.

By the way, you can go services->pause->repositories sync to temporarily turn off repo sync.

To run the client from a terminal, navigate to the base install directory and type:

./client.py

Everything that is written to the terminal is also written to a client.log in your install_dir/db directory. Since this is a hard crash, there likely isn't a useful error message there, but perhaps there are some window manager warnings?


974114 No.5378

>>5373

>>5368 speaking; I noticed that last line as soon as I posted that. This is a different error, not the one I was getting when I tried updating from v243 to v244 a few weeks ago. Also, space definitely wasn't an issue at that time either (client.mappings is under 4gb).

By your estimate, I still have enough free space on that drive (24gb). Regardless, I'll free up some more room just in case and see if I can get back to the original error.


2432ab No.5379

>>5376

I am, yes. Making popup works ok.

Oh, I found a workaround: I just need to keep Hydrus window as active all the time (no switching), and it won't crash?

It still crash during processing, though. But the above workaround works.

Haven't done the giving random local tags, will do that later.

Have done maintain-> vacuum/analyze, though is done independent of this. Vacuum is hard, analyze is hard then soft.


9d9f42 No.5380

>>5379

> I just need to keep Hydrus window as active all the time (no switching)

Is Hydrus by chance minimized when the crash happens?


830864 No.5381

>>5380

Not necessarily. Sometimes it's just behind other window, like if I opened both hydrus and Chrome together.

And no, it's not because Chrome loves eating RAM. Have tried closed it, just running IRC client and hydrus. Still crashed anyway.


38307d No.5384

>>5374

Hey, I just meant that with previous versions of Hydrus you could only check for 100 minutes at max with the fastest interval settings, since you couldn't set the repetitions to above 100. (Nevermind, I just checked, 30 seconds is the lowest, but 3000 seconds are over way too quick, the thread will definitely still be alive by then).

The changes displayed in your image already remove much of the tediousness, thank you!



[Return][Go to top][Catalog][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / ] [ dir / cute / egy / fur / kind / kpop / miku / waifuist / wooo ]