/hydrus/ - Hydrus Network

>>4950

You bet I'll play around with the dupe search

> How inconvenient was it doing the regen in real time? Approximately how fast did it run?

Not at all inconvenient, my ~62k files took about 30 minutes total on a cheap SSD and did not impact performance enough to prevent use of the computer. Search tree rebalancing took about two minutes for 55k entries.

> At 'exact match' search distance, roughly how many potential duplicate pairs per thousand files does it find? What about 'very similar' and (if it isn't too slow) 'similar'?

total files: 62119

exact matches: 2317

very similar matches: 4221

similar will take a while

> How much of this heavy CPU/HDD work would you like to run in the background on the normal idle routines?

Personally, I'd rather it be run only on import and manually for the entire DB. If I'm just using the client for a quick lookup I likely won't be needing duplicate info at all, and when I'm doing the actual sorting, dupe cleaning and tag input I'll have time set aside to begin with, so calculating an additional x minutes for running an update to the duplicate search won't be much of a problem.

This way I can also just keep the client open without having to worry about it impacting the performance of other processes at inopportune times. But as long as there's a way to disable automatic processing on idle, I'll be happy

> Did anything go wrong?

Nothing big. The client froze two or three times during exact match discovery, but never for longer than a few seconds. The process indication for rebalancing metadata was stuck at zero/55k for about a minute, before jumping to 30k files done and staying there until finished. I'm guessing the underlying process won't allow for smaller progress updates?

>>4952

Update for similar: 5361

>>4950

>How inconvenient was it doing the regen in real time? Approximately how fast did it run?

The regen took 49 minutes to run for 35,155 total files. Disk utilization (for a 7200 RPM hard drive) maxed at 30% during that time. 30317 branches had to be rebalanced after this, which took 2 minutes.

>At 'exact match' search distance, roughly how many potential duplicate pairs per thousand files does it find? What about 'very similar' and (if it isn't too slow) 'similar'?

Exact match found 3671, took 3 minutes.

Very similar found 4496, took 6 minutes.

Similar found 4766, took 31 minutes.

>How much of this heavy CPU/HDD work would you like to run in the background on the normal idle routines?

CPU / Disk usage wasn't really noticable. I was able to use my computer just fine during all of it. So having it run in the foreground is fine with me.

>Did anything go wrong?

The number of potential duplicates it found, especially for exact match, seems very high. I can't imagine my database has more than a dozen exact duplicates as I have no subscriptions and manually review all the files I use hydrus to download. Also, the potential duplicates counter does not reset to 0 between searches.

Hey dev,

For users who keep their database outside the install dir, choosing "launch Hydrus" from the installer will open the default db instead of the preferred db after install. Just fyi.

Computer: AMD Phenom II 3 core 3ghz, 12gb ram

49567 files in db, on an old 5400rpm laptop HDD

>How inconvenient was it doing the regen in real time? Approximately how fast did it run?

About 2h10m

>At 'exact match' search distance, roughly how many potential duplicate pairs per thousand files does it find? What about 'very similar' and (if it isn't too slow) 'similar'?

exact: 3409, 6m

very similar: 6761, 13m

similar: 9211, 1h12m

>How much of this heavy CPU/HDD work would you like to run in the background on the normal idle routines?

It's maxing one core when searching for dupes, less when doing the regen. No noticeable slowdown in the rest of the system. I don't mind if it runs during idle.

>>4950

AMD 5350 running Gentoo.

>How inconvenient was it doing the regen in real time?

Outside of Hydrus there is no real problem, Linux schedules it fine.

Inside Hydrus it causes less GUI lockups or such than imports did (haven't tried these on the current release yet).

It's not really intrusive, just slow.

> Approximately how fast did it run?

~10 files/second for phash creation.

~1.5 branches/second for tree rebalance.

~4 files/second for exact search, but IDK if that's affected by me not creating a complete search tree.

> At 'exact match' search distance, roughly how many potential duplicate pairs per thousand files does it find?

About 30 per 1000 on the early search

Take my user experience, onee-chan!

The Machine:

OS - Windows 7

Processor - 3Ghz dual-core AMD

Memory - 8GB, ~4 free (about 3 used by system and misc processes, about 0.9 used by hydrus client when open and idle)

Hydrus running inside a multi-TB Veracrypt partition on a USB3.0 external drive, but AES-only encrypted so it performs pretty well.

The DB:

# of files - 99500

# in inbox / # in archive (probably doesn't matter) - 86000 / 13500

regenerate -> similar files search data:

I paused subscriptions and made sure all of my tabs were fully downloaded before running this, even though it should lock the db, just to be safe.

I have never compared files for similarity before so unless it did it in the background or on a closing update, this is the first time the similarity tree has been made for my db

82162 leaves found

CPU use jumped from basically nothing to around 55%, a little choppy but smoother than I expected honestly

Didn't use much RAM

GUI hang/(Not Responding) - GUI remained smooth switching tabs and opening menus, trying to open the settings panel froze it hard though, I force-closed it and restarted the process because I wanted to watch the progress

Seemed to do one or two per second at first

Smoothly accelerated to ~5 per second around 500 branches

Seemed to continue getting slightly faster as more were finished, as would stand to reason given a shrinking pool of comparisons

By 1500, generating ~20 per second

1400, ~100

10000, ~1000

50000, ~10000

Started regen - 12:42

Finished regen - 12:48

6 minutes for 100k files.

For everything below this, I was running some games and programs off and on during it, which seemed to slow it down a bit, so it's probably somewhat faster when left alone.

duplicates "preparation":

88114 eligible files, 0 up to date.

"Search tree is fast!"

It started faster than the above, ~8 per second

Seems to fluctuate somewhere between 5 and 10 actions per second

Hung up a few times near 2000 but didn't hang GUI

RAM usage increasing in nice gradual line, was 4 GB at idle, up to 5.5 GB by 2500

Got up to 6GB by 5000

CPU fluctuating between 40%-60% mostly

Occasionally the client gets stuck on a file for a few seconds and CPU drops to nothing, but it always starts again

Sometimes slows down to about 1 per second, not sure if it's because of the other programs running or just hydrus performance issues or larger images

Took 4 hours, 3 minutes

"All 88114 eligible files up to date! 77027 search branches to rebalance"

Rebalance (again, but through duplicate tab this time):

As the other poster said, doesn't seem to update the counter when I do it through this screen.

Took 8 minutes

Exact match search distance:

88117 files

13595 potential duplicates found

This seems about right, I've been downloading mostly monothematic content from various more-or-less interlinked Tumblr accounts, so cropped/resized/reformatted/etc dupes are to be expected…and at least one of the blogs I'm subscribed to got migrated and the images reformatted like you talked about. The one I noticed was one I had to redownload 13000+ files from, which is what most of these are, I'm sure.

Uses up to 6GB (extra 2GB) of memory and soft-maxes CPU

gets hung up on a file for a little while sometimes

Took 39 minutes

Very similar:

21702 potential duplicates found

Took 1 hour, 20 minutes

Haven't tried the other two yet, may try on 1000 or so files.

>How much of this heavy CPU/HDD work would you like to run in the background on the normal idle routines?

As much as possible, really, but my experience with idle stuff is that it tends to jam up the OS a bit, so maybe you could make it optional and give us a setting for the percent of system resources it could use, and default it to something slow but lightweight like 20%?

>Did anything go wrong?

Trying to open the options panel froze on of the processes so I restarted it. I figured it would. Switching between tabs and looking at menus doesn't seem to cause problems, though.

This just popped up, wasn't doing anything with dupes, just looking through tabs to prune dead ones and downloading from tumblr subscriptions:

PyAssertionError
C++ assertion "(itemid >= 0 && itemid < SHRT_MAX) || (itemid >= wxID_AUTO_LOWEST && itemid <= wxID_AUTO_HIGHEST)" failed at ..\..\src\common\menucmn.cpp(260) in wxMenuItemBase::wxMenuItemBase(): invalid itemid value
  File "include\ClientGUI.py", line 2740, in EventNotebookMenu
    menu.Append( ClientCaches.MENU_EVENT_ID_TO_ACTION_CACHE.GetTemporaryId( 'tab_menu_close_page' ), 'close page' )
  File "site-packages\wx-3.0-msw\wx\_core.py", line 12007, in Append

It was while I was closing a dead tab of 4chan /wg/. Doesn't seem to have actually had any "real" error, looks like just a noisy warning of some kind, but I thought I'd let you know.

>>4966

I take that back, it recurs whenever I try to open the right-click menu for that tab, so I can't close the tab. Can still just navigate around the tab though.

It happened on another dead tab too, this one from /s/. Not sure if a bug with this release or if my install just has an error.

Have you considered making a flatpak?

>>4966

>>4967

>>4968

Disregard these, found solution: don't leave Hydrus running for a day and a half and then do stuff without restarting it.

>>4962

I should add that this HDD is AES encrypted as well

Can this duplicate search method detect mirroring or cropping? I'm guessing no.

Due to an error on my part, hydrus couldn't find the database anymore, and the only solution I could think of without having to mess with configurations, etc. was to re-import the whole db, but it also pulled all the thumbnails that were generated since I first run the program

I'm already weeding them out by searching for duplicates, but it's rather slow and tedious

Will those new dupe search features help me making this process faster?

I don't really understand what they do, so before messing with them I'd rather ask

>>4981

Well my log is huge now: https://my.mixtape.moe/vqkoau.log

I question if these final entires may have something to do with it

2017/01/21 15:28:22: Profiling read load_into_disk_cache
2017/01/21 15:28:22: Stats

         4248 function calls in 0.247 seconds
2017/01/21 15:28:22: Profiling write maintain_similar_files_tree
2017/01/21 15:28:22: Stats

         95 function calls in 0.001 seconds
2017/01/21 15:28:22: Profiling write vacuum
2017/01/21 15:28:22: Stats

         42 function calls in 0.001 seconds
2017/01/21 15:28:22: Profiling write analyze
2017/01/21 15:28:22: Stats

         414 function calls in 0.013 seconds

There's other bits in between them, but calling all of those operations at the same time, during a gallery download is odd to say the least.

Are there any plans to add an archive indicator to 4chan download tabs? It would be useful to clear out clutter since many boards have threads that stay in the archive for a while before truly 404ing, and as far as I can tell they still show up in Hydrus just like normal threads do, don't go to 0 reps or anything. It would also be useful to have an indicator to know it might be time to search for the next iteration of "perennial threads".

>>4980

I think it was indicated in this thread that this would probably work eventually. It would still be a slow method.

How about you just delete all images with the dimensions of a thumbnail instead? Should be way faster.

Use system:dimensions in the search.

>>4990

I also have small images that are thumbnail-sized for various reasons, doing a search by dimesion would mean I can accidentally delete them too

Doing a dupe search is better as often the thumbnails have worse quality for the same dimensions

Anyway, it seems right now I still have to go with what I was doing now, but that's ok

>>5007

>>4981

>>4982

Nope, HDD. The files weren't particularly huge either, just an average scrape from Konachan and Yande. I doubt I could get to the print garbage option since the db locks up entirely when this happens, but I guess I can give it a shot. This is at least repeatable on my end, since for my last 3 download attempts I've gotten it every time.

>>5020

>>5007

Tried to print garbage and this is pretty much all it managed to do before I had to kill it.

2017/02/03 02:15:15: hydrus client started
2017/02/03 02:15:15: booting controller...
2017/02/03 02:15:15: booting db...
2017/02/03 02:15:15: preparing disk cache
2017/02/03 02:15:16: preparing db caches
2017/02/03 02:15:17: booting gui...
2017/02/03 02:21:01: C:\HYDRUS~1\PIL\Image.py:2244: DecompressionBombWarning: Image size (225829593 pixels) exceeds limit of 89478485 pixels, could be decompression bomb DOS attack.

The download was running fine for the most part, with the client using between ~260-~300MB of memory; it would occasionally inflate to 400, 600, etc. for a second or two. I managed to catch it inflate to 1.5GB but it also went away after a second, so even though I hit print garbage I don't think it caught anything there.

2017/02/03 02:00:01: Printing garbage to log
2017/02/03 02:00:01: gc types:
2017/02/03 02:00:01: (<type 'method_descriptor'>, 2263)
2017/02/03 02:00:01: (<type 'member_descriptor'>, 522)
2017/02/03 02:00:01: (<class 'pyparsing.And'>, 117)
2017/02/03 02:00:01: (<type 'getset_descriptor'>, 2851)
2017/02/03 02:00:01: (<type 'wrapper_descriptor'>, 3414)
2017/02/03 02:00:01: (<type 'instance'>, 1728)
2017/02/03 02:00:01: (<type 'set'>, 2695)
2017/02/03 02:00:01: (<type 'property'>, 2452)
2017/02/03 02:00:01: (<type 'StgDict'>, 361)
2017/02/03 02:00:01: (<type '_ctypes.CField'>, 266)
2017/02/03 02:00:01: (<type 'classobj'>, 642)
2017/02/03 02:00:01: (<type 'module'>, 1402)
2017/02/03 02:00:01: (<type 'type'>, 3342)
2017/02/03 02:00:01: (<type 'builtin_function_or_method'>, 11574)
2017/02/03 02:00:01: (<class 'comtypes.dispid'>, 118)
2017/02/03 02:00:01: (<type 'instancemethod'>, 1126)
2017/02/03 02:00:01: (<class 'include.ClientRatings.RatingsManager'>, 111)
2017/02/03 02:00:01: (<type 'cell'>, 1651)
2017/02/03 02:00:01: (<class 'include.ClientMedia.LocationsManager'>, 111)
2017/02/03 02:00:01: (<class 'zope.interface.interface.InterfaceClass'>, 117)
2017/02/03 02:00:01: (<type 'weakref'>, 6163)
2017/02/03 02:00:01: (<type 'operator.itemgetter'>, 245)
2017/02/03 02:00:01: (<class 'zope.interface.interface.Method'>, 312)
2017/02/03 02:00:01: (<type '_ctypes.PyCFuncPtrType'>, 139)
2017/02/03 02:00:01: (<type 'collections.defaultdict'>, 404)
2017/02/03 02:00:01: (<type '_cffi_backend.CTypeDescr'>, 306)
2017/02/03 02:00:01: (<class 'zope.interface.declarations.Implements'>, 129)
2017/02/03 02:00:01: (<type 'list'>, 6695)
2017/02/03 02:00:01: (<class 'include.ClientMedia.MediaResult'>, 111)
2017/02/03 02:00:01: (<class 'bs4.element.NavigableString'>, 779)
2017/02/03 02:00:01: (<type 'tuple'>, 12633)
2017/02/03 02:00:01: (<class 'wx._core.PyEventBinder'>, 271)
2017/02/03 02:00:01: (<class 'zope.interface.declarations.ClassProvides'>, 122)
2017/02/03 02:00:01: (<class 'include.ClientMedia.TagsManager'>, 111)
2017/02/03 02:00:01: (<type 'function'>, 27928)
2017/02/03 02:00:01: (<type 'classmethod'>, 240)
2017/02/03 02:00:01: (<class '_weakrefset.WeakSet'>, 316)
2017/02/03 02:00:01: (<type 'staticmethod'>, 606)
2017/02/03 02:00:01: (<class 'wx._core.MenuItem'>, 141)
2017/02/03 02:00:01: (<class 'bs4.element.Tag'>, 532)
2017/02/03 02:00:01: (<type 'frame'>, 119)
2017/02/03 02:00:01: (<type 'dict'>, 14138)
2017/02/03 02:00:01: (<class 'include.ClientGUIMedia.ThumbnailMediaSingleton'>, 111)
2017/02/03 02:00:01: gc classes:
2017/02/03 02:00:01: ('ValueSizeConstraint', 105)
2017/02/03 02:00:01: ('__new__', 409)
2017/02/03 02:00:01: ('ConstraintsIntersection', 112)
2017/02/03 02:00:01: ('WeakKeyDictionary', 373)
2017/02/03 02:00:01: ('NamedType', 141)
2017/02/03 02:00:01: ('NamedTypes', 118)
2017/02/03 02:00:01: ('from_param', 313)
2017/02/03 02:00:01: uncollectable garbage: []

After that I closed and restarted the client so it would be fresh and about 5 minutes into the download it spiked from 250-ish MB to 3.5GB. The client locked up but before I could force close, it managed to get itself under control somehow; the memory dropped to 180MB and it was laggy but that's when I managed to get the print garbage from up top. Unfortunately it started rapidly inflating memory usage (400MB->900MB->1.5GB) so I killed it and just kind of hoped it managed to print garbage.

And yes, I did update to this week's release. Haven't had any other problems except this gallery download issue.

>>5020

>>5082

Thank you. I will have another think about this.

>>5104

I think I've found the image that's causing the issue, atleast on konachan: https://konachan.com/post/show/161317/bicolored_eyes-black_hair-bow-cosplay-gokou_ruri-l

The full png doesn't load properly in chrome, and even after downloading, mspaint doesn't want to open it either. And sure enough, the gallery download commences smoothly if I skip that file!

Yande.re doesn't have it though, so their must be some other large file tripping up the download there.