/hydrus/ - Hydrus Network

>>4781

Hey boss,

Does this release fix the performance degradation that occurs as download import list expands?

Or is that unfixable due to how sqlite databases work.

Hey admin, I exchanged some emails with you in the past, topic "Tag Repository not processing" some time ago. 15 days ago I wrote you again with another set of doubs and you didnt reply. I bumped it 6 days ago and nothing.

Hi dev!

I currently find myself struggling with a subscription, that hits a lot of deleted files on e621. Every 5th encountered error the sync stops and I have to edit subs & force check on dialog ok.

Since these are deleted files, they're not going to come back and from my point of view it doesn't make a lot of sense to count them towards the error threshold.

Ideally I'd want the downloader to ignore these files towards the error count (some kind of configurable "deleted files" handling per booru) and just march on (and, ideally sure, tell me how many files had to be skipped because they were deleted).

Given that'd likely take some work on your end, is there a way to, as a workaround, increase the error count limit from currently 5 to something much higher on my end? I'm prepared to manually sift through the error list to figure out which ones are "real" errors to retry them later.

Thanks for all your work!

>>4781

Tried regenerating the autocomplete cache, gave me this

OperationalError
table specific_current_mappings_cache_1_6 already exists
Traceback (most recent call last):
  File "/opt/hydrus/include/HydrusDB.py", line 469, in _ProcessJob
    elif job_type in ( 'write' ): result = self._Write( action, *args, **kwargs )
  File "/opt/hydrus/include/ClientDB.py", line 9380, in _Write
    elif action == 'regenerate_ac_cache': result = self._RegenerateACCache( *args, **kwargs )
  File "/opt/hydrus/include/ClientDB.py", line 7026, in _RegenerateACCache
    self._CacheSpecificMappingsGenerate( file_service_id, tag_service_id )
  File "/opt/hydrus/include/ClientDB.py", line 2273, in _CacheSpecificMappingsGenerate
    self._c.execute( 'CREATE TABLE ' + cache_current_mappings_table_name + ' ( hash_id INTEGER, namespace_id INTEGER, tag_id INTEGER, PRIMARY KEY( hash_id, namespace_id, tag_id ) ) WITHOUT ROWID;' )
OperationalError: table specific_current_mappings_cache_1_6 already exists

Running on Linux, if that's relevant

>>4796

Exactly, deleted on e621's end.

To be honest I'm not even sure these files show up anymore in a search, probably not. I think for me specifically the problem is, that I created (and synced) a subscription a while ago, when those files still did exist. And now that the downloader is finally getting around to them, they are no longer there.

You can look at a takedown notice for a sample of deleted files:

https://e621.net/takedown/show/4937

As always, thanks for all your work.

>>4796

Welcome back devbro, hope you enjoyed your holiday.

I'm not sure whether the degradation is due to algorithm or because of software aging as I do have a tendency to run hydrus for very long periods of time without termination.

Anyway, heres my situation: I'm in the process of scraping all the major boorus using the in-built downloader. I like to import in batches of 100000 as theyre easier to keep track of in the long run.

I've used hydrus for 6 months now. And for a while now I've noticed that hydrus has sort of a "sweet spot" regarding interaction with the file import status menu where everything responds quickly and smoothly. This sweet spot is around n=~20000 objects. Past this number and performs noticeably degrades when interacting with the import status list as the system hangs.

This sweet spot kind of shoots down alternate theories as to why this happens, such as running out of system resources, as something like that would happen gradually and would be felt at all levels.

I'm also definitely not running out of RAM (8GB here).

Now, this is purely conjecture. But it feels to me like hydrus must refresh every other item in the list before acting on the selected object, leading me to believe some sort of O(n^2) type operation is going on. I dont have any hard numbers supporting this, but last week I had to redownload roughly 60000 files that timed out (my own fault) out of a 98000 object list. This operation took roughly 40 hours to complete. This is obviously a blue moon type of scenario, but even normal attrition rates after downloading a batch is usually in the 1k-2k range due to server refusing requests and timeouts, and even these takes roughly an hour or so to clear.

>>4801

Sorry, replace "redownload" with "refresh", as in "try again".

The linux tar.gz just links to the windows .exe, just so you know.

>>4804

I'm near-speechless you included it this quickly. I gave it a whirl over night and it's working fantastically.

Thank you so much! :D