/hydrus/ - Hydrus Network

$250 added to the Patreon for this month. Thanks for fitting it in this week!

>>9288

Found one flaw in my request and it's my bad for not thinking about my "actual" problem when importing new tags from boorus. Specifically, the unnamed space is trash on most boorus - while the character and series namespaces are usually useful. I still want to include tags from the unnamed space since I have a pretty significant overlap (several hundred tags).

Could the option be expanded so that I can choose to only apply it to tags without a namespace? If that would be too difficult, as-is still works better than before since I already have several thousand characters/series as tags. I can always add the tag once manually before an import to make sure they get imported for every image - since most of my imports tend to me of a certain character or series.

>>9288

Based $250 Anon.

>>9287

>I have added a Deviant Art parser. It now fetches the embedded image if the artist has disabled high-res download

Was going to ask or make a suggestion in another thread but since you brought this up now. In the past versions, occasionally I would come across certain artist on DA that have their images with these large resolution sizes that would slow hydrus down to a crawl for me just by selecting the thumbnail and/or viewing them. Before version 312 I was going to ask if there was a way to select smaller res sizes when downloading from DA if the largest reached a certain resolution size.

I don't know if it's related to this release or not but I was in the middle of migrating my storage(bought a new HDD and setup a new path to move files there) and now I keep getting "database disk image is malformed' every time a subscription starts. Any way I can fix this? Thanks

DBException
DatabaseError: database disk image is malformed
Traceback (most recent call last):
  File "include\HydrusThreading.py", line 197, in run
    self._callable( self._controller )
  File "include\ClientDaemons.py", line 275, in DAEMONSaveDirtyObjects
    controller.SaveDirtyObjects()
  File "include\ClientController.py", line 1114, in SaveDirtyObjects
    self.WriteSynchronous( 'serialisable', self.network_engine.bandwidth_manager )
  File "include\HydrusController.py", line 640, in WriteSynchronous
    return self._Write( action, HC.LOW_PRIORITY, True, *args, **kwargs )
  File "include\HydrusController.py", line 201, in _Write
    result = self.db.Write( action, priority, synchronous, *args, **kwargs )
  File "include\HydrusDB.py", line 908, in Write
    if synchronous: return job.GetResult()
  File "include\HydrusData.py", line 1710, in GetResult
    raise e
DBException: DatabaseError: database disk image is malformed
Database Traceback (most recent call last):
  File "include\HydrusDB.py", line 537, in _ProcessJob
    result = self._Write( action, *args, **kwargs )
  File "include\ClientDB.py", line 11063, in _Write
    elif action == 'serialisable': result = self._SetJSONDump( *args, **kwargs )
  File "include\ClientDB.py", line 9027, in _SetJSONDump
    self._c.execute( 'DELETE FROM json_dumps WHERE dump_type = ?;', ( dump_type, ) )
DatabaseError: database disk image is malformed


Database Traceback (most recent call last):
  File "include\HydrusDB.py", line 537, in _ProcessJob
    result = self._Write( action, *args, **kwargs )
  File "include\ClientDB.py", line 11063, in _Write
    elif action == 'serialisable': result = self._SetJSONDump( *args, **kwargs )
  File "include\ClientDB.py", line 9027, in _SetJSONDump
    self._c.execute( 'DELETE FROM json_dumps WHERE dump_type = ?;', ( dump_type, ) )
DatabaseError: database disk image is malformed

>>9294

I should note I read 'help my db is broke.txt', and the errors originate from client.db & client.master.db. It specifically gives 'database disk image malformed' when running a PRAGMA integrity_check on client.master.db.

I cloned both of them, and then ran another integrity_check, and they came back ok. I don't get 'database disk image malformed' anymore when running Hydrus, but now I can't import any new files(they fail), and subscriptions are still error'ing out.

I have a full .db backup from 2 weeks ago. My main concern is, if I use the backup, what will happen to all the files I added within that time span? Will they just remain floating in client_files unused since the .db doesn't recognize them? I believe I added something like 200k files within that time.

Hello hdev, Back to report about how fast my massive fuck off session loaded this time.

I watched it load the first page in chunks of 400-1500 a second, there were no hangs, and the moment it was loaded it was able to bring up the images, jumping around to other tabs, im not sure, it locked up again while its loading, but it only did it once finished loading a second tab, all in all took about 5 minutes, im sure if I restarted it now it would load slower.

I don't understand why sometimes it loads fast and other times its slow as hell, especially when everything is loading off of an nvme outside of full size files.

>>9296

After a bunch of trial & error, I decided to just use my backups from 2 weeks ago. It contains a backup of client, caches, master, mappings and cache. I extracted a clean Hydrus (new version) and moved the databases into the db folder.

After the update finished, it gave me a "database disk image is malformed'. Now, I know for a fact that can't be possible considering I made those backups while my DB was working perfectly fine.

I don't know why this happening

https://www.politesi.polimi.it/bitstream/10589/132721/1/2017_04_Chen.pdf

>>9301

update: this only seems to happen on multi-page submissions.

single file submissions actually grab all the tags

I LOVE YOU

You're doing gods work hydrus_dev. The god of smut that is. Thanks.

>>9306

>Even on a super SSD, the difference between that and ram is something like 50µs vs 10ns, I think, so it still matters for 'loading lots of small things from random places' requests.

I leave my computer on for months at a time, really its only hard crashes or power outs that took it down, and since I got the battery around november, there has been 1 power out long enough the system went down where the town transformer blew.

every other time it on an upgrade or the program crashing, however I haven't had the program crash in awhile, since I moved over to saving large amounts of tabs to grab all at once, and more recently when you added the multi watcher. sometimes when I close and immediately reopen, everything goes fast, other times, everything chugs, sometimes when the program would crash, and this was on a 300k file session, it would load slow, but then the next time, it loads faster than ever before.

if I have the session below 60k images, regardless of how it closes, its open almost instantly opened, but with my 300 and this current 600k image session, it loads very slow, even when it loads fast, its proportionally different then when its loading smaller file sizes, it leads me to think it's something in program bottlenecking rather then storage, unless its trying to load storage all at once and hitting storage at much as possible. if that's the case, it may be replicable with low numbers of images but over many tabs, currently I have 78 tabs open thanks to multi watcher being able to trim the crap down.

this is just my thought based on seeing every tab trying to load at the same time, if one tab tried to load sequentially rather then every tab trying at once it may load differently.

with that being said, would it be possible to to have a mode in the multi watcher to not show the date and time but instead have relative time always? At least for me its easier to see when I loaded batches of threads with 5 days 16 hours rather than 2018/06/23 00:34:01

Is there an easy way to mass replace tags or add namespace to them? I see that you added meta namespace with the latest update, but previous images I've fetched from danbooru were unnamespaced, unlike the current ones. It's not that much of a big deal but it bothers my autism.

>>9310

I'll check it tomorrow and will ask if there's anything I couldn't figure out, thanks hydrus dev. You're the best.

I tried to download on PIXIV, this profile:16488

-From pixivutil : 518 files + the profile avatar

-From Hydrus (subscription) : 354 files

-From Hydrus (gallery downloader) : 399 files (354) total files contained in the downloader page

Coud someone explain what I could be setting up wrong?

@hydrus dev specifically, would it be doable to configure the page numbering between 1,2,3,… and 0,1,2,… in the subscription windows?

>>9315

Indeed, I had missed the linked post.

Thanks for taking the time to consider my request.

And no, it's need for every pixiv link since pixiv's work are numbered from 0.

Being able to fetch ugoira's tags is a really appreciated, I wonder if it'll fetch even response work depending of how pixiv set their site.

>>9306

>TBH I recommend just downloading the big files and giving them a 'check back in three years' tag or something and seeing if your next computer handles them better. It depends how crazy the artist is–if it is 20MB pngs, I say keep them and deal with them later, but if it is 120MB pngs, you may want to follow that artist by hand anyway as there may be no easily automatable solution to their nonsense.

The image's file size dosen't seem to be a problem for me as most are less than 5MB, its just these 7,000~10,000x7,000~10,000 resolution sizes that's slowing Hydrus down a bit for me. DA is the only site I noticed that has these artist going crazy with resolution sizes for their images but its not that big of an issue thought, just annoying.

>>9315

This work perfectly thus far, in the few profie I tested.

>>9317

The reason da has sizes like that, at least on artists you like/thing its worth downloading is because they also sell prints of those images.