/hydrus/ - Hydrus Network

>>5110

Thank you for this report. I can't repeat it here, but I expect something got switched around with all the changes this week. I will look into this this week.

>>5111

I am not sure about this. SQLite can typically go up to zillions of TB, and I don't overrule that anywhere, so I don't think the db is full. Sometimes it needs to create temporary db files in your temporary directory to do big transactions, and that has sometimes hit Linux users who have 500MB limits or whatever. I remember one user who had a ramdisk tempdir who had that problem. If your system partition is full up or your tempdir has a different kind of limit, I think you'll need about 3GB of spare space. If you are interested, you should be able to watch the file(s) being populated in your tempdir as the bigger parts of the update occur.

Please let me know how you get on. If this is the problem, I will improve the free space check for this update step.

>>5116

I'm glad this got sorted out. Unfortunately, I haven't figured out a way to tell sqlite to use a different location for its temporary tables. This might not ever affect you again, but as this update had to deal with a single gigantic table, there wasn't a good way to break it up.

Please let me know if you get hit by this again in future.

>>5120

Thanks–I'll have a look to see why that is happening.

>>5117

>>5118

>>5122

>>5123

>>5124

Do you have a backup? Is that backup your 'previous version' of hydrus? If so, does that older db also have problems with an integrity check, or is it clean?

If you haven't checked 'help my db is broke.txt' in the db dir, give it a quick read. It describes another dump/restore recovery method that may be different to what you have tried before.

This sort of thing is almost always due to a hard drive problem, and while it may just be a blip from a recent hard power cut, it could suggest something more serious. It may be this broken record has been around for ages, but you've never stumbled over it as it is in a gigantic mappings table, and now my update code here is iterating over absolutely everything, it can't avoid it. It could also be this error is getting thrown mistakenly.

Please run chkdsk or your OS's equivalent as the help document describes and make sure everything is backed up.

If you don't have a clean backup and this can't be fixed by a simple clone, I can walk you through some sql to try to fix this manually. I am confident we can recover your db, but it might take a bit of back and forth.

Hi dev, may I request for the thread watcher a persistent icon or something that indicates when a thread has been archived or has 404'd?

Thank you

Hello hydrus,

I want to fully migrate into hydrus but I also wish to retain a lot of the metadata in the tags, the mass importer allows me to pull in filenames and paths but I was wondering how complicated it would be to add support for pulling in the rest of the file stat, like creation time, mod time, filesize, etc. and possibly formatting it however I want with something like regex groups or any other special formatting sytanx (whatever is easiest to implement). For example I could take the date and translate it from whatever the OS/API provides (maybe "mm/dd/yy") into a tag "file-creation-date:yyyy/mm/dd"

I personally only care about creation time but maybe others would care about importing other meta data that is exposed by the OS/Python API.

I'm also curious if these can be applied to the auto importer, I want to setup a directory to save things to so they get imported into hydrus then deleted from the original file system, and I'd like the same metadata to be preserved, as well as maybe a special dynamic tag like "import date" i.e.

filename:whatever.ext

path:/usr/home/coolman/Pictures/autistic things/

file-creation-date:yyyy-mm-dd-hh-mm-ss

file-import-date:yyyy-mm-dd-hh-mm-ss

What do you think about this request? If it's too complicated I can live without it, but I would like to retain this data for indexing purposes. "show me all the images I saved in 2015", "images I saved in May of any year", "images I imported yesterday", "images imported from this directory", etc. I feel like these things would help me divide things into smaller workloads for the initial tagging passes of my big collections.

Also forgive my English I injured my legs and they gave me drugs so maybe I'm explaining things poorly or not seeing the feature in the client.

>>5128

Yeah, I think I might just bite the bullet and use the pragma, even though it is deprecated. Or at least fall back to it when the temp partition doesn't seem large enough.

>>5133

Great! Let me know if anything else comes up.

>>5136

Sure–I'll make sure it disables the controls and leaves a message permanently saying what happened.

>>5156

I would like to expand the manual import and the auto import to allow this sort of parsing, but I can't get to it quick. I would also like to add regex parsing to the auto importer to allow for better workflows here.

I think your best bet at the moment is to use another program or a script to convert this information you want into the import file's path or filename and then parse it through hydrus using regexes. This is a little awkward and cumbersome at the moment and will only work for manual imports, but if you batch them up it might be ok.

The client adds import date as the file's 'age', btw. You can search for that with system:age.

>>5162

Thanks for letting me know.

>convert this information you want into the import file's path or filename and then parse it through hydrus using regexes

I think I'll end up doing that but if there is any way to interface with hydrus directly, I might pursue that instead. Is there a way I can pass tag information to hydrus, either though an API, manipulating the database myself, or something else? I figure I just have to implement the means to get the metadata I want then figure out where to move the actual files and how to tell the client/database about it, is that right? Or even just insert the tags into hydrus and let the importer handle the placement of the files themselves, and hopefully the client would pick up on the tags I inserted for the matching hash.

I figure if I can make something like that I can schedule a task to have it run on a directory and simulate the automatic import until later. I just need to know how to approach it.

>system:age

Neat.

Thanks as always hydrus.

>>5171

I am sorry you are having problems.

Have we previously done some database recovery work, or have you played around with your db? Anything that might have created some unusual spare tables in your database?

Which OS are you on, and which hydrus release are you using?

In any case, please go to your database directory (default install_dir/db) and double-click the sqlite3 executable. This should open up a new terminal window. Copy/paste this block into it (and then hit enter if you need to):

.open client.caches.db
.mode csv
.once muh_sqlite_master.txt
select * from sqlite_master;
.exit

It should have created muh_sqlite_master.txt in the same directory. Please post the contents of that here, or pastebin it if it is too large, or email it to me.

>>5172

There's no API yet, but I would like to create one through a localhost http server the client will optionally run. It'll take hash/tag pairs through a POST request, so if you can script that, you could store the tag/hash pairs for now and wait for that to come in.

That might take a while though.

Wait–I just remembered I support importing newline-separated tags through 'neighbouring' txt files, and this works for automatic import folders as well. If you have a file 'blah.jpg' and have 'import tags from neighbouring .txt files' checked, the client will look for 'blah.jpg.txt' and parse it for newline separated tags. If you knock up a script to automatically create your tags and stick them in a .txt file and then pass both files into your import folder destination for automatic import, I think you'll get what you want.

Let me know if you try this but it doesn't quite do what you need.

I would strongly advise against inserting tags to the hydrus db directly. You can play around if you want, but make sure you have a backup. There are a bunch of caches that it would be a huge headache to manually adjust.

>>5175

>'neighbouring' txt files

Perfect!

The only thing I'd like to see is a checkbox for deleting those neighboring text files after a successful import in the same way you can delete the files themselves after import. Both in the manual and automatic importer.

`add tags based on filenames>delete neighboring .txt files after successful import`

In case anyone else wanted it, Here is what I wrote for the task+a Windows binary:

https://ipfs.io/ipfs/QmWPUxfKo6FcNzc3TCTm69ZoUcbteyLAaqYWos95nx9FFV

Usage is: Just run it and it will process the current directory recursively, you can also provide as many space separated paths as you want.

I only tested to make sure it works and imports, I haven't actually used it to migrate everything yet, I'm gonna hold off for a little bit to make sure that's the only data I want and that's how I want it formatted before doing it for real, when I do though I'll be sure to report any issues or success in the latest thread.

Thanks hydrus~

>>5190

Woops. For some reason it wasn't working earlier when I was testing it, it would delete the images but not the text. I removed everything but the db and replaced my install with an updated copy and it's working as intended now.

I usually just replace my install with newer versions so I had some left over files like "cv2.pyd" and various dlls that are gone in newer releases, not sure if that was causing issues before.

>>5177

>>5186

Thank you for this follow-up. It looks like the update breaks for anyone updating from v237 or before. If you would prefer to wait, I think I have fixed the update code for v245, which will be out on the 1st March.

If you would like to fix it now, I think you have two options:

If you still have a backup of your v230 db, download v238 and update it to that. Then retry v244 and it should work.

If you do not still have a backup, please make one of your database as it stands right now. Just copy the .db files somewhere at least, in case my instructions here break something permanently. Then go to your database directory and double-click the sqlite3 executable. A terminal window will appear–copy-paste this into it and hit enter if you need to: (Please note any other users with this problem, these commands only work for this user.)

.open client.caches.db
DROP TABLE specific_files_cache_8_3;
DROP TABLE specific_current_mappings_cache_8_3;
DROP TABLE specific_pending_mappings_cache_8_3;
DROP TABLE specific_ac_cache_8_3;
DROP TABLE specific_files_cache_8_7;
DROP TABLE specific_current_mappings_cache_8_7;
DROP TABLE specific_pending_mappings_cache_8_7;
DROP TABLE specific_ac_cache_8_7;
.exit

Then try to update. If the client boots, you might still get a couple of popup errors about the missing autocomplete cache tables. Even if you don't, go database->regenerate->autocomplete cache to recreate them. It will take a few minutes to complete.

If you decide to try to fix it either way, please let me know how you get on, and if not, let me know if v245 does it in the end.

>>5185

>>5190

>>5192

This is supposed to work all the time, but I'll make a note to check it out.

>>5197

>>5199

Thanks. I still have 161 tasks left in the 'rewrite' job wew lad, but a bunch of that is just stuff I haven't ticked off yet. Although I pushed a little too hard this week, it was fun work, and it is 90% done now. I unfortunately just can't hit tomorrow.