/hydrus/ - Hydrus Network

Is there anything especially good about cppcryptfs besides being able to run on Windows and providing encryption of loose files and filenames? I would be worried about using something that's clearly marked "experimental" and only used by a few people.

Have you tried cryptomator, encfs4win, encfsmp, or mounting a secret rclone provider?

Question is, why do you feel the need to have your imageboard memes encrypted?

>>6712

Use LUKS

Encrypt the whole volume.

>>6718

>losing your privacy to some Silicon Valley nascent AI, wew lad.

Does encrypting the whole db file work for backup only if you don't plan to use it actively?

Also, on this "incremental backup" where only the actual changes and not the entire things are copied over, how does encryption play in this?

I add a single image to my db, then encrpyt it again.

The added image is just a few mb, but will I have to update every GB of the entire database if a single thing changes and it's re-encrypted?

(I am using veracrypt)

>>10556

>>6718

Encrypted ZFS could be a solution

>>10556

To make it simple, backup the contents of one encrypted container to another. Do not backup encrypted images, as you'll be doing 500GB every time. Instead just mount the two images and backup their internal folder structures as normal, as if you were backing up an unencrypted location.

Afaik, there's no super easy way to do a minimal backup where you only update the sectors of the image/files that have changed, although there is probably some specialised software out there that does that by tracking sector hashes or something (although this would presumably have to rescan the whole contents of the source anyway to spot differences, or be part of the fundamental file system like >>10564 suggests).

It is simpler just to use FreeFileSync and copy-overwrite the files with newer modification timestamps to the backup. db files can be pretty big–maybe on the order of 10GB for hydrus, but 10GB is ultimately not a big deal for local backups once a day/week/month. The media files are just jpgs and whatever and copy in reasonable time as they would in any other situation. I do a weekly backup of a 1 million file hydrus db to a usb drive, and it takes about five minutes to scan and five minutes to action. Easy peasy.

I don't know enough about veracrypt to comment cleverly about it. If you are putting files in an encrypted container, it is easy, but if it does some mickey-mouse on-the-fly per-file encryption, then your situation may be different.

>>10578

This is very late, but I want to thank you for your reply, Dev.

If it'll run with the entire hydrus directory in an encrypted container, then it's all dandy. I'lll just try.

Read only if time is there:

I will probably put the entire thing into an encrypted container, I don't know what I was thinking, murmuring about some sort of container only for the db folder.

Plus, it's not like excluding the program files has any benefits, they're tiny in comparison.

I just have to live with mounting and dismounting each time I backup.

What am I babbling, obfuscation is the primary objective for encryption, so of course

I can't backup unmounted and expect the program to not see it as the entire thing having changed.

Inconvenience or security is the dichotomy I guess.

>>6712

>Hydrus runs fine inside a Truecrypt container

Pretty mediocre at best. LUKS runs better.

> leaving it too big or too small all too easily

Use Linux' LVM, then you can grow it (shrinking is not directly possible in most situations, but you can create a new smaller LV, copy the data, delete the old one).

>>10578

>>11116 (cont'd)

One more thing, if you use Borgbackup for backups, that can actually make backups relatively efficient - even for the Sqlite databases you should save like 70%, and for the actual files it will be next to 100% of what was in the older backup.

So on a re-run your 500GB backup will likely be just the new / changed data plus maybe 1-2GB or something on that fast method of [backup-side] deduplication not being entirely perfect.

This is basically a "super easy way" to do a relatively minimal backup. Well, maybe it's only "easy" rather than super easy if you're currently on Windows and need to use the Win 10 Linux Subsystem, a network share or a live USB or a VM or something to run Borgbackup on that directory, but it's not going to be terribly hard either.

>>10578

That is not going to be convenient unless we can make it simple (especially when someone has TBs of data on a RAID-enabled NAS)

>>6718

Any comments on these?

https://github.com/restic/restic (Golang, not too slow)

https://github.com/gilbertchen/duplicacy (Golang, fastest)

https://github.com/duplicati/duplicati (C#, fast)

https://github.com/bup/bup (Python, slow)

https://github.com/jborg/attic (Python, slow)

See https://github.com/gilbertchen/benchmarking for info

>>11118

Borgbackup is your ticket to make it simple.

>>11123

> https://github.com/bup/bup (Python, slow)

Actually this one is a fine CLI tool too, but unlike borg it's not considered stable by the developers, and you likely do want some borg features like the ability to prune older database backups and staggered retention for the other files.

Slow is extremely relative. Yea, python is not the fastest language around even if you use pypy (which however definitely makes most software that crunches some data quite a bit faster over standard python). But even that may not matter much if you're mainly waiting for IO coming from your HDD, which is usually what happens if you try to get data about your 40TB of image files (or whatever you might have on your array).

> See https://github.com/gilbertchen/benchmarking for info

A very limited test without the various configurables and apparently without any attempts to isolate how much time was spent waiting for IO or whatever, done only on the weirdly scheduled OS that is OSX?

And I don't even see what interpreters they used for python (was the "python" symlink pypy or mainline python and not a link?) and such. Never mind there are actually a LOT of configurables in these tools, and for some weird reason they didn't even try to pick the same compression algorithms even when it was possible (why not run the same settings on the same zlib or lz4 or zstd compression on all the tools so they become more comparable?).

Anyhow, use Duplicacy or restic over borg if you prefer, but I wouldn't pay too much attention to that benchmark.

>>11123

I'm using duplicati myself, it's still slow because OS chokes on the amount of hard drive seeks, takes around 30 minutes to back up a 100k set for me, probably going to be even slower when you're backing up from an encrypted volume. It doesn't even matter if there were any changes to the collection, just the act of checking 100k files on a hard drive is a very slow process.

>>11123

I haven't used any of those, so I can't comment cleverly. I expect they do the job well. I use FreeFileSync once a week on my laptop's IRL 1.2M-file hydrus db, backing up to a WD passport. It takes several minutes to compare all the millions of files and thumbs and then a few more to sync the backup. Unless I have a gonzo week, I'd estimate it is usually less than 20min total. Since I have it going in the background as I put the hydrus build together on my dev pc, it is no trouble at all.