/hydrus/ - Hydrus Network

>>12423

No worries. Hydrus may fit you. It can definitely do what you want, but the technical problem of 'what tags should these files I already have?' is actually a little difficult to solve. A core piece of hydrus is my Public Tag Repository, which solves this very effectively but takes a decent CPU/HDD investment.

I recommend you give hydrus a try with just a hundred or a thousand files, without deleting any from your current folder structure. Try it out for a few weeks, maybe give the downloader a go (which is also another great place to get tags), and think about if you like it overall. If you find you don't like the program in terms of looks or workflow, no problem–you can just delete it with no harm, and if you do like it, you can migrate the rest of your 10k files over and maybe sync with the PTR. Lots of people who now like hydrus a lot have found this a good way to go.

The help has a rich getting started guide that can walk you though doing this: https://hydrusnetwork.github.io/hydrus/help/index.html

If you decide to give it a try, I'd be interested to know how you get on, whatever you decide.

>>12411

Both of the drives have a significant amount of free space. The drive Hydrus is on was indeed getting pretty full percentage-wise before, and I thought that may have been the issue as well, but there was still quite a bit of actual space (at least multiple tens of gigabytes). It now has 413GB free, but that doesn't seem to have remedied the issue.

I assume it has something to do with the total amount of information being saved in the subscription. I noticed that at first it was only happening when I'd let the subscription run for an obscene amount of time (multiple days) and it would fail at saving the subscription progress. When left for a shorter amount of time (and thus less items downloaded), it seemed to wrap up just fine. But now it seems even smaller jobs are too much. As mentioned, however, the images and tags themselves are apparently being saved to the database just fine as I'm able to access them after closing and reopening the program, using whatever applicable search query I desire.

I currently have all of my Pixiv follows in the same subscription, mostly due to the fact that Hydrus detects if I try to add a duplicate artist to a subscription. Currently it's monitoring 5,212 artists with a progress of 1,369,868/1,543,701 - 12,763 ignored. Going into the sub and modifying it seems to behave normally. I haven't had any issues with that, and testing it just now worked as well.

There wasn't anything before the 953MB dump in the log, but this was what followed it:

https://pastebin.com/UpYmWXRe

Other runs and log entries were identical, aside from the line numbers between upgrades. This one is from v348.

I split off about a thousand artists that hadn't completed yet into a new subscription. I let it run for over a day and stopped it just now. It looks like the subscription status for the new, smaller subscription saved just fine (currently about 82k/410k items processed).

Shit man I'm sorry, I tried but I'm too stupid to work this out, even with the guide. I give up. Good luck in your later endeavours and thank you for answering in the first place.

Is hydrus network currently able to download from artstation? I tried running a gallery downloader using the artist name as a query just like I would for other sites and the status of the search immediately becomes Done without giving me any pictures. I tried a bunch of different artists to be sure and none worked.

>>9327

How to disable auto subscription checks so I could run check-ups only when I want them, but let them retain check times?

it wont let me import webms. just says failed to import with no reason given. Also it repeatedly crashes when trying to organize the pictures, is there any logging I can review?

running on Debian buster.

So is hydrus hosting its own server to use sql databases or such?

For the last week, without fail when syncing with the repository without shutting down to do it hydrus hangs for near ever. Is this normal?

>>12465

I should note that it also freezes up when checking db integrity

>>12433

Yeah, sounds like it is just the size of the object. 1.5M is much larger than I expected you would say. I guess some part of python/json/SQLite can't handle it. Thank you for this feedback. I'll see if there is a generalised way I can have a check here that says 'woah m8 that sub is too big, split it up' once it reaches too many total URLs. At least until I break up the subs object overall and make it more lightweight.

>>12435

No worries, thanks for giving it a go.

>>12445

It looks like the Eternal Cloudflare got applied on their gallery request URL and now hydrus 403s. There isn't a good fix for this atm, so I recommend you pause your Artstation subs.

>>12449

Best bet, I think, is network->pause->subscriptions. Turn it off when you want them to run, then turn it back on to pause them.

>>12450

Unfortunately, some flavours of Linux do not work well with my build release. The webm problem is likely because it cannot figure out the right environment to find ffmpeg. The crashing is probably related to your OS not liking my Ubuntu .so files.

I cannot guarantee this will work perfectly, but a good number of Linux users run from source. Here is my help on this:

http://hydrusnetwork.github.io/hydrus/help/running_from_source.html

Let me know if you try this and run into any trouble.

>>12463

No, I use SQLite, which all runs in the same application.

>>12465

>>12466

When it does this, is it using any CPU/HDD under Task Manager? Are you on HDD or SSD? Is that drive full, or is your system drive near full? What happens if you restrict it to only, say, three minutes of shutdown work–is it ok then, or still hanging?

Hydrus consistently crashes if I leave it processing gallery downloads while I am not using the computer for an extended period of time. It doesn't crash while doing the same lengthy tasks while I am actively using the computer.

I have checked all of windows 7's advanced power settings and made sure my computer is never entering a sleep-like state or otherwise diverting power due to inactivity, but the problem persists.

Anything I can try to fix this?

I'd like to use SyncThing to sync up the databases between my Windows desktop and my Ubuntu laptop. Would migrating the database to a shared folder as described (https://hydrusnetwork.github.io/hydrus/help/database_migration.html) be the best way of doing this? I don't want to unintentionally bork my files.

Many thanks for the excellent software by the way, it's renewed my interest in collecting images.

Is there any way to customize the image viewer?

Stuff like getting rid of the tags on the side and showing 2 images at a time.

>>12491

Hi based dev, different anon here on Linux Mint. I think this might be related to my own problem and just wanted to confirm it. When I first made the switch to Linux a few months ago, my subscriptions worked fine for the most part. There were a fair few of the files that would download, hang for a split second at 100%, and quickly say "error!" before moving on to the next file.

I assumed it was due to my new place's poor internet connection, but I've noticed that Hydrus keeps trying to redownload the same files to no avail. Fast forward to today and Hydrus now goes through several dozen of these errors before moving on to newer files. The newer files are more of the same, most download fine but there are still some errors mixed in.

So I'm assuming that Hydrus has been accumulating these "erroneous" links ever since I started using it on Linux. Can you confirm if this is this a known symptom of Hydrus not knowing what to do with webms after it downloads them? If it is, do you have any plans to address Hydrus' Linux webm support in the near future?

Regardless, thanks for all of your hard work dude.

Is there any way for me to find zip/rar files? using filetype:zip or filetype:rar doesn't seem to work at all. Yet these types of files are visible in the database (found them by sorting by filesize > largest first).

>>12496

Unfortunately, siblings, parents, and censorship is a rats nest when it comes to system interaction at the moment. I recommend opening the files in manage tags and hitting the new 'fix siblings and parents'' button, but I can't promise anything. Like much else, it strains at the seams and is all pending a big rewrite.

>>12497

Thank you for this report. I am sorry you are having trouble. I haven't heard about consistent crashes on Win 7 before. I have two first thoughts:

Could your video card drivers have some problem with doing OpenCL for client.exe? OpenCL is GPU hardware acceleration for image and video processing that hydrus uses when importing files. Your driver may have per-executable options for this–if you turn off OpenCL (may also be called CUDA for Nvidia, if you have an old card), does that improve stability?

Is the program minimised when you leave it alone? I remember a long time ago, some Linux users had problems with minimised hydrus queueing up various UI events that couldn't be executed due to the program being minimised, which ultimately crashed the program. If you leave the program alone, but as a window, does that improve things?

Otherwise, what happens if you turn on help->debug->report modes->db report mode and leave the client alone? It'll make lots of popups and write them to the log about current db work. When you come back and find it crashed, please then check the client log in your install_dir/db directory and scroll to the bottom–what was it doing last, before it crashed? Is there any obvious cyclic pattern or similar in the minutes before the crash? Feel free to cut the relevant data and pastebin it here or email it to me to check more closely.

>>12515

I don't recommend storing the actual db files (client.db and its friends) in a non-local folder, like a network drive. Many network interfaces do not provide the true locking ability that SQLite wants to do its work reliably. If this is your shared folder solution, you can give it a go, but I strongly recommend you make a backup somewhere safe first. I'd be interested in how it goes. If it doesn't work, or your SyncThing idea is somewhat this already, I'd say having two scripts that overwrite the db and its files from the Win and Linux machines and back again any time you want to switch over. If this is practically only a five minute job for your situation and you switch between the machines on a reliable schedule, this could work ok. If you were feeling clever, you could store the client_files folder on the shared folder and just copy the client.db etc… db files back and forth, which could take just like 30s.

Otherwise, there is no problem having the Windows or Linux versions talking to the same db. As long as the relative paths line up right and only one client talks to the db at once, you should be good. Please let me know if you encounter problems.

I am really glad you like it!

>>12521

Not really, I am afraid.

>>12524

Yeah, the situation is bad when the client can't find ffmpeg. It generally assumes it has it. I will make a job to improve the handling here–I may even be able to put in a catch for "this client can't do video, so bail on any video downloads".

Can you do me a favour and try to put one of these https://johnvansickle.com/ffmpeg/ static build ffmpeg exes in your install_dir/bin folder? Do webms then work for you?

Does your Linux have ffmpeg, btw, if you just open a terminal and hit 'ffmpeg -version'? I know some users just didn't have it for some reason, and since Linux Hydrus assumes you have it rather than bundling it in the release like for Windows, that was the real problem, rather than PyInstaller path environment fun.

>>12536

Is 'filetype:zip' a tag you are searching for? My PTR has a variety of filetype and 'medium:webm'-type tags, which I am leaving in place for now but will eventually purge.

Please try system:mime, and select the 'application' archive mime types you would like. Does that work?

>>12547

That did exactly what I needed it to, thank you so much!

does hydrus have a "open file location" option for images? I know they're gonna be in random clusters of fucky folders, but I need it for drag and drops since doing it from the hydrus client doesn't seem to work everywhere

>>12561

right click image >share>open>in file browser

Is there a way to do a partial Hydrus backup? Don't exactly want to make my external HD go through copying 50GB of images in one go. Would just copying a few folders over at a time work?

the rightclick menu on an image has an "archive" option, what does that do exactly?

>>12567

It sends the image from the inbox to the archive.

>>12561

>>12562

You may need help->advanced mode on for this to work, and it doesn't focus the actual file for most Linux default file managers iirc. A little funky on Windows, too, but if it doesn't select the file on the first attempt, try again.

For weird drag and drops, try options->gui->BUGFIX: Discord file drag-and… checkbox. Other solution is to DnD to a staging folder, like your desktop, and then to your destination program. It is all OS-level inter-application DnD permission bullshit that I unfortunately don't have easy access to.

>>12565

Yeah, for any kind of clever backup, use a program like FreeFileSync while the client is shut down. Set up a backup of your install or db dir and do it in parts. The db/client_files folder is where all your files are. The 256 subfolders in there in the format 'fxx' are where your files are. Just do them in batches of 32/64/whatever. You can just copy manually if you like as well. More info:

https://hydrusnetwork.github.io/hydrus/help/getting_started_installing.html#backing_up

https://hydrusnetwork.github.io/hydrus/help/database_migration.html

>>12567

>>12568

When new files are imported to the client, they get put in an 'inbox', which is basically a special kind of tag. I designed it to work like your email. They get the little 'envelope' icon in their thumbnails. It is an easy way to mark new things that you likely haven't looked at once yet.

You can find files in your inbox with the 'system:inbox' search term, and 'system:archive' to find those that are not.

I recommend you set up a workflow where you process your inbox files in batches of, say, 100 at a time. Anything you like, you can hit F7 on (send to archive) to say 'keep this', and anything you do not like you can delete.

Once you have played around with this a bit and feel comfortable, check out the 'archive/delete filter', which is a fast way of doing this job. More to read at the bottom of this document:

https://hydrusnetwork.github.io/hydrus/help/getting_started_files.html

>>12545

SyncThing isn't a networked drive, it uses PTP to sync files between two local folders on different devices. It sounds like this is likely to break Hydrus, but I can give it a try. Ideally I would like to have the same database on both machines…

>>12575

If you can easily turn the sync on and off, or otherwise make it only push when you tell it to, you are good. As long as you can trigger a sync only when both clients are off, you are good.

But any kind of continuous sync will not work, I think. I know a user who had his Google cloud backup sync working all the time on his hydrus folder and it ate a ton of bandwidth and messed up SQLite's write locks while it was working and actually broke things locally.

>>12573

I recommend adding a unique local tag like 'muh sibling swap' to help batch this job up. The implicit default file limit on any normal search is 10,000, which I recommend. So:

Apply that tag to all files (i.e. search for [files you want to swap, '-muh sibling swap'] and then apply the tag to batches of 10k files)

Search for 'muh sibling swap' in batches of 10k and:

- Do a big ctrl+a, F3, swap siblings and add parents

- Go to local tags and remove 'muh sibling swap'

Refresh search to get another 10k.

Depending on your situation, each job might be 20s or a few minutes of CPU work. If that is 22 jobs of 10K files for your inbox, then plan it over a day while you are working on another computer or something else that doesn't need a lot of CPU on the same machine and it'll be manageable.

Do not try to remove the file limit and apply to all 200K images in one go. These operations are not usually linear. I'm willing to bet that sibling replace is n-squared, so a single job 20 times bigger will take 400 times as long! If 10k mega-lags for you, you can even play with adding system:limit=3300 or something, although of course you are then increasing total human overhead of the job.

However, since I now have this auto-fix tech, I expect I'll duplicate it at the db-level and at some point just let you auto-apply it to everything. You might like to just wait 12 months for that to come in instead.

If you give it a go, let me know how you get on!

Is it possible to add a sub tag to the system:archive to create a clear separation between types of porn. When I browse without tags, I want to be able to see only one type of porn, like in a normal folder.

>>10095

So if I change my mind about a tag name, I have to individually remove the old tag and add a new tag for every file? This is crazy pants for a program centered around tagging. What am I missing here?

>>12593

>What am I missing here?

Tag siblings.

How do I force a specific white space character? I want to disallow spaces and force all tags to use underscores as spaces.

eg: "short hair" should always be "short_hair".

I understand I can search for each of my several hundred tags and mass update them but this is a very slow process due to the sheer amount of images that need to have tags fixed.

>>12596

>Underscores in a tag such as "Short hair"

>Not "Hair: Short Hair"

You disgust me.

Is there a method to save the threads that goes with the images?

>>12610

Namespaces are for meta information like the artist, name of the character, series the character is from, whether the image is lewd or not, etc. Tags are for descriptions of what the image contains. Short hair is a description of a character in the picture. With very few exceptions a namespace should only exist once per image. A notable exception is when a picture shows a character who exists in multiple series (eg. Rage of Bahamut / Granblue Fantasy or if you're someone who tags seasons as separate series) or characters from multiple series (eg. crossovers).

You're doing it wrong because you end up with garbage like the following on a single image featuring two characters.

>hair: long hair

>hair: short hair

>>12626

You're doing it better in terms of naming, but you're still doing it wrong.

>>12629

sounds like a very subjective description of namespaces to me.

not sure what the difference to you is between having two namespaced tags "hair:short","hair:long" and having two regular tags "short hair","long hair". Either way you have two tags.

In fact, using namespaces liberally for tags that are important to you (e.g. if you like to add lots of hair descriptors) is useful because the tags will be grouped together when displayed. When checking which such tags are already on a file, it is much easier to find them if they are grouped together.

Maybe you have almost entirely files which contain only one character? I know I don't

>>12593

>>12594

Yeah, try tag siblings:

https://hydrusnetwork.github.io/hydrus/help/advanced_siblings.html

The system is very imperfect, but it generally works. More time will be put into this this year, including better clientside preference management for tags synced over the PTR.

>>12596

There is no solution for this yet. I expect a future version of the tag siblings system to allow some sort of regex-based replacement (think a global rule for 'replace all "\s" with "_"') that will allow this.

Since desiring underscores to either go away or be mandated is common, I am likely to hardcode a solution to manage this earlier than a generalised regex solution, but the tag siblings system just isn't clever enough to handle it yet.

>>12629

>>12632

I have yet to come down to a good description for what namespaces are myself. Here is my current first draft:

Namespaces are good when:

They are a higher category (evangelion is a series)

They are useful to highlight (characters are important)

They are useful to search for (creators are often searched for, and 'creator:*anything*' has value)

They are useful to group together (multiple creators is useful to know, when listed next to each other)

I've been approving 'clothing:' siblings on the PTR for a little while now and I overall like it. I'm mixed on some others though. My ideal solution here is to extend tag siblings to allow clientside preferences and then for you to say "If a group of siblings includes one with 'hair:' namespace, prefer that". I have found that it is easy for users to objectively agree that 'hair:long hair' and 'long hair' have the same semantic meaning, but the big subjective disagreement is over which is better.

An actually bad namespace, imo, is one that breaks the first rule above. 'male:erection' is an artifact of how some gallery sites do various female/male focus on tags, but I would rather 'male erection' (which could nicely have parents 'erection' and 'gender:male'). 'erection' is not a 'male'. Also 'erection:male'.

'general:coffee cup' and 'object:pencil' are technically correct but overspecific for most users, but again I think the specificity of namespaces is highly subjective, so the true answer is to let users define what namespaces tags could have and let them then customise what they prefer to show.

If you were going for 'hairlength:short', I'd probably try to aim for hairlength:short hair for similar reasons, or go to the higher category of 'hair:short hair'. 'short hair' is both a 'hairlength' and a 'hair', but 'short' can mean a bunch of stuff.

I am not confident though.

NEW THREAD

>>12641