/hydrus/ - Hydrus Network

>>14627

Every situation needs a little wiggling, but try something like:

For a file
E:\downloads\comic_name\12345 - skirt blue_eyes samus_aran.jpg

Try:

id -> (?<=comic_name\\)\d+
tags -> (?<=\s).+(?=[\s\.])

Basically, for id, 'all the numbers after "comic_name\", and for tags "all characters between space and space or period".

>>14636

Ah shit, try:

(?<=\s).+?(?=[\s\.])

instead. That ? makes the .+ non-greedy so it pulls each word as its own tag, rather than the whole string of tags as one.

>>14632

Thanks!

>>14637

>>14636

Thanks, regex on this level is a mystery to me, but I kind of understand how yours work. I just wouldn't be able to create it myself.

>>14637

yo, i'm the poster that had problems with favorite tags resetting to 0 after 200 favorites. givin an update to say after i recovered my db in v405 and cleaned the image/thumbnail and ran memory maintenance i now have 500 favorites with no reset to 0 so far.

now that i got the favorites in order, i wanna see about blacklisting some stuff. it look like u gotta go through several menus to blacklist a tag. any thoughts on adding a rightclick tag option to blacklist once other actual important stuff is done?

Would it be possible to have Hydrus automatically search for an exact URL if I paste one in the search field, instead of having to select the URL option and then pasting it in the exact URL field?

>>14643

Or at least an option to set a shortcut to get to the "known url" window, because the "system:" menu has a lot of options. Maybe shortcuts to each item in the "system:" menu are in the works already?

Hello.

I updated from version 388 to version 407 today and noticed that when I am adding tags through the 'manage tags' window, if I hover over the media viewer window, the focus completely shifts to that window. Then I have to manually re-select the tag textbox to type in more tags.

I used to be able to scroll to the next image with my mouse and type in the page number in the meantime.

Was this change intentional, or is this just a bug on my side?

>>14593

file > options > sort/collect > add > type in artist

alternatively, type in something like artist-title-page or whatever if you want to incorporate the artist namespace into whatever your preferred sorting schemes are. dropdowns near the top let you set defaults

>>14647

>>14641

>>14646

>>14642

This is for blacklisting in the downloader, saying 'don't download the file if it has "guro"' etc…? Yeah, there isn't a way to spam this with tags. Since you only usually set it up once, and most users only have a handful of no-nos, I haven't expanded it.

I'll add a paste button to the main input, since that is easy. See if making a notepad of bad tags and then pasting them in one go works out, and let me know if it is still too awkward.

If this is for just hiding tags, yeah atm it only works when you right-click one tag. I think that could also do with improvement.

>>14643

>>14644

Hmm, that sounds reasonable. I do want to get completely editable system predicates in the near future, so if you are doing lots of individual URL lookups, one after the other, you'll be able to quickly edit the one currently in the search and just put in a diff URL.

Also, one easy thing I can do is let you paste multiple URLs in the search pred, and it'll do an OR search over all of them. Would that make your situation easier, before I get the complex side done?

I added a subscription (nitter retweets) to an account with a long archive, but I really only want to get the new files from this point on. I set the first run files number to 100, but every time it runs it goes looking for progressively older files instead of grabbing the new ones. Is there any setting I can change to make it stop doing that?

I setup hydrus-dd and api-thingy. How can it help me tag my files, do I just batch the client_files into it and it just werks? lel

Is there an existing/planned support for symlinks/hardlinks instead of "copy," particularly on winblows?

Primarily to save space for large synchronized folders.

Why Hydrus doesn't support *.jp2 (jpg200)? It's so much better compared to jpg in terms of compression.

Was the in-client backup option removed?

can i import/export the list of previously deleted files?

>>14662

Hmm, I am not sure. It sounds like you have it set up correct, so perhaps the way nitter is delivering files is unusual. Is there any chance the files are coming in with a different 'sort' than usual (i.e. not 'newest first'), or it is just a normal username search?

If you go into the subscription and drill down to the precise query, going even further into its edit window, there should be the 'file import status' button which will show you the actual URLs it has been working on, and also the 'gallery log'. These will have the full data, and may shed light on whether it is getting things in the wrong order, or maybe how it is making decisions on fetching new gallery pages. Are the URLs in ascending id (10000, 10001, 10002) order, or are older ids appearing later, after that initial sync? Is the gallery search doing more than one page per subsequent check (it should have info about how it worked in the 'notes' column of the gallery log)? Normally the gallery will be like 'found 5 new urls, not fetching any more pages'.

Normally 100 initial files, and 100 per-check files works well for most sites, nitter included. Does this artist have a lot of retweets of other artists? Do their tweets tend to be single images, or multi-image galleries?

>>14663

I am not sure how much it is in total, but it will be a decent multiple of the total end database size, which I think is now about 30-40GB. It makes regular checkpoints to stay safe and not lose data, which involves saving to disk.

Catching up on the PTR is essentially doing about eight years of processing work at once. I generally recommend to leave it work in idle time over a few weeks, but if it is set to do lots of work in one go, it'll add up. If you are completely synced up now, it should settle down to a little bit every day or so.

Depending on how that data is exactly counted in your Windows version, that number may be slightly misleading. I know some versions count 'reading output from an executable' as disk I/O, so when hydrus reads file metadata and generates thumnails from ffmpeg during file import, it can add a casual 400MB when it is really just reading from memory. So if you were doing file imports, or using something like my old video renderer, it may be slightly off. I don't know if those read bytes are legit or include pulling from the disk cache in your ram.

In any case, the PTR involves periods of heavy CPU and disk activity. It is something like 880 million mappings now, and a mechanical HDD can't keep up any more.

>>14668

I believe there are several ways to use it.

I think the simple method for already-imported files is:

https://gitgud.io/koto/hydrus-dd#lookup-script

under network->downloader components->SEMI-LEGACY: file lookup scripts, and then turn on 'file lookup scripts' under options->tag suggestions. Make sure the dd server is running. Then, your manage tags dialog for single files will have a new 'file lookup' column with the 'dd' entry, which lets you query the dd server from hydrus with the file and get tag suggestions back.

With the Client API connection, things are more complicated. If you have an external script that can generate your imported files' hash names, I think you can make that script call dd to tag through the client api, like so:

https://gitgud.io/koto/hydrus-dd#for-api-intergration

I don't have personal experience with it. Koto is in the discord, if you are comfortable with that, and some other users who do have experience with it.

>>14671

I've played with it a bit, mostly Linux experiments, and some stuff to help IPFS access the same files. I am not completely opposed to the idea, but I will first have to extend my file storage system to deal with more error states. At the moment, it assumes that all files are there and actually there. My next plan for file storage is to allow multiple potential storage locations for a single file and to have graceful 'this file storage isn't available atm' mode, which will include multiple 'missing/weird location' error handling.

I understand you can hardlink folder locations now and the file storage system is generally ok with it. Some lads have to do that to get some NAS systems working right, I think.

I'll caveat all this with my thoughts on preserving original folder structures, though:

https://hydrusnetwork.github.io/hydrus/help/faq.html#external_files

If a media collection is large enough that space is a worry, it may not be worth trying to manage it anywhere with a friendly folder structure. I recommend 2GB movies keep their nice filenames and are kept out of hydrus, but all the tens of thousands of webms are just too difficult to keep track of. If you want a small collection of favourite files somewhere easily accessible, then an 'export folder' in hydrus may suit you well.

>>14673

Is that JPEG2000? Do you have any example files I can look at and try to get working my end?

>>14676

I don't think so. If you are ok with a bit of SQL, you can pull it right out of the database. Would something like a list of hex sha256 hashes work? Do you want to migrate this list from one client to another?

>>14682

>Do you want to migrate this list from one client to another?

pretty much

>>14651

That would make it easier for sure! It would be great to have all of those features, but beggars can't be choosers!

Is there a way to change the default tab names? For example if I create a new gallery tab I would like for the tab name to simply be G. Files tab would be F. Etc.

I often have enough tabs open that I need to rename them to more condensed names. Otherwise they take up so much space that the navigation arrows appear and the Qt glitch that causes the current tab to be on the far right makes it hell to navigate through the tabs. I am aware of the page of pages workaround but dislike that way of doing things.

Alternatively is there a way to enter the next/previous tab using keyboard shortcuts? I looked in the shortcut command options but couldn't find a command that would do that.

Would it be possible to integrate YouTube-DL into Hydrus?

>>14695

Nevermind, I reinstalled it and it works now, but I have no idea what I did differently though.

>>14671

Bumping because it got mixed with the wrong replies:

>Is there an existing/planned support for symlinks/hardlinks instead of "copy," particularly on winblows?

>Primarily to save space for large synchronized folders.

Any chance of multiple local file domains in the same installation happening within the next few years or it still a pipe dream?

I've been using hydrus for a while now, and it's been wonderful for my image collection.

Right now, I'm trying to do something similar for the zip archives I have (10k+ archives mostly from sadpanda). Is there a tag repository for things like this? I've tried the PTR, but it doesn't seem like the tags for archives is well developed. Another problem I have is the thumbnails. Without having to do intensive stuff like change the code to read into the zip file to get the thumbnail, is it possible to change the thumbnail or find the thumbnail in the filesystem and replace it with the desired thumbnail?

>>14709

>is it possible to change the thumbnail or find the thumbnail in the filesystem and replace it with the desired thumbnail?

Yes, but you'll have to do it yourself.

If the file is under db/t[number]/[fille].something then the thumbnail is under db/f[number]/[file].thumbnail

Tried posting this once but I don't think it worked becaues captcha

I can create a new database on a Samba share but I can't move an existing database to the Samba share through any method. I'm on macOS, so I've tried both the built in database migration tool and manually copying ~/Library/Hydrus. Nothing worked. If I manually copy ~/Library/Hydrus to a USB drive, I can start Hydrus from that database successfully.

The error and client.log just say "serious error" and dump out a call stack trace. Here is the trace:

2020/09/11 12:28:11: Traceback (most recent call last):
  File "Hydrus/hydrus/client/ClientController.py", line 1798, in THREADBootEverything
  File "Hydrus/hydrus/client/ClientController.py", line 803, in InitModel
  File "Hydrus/hydrus/core/HydrusController.py", line 498, in InitModel
  File "Hydrus/hydrus/client/ClientController.py", line 189, in _InitDB
  File "Hydrus/hydrus/client/ClientDB.py", line 299, in __init__
  File "Hydrus/hydrus/core/HydrusDB.py", line 185, in __init__
  File "Hydrus/hydrus/core/HydrusDB.py", line 454, in _InitDB
  File "Hydrus/hydrus/core/HydrusDB.py", line 494, in _InitDBCursor

(don't know if code tags work here, never posted on 8kun)

and the offending line:

    def _InitDBCursor( self ):
        
        self._CloseDBCursor()
        
        db_path = os.path.join( self._db_dir, self._db_filenames[ 'main' ] )
        
        db_just_created = not os.path.exists( db_path )
        
        self._db = sqlite3.connect( db_path, isolation_level = None, detect_types = sqlite3.PARSE_DECLTYPES )
        
        self._connection_timestamp = HydrusData.GetNow()
        
        self._c = self._db.cursor()
        
        if HG.no_db_temp_files:
            
            self._c.execute( 'PRAGMA temp_store = 2;' ) # use memory for temp store exclusively
            
        
        self._c.execute( 'ATTACH ":memory:" AS mem;' ) # ERROR HERE

I can't figure out what the problem is. My first thought was some kind of permission problem, but the file permissions are the same whether Hydrus creates a DB on the Samba share or whether I manually copy or use the built in migration to move an existing DB there. The permissions match the ones of a DB on my local machine (rw-rr on files, rwxr–r-xr-x on directories), and they appear as owned by me and I can read/write/delete them no problem. I'm hoping someone with experience with Hydrus internals can at least point me to things to try, because I am lost.

hi this is my first time posting here

I use manjaro and I downloaded Hydrus and wanted to get a server running, but when I launch the server executable it doesn't seem to do anything. I then tried running it by using "python 3 server.py and got this error in the hydrus_crash.log

Traceback (most recent call last):
  File "hydrus/server.py", line 120, in <module>
  File "hydrus/hydrus/server/ServerController.py", line 32, in ProcessStartingAction
EOFError: EOF when reading a line

I don't seem to be scraping translations on gelbooru.

Is there some parser that will grab these ("notes" / page_notes on gelbooru)?

>>14715

Found my problem. I just ran a text diff on client.db created on my local machine vs one created on the Samba, and a few bytes different in the header led to me discovering that the database had journal_mode=WAL on my local machine but the "legacy" journal mode when created on the Samba share. SQlite docs say WAL-mode does not work with DBs on network storage. I also found the Hydrus option no_wal. Running Hydrus normally, creating a new WAL DB, then running Hydrus no_wal to change it to legacy journaling, and then finally copying it over to Samba and running Hydrus again worked. Going to try it with my actual DB now. I don't have SQL experience so please let me know if what I am doing will lead to breaking things in the future. It seems to work for now.

>>14684

I am sorry I have been procrastinating on this question. When I looked at the SQL, it is actually a pain in the ass to insert back in manually. Since this has been asked a couple times before, I will just write a way in the client to do import/export, and then it will be easily sorted.

>>14693

There's no keyboard shortcut yet, but I want this easy navigation.

I will think about adding options to edit the default names. Although I can see your problem, I also encourage slender gui sessions as much as possible. If a client has dozens and dozens of pages open, a better solution is often to figure out a neater workflow, perhaps with temporary processing tags on 'my tags' and saved searches to quickly bring those files back. I will think about it though.

I collpased my mess of static 'sort this shit out' pages down to a single 'processing' page when I added favourite/saved searches, and it has been a godsend.

>>14694

I would like to. My current plan is to write a 'external executables that (optionally) eat files and spit out files' system that we can enter any exe into, and then plug that into the downloader or general file conversion. youtube-dl, ffmpeg, imagemagick, waifu2x, anything like that.

>>14695

>>14701

I am sorry for the trouble. I am not sure what was going on there, it looks like the downloader jobs did not start. Please let me know if you get this again.

>>14700

Do you have help->advanced mode turned on? I think I hide some of those clever commands behind that.

>>14702

Ah, do you mean for 'export folders'? I experimented with this with manual export (I think you need help->advanced mode turned on), and I did have a plan to try this out for export folders. I'll bump this up and take a proper look.

>>14703

It is not a pipe dream, but it is such a big job that it'll have to win a 'big job' poll. I expect to get through three or four big jobs a year once I am back to normal schedule, and the next poll should be up by Christmas. Let's see how it does there, and we'll see when we can expect it.

>>14709

>>14712

>>14713

I plan to have nice support for traditional zip/cbz/cbr files in future. It is sufficiently popular that I expect it to win a 'big job' poll pretty soon. I'll have proper thumbs, browsing of internal pages, and support for conversion back and forth between individual files and compiled 'chapters' or 'volumes' or whatever. This will be a part of my attempt to solve the problem of handling awkward 'page' tags.

>>14715

>>14721

Thank you for this information. I will attempt to detect WAL and networking gubbins in future. I generally do not recommend running the db itself across a network share due to latency issues, but I hope I can at least detect the situation and throw up a warning, and ideally recover and set WAL off.

NEW THREAD

>>14731

>>14731

>>14731

>>14731