/hydrus/ - Hydrus Network

>>15124

The plan to move to Endchan only has changed!

Codexx on 8chan.moe has kindly offered to host me on /t/ over there. As I have never been a comfortable BO, this is a much better solution for me as the primary place to talk Anon about hydrus.

I will be maintaining a Hydrus Network General, the first instance is here: https://8chan.moe/t/res/2219.html

The links in the release tomorrow will reflect this. Endchan will remain as a bunker.

>>14683

>If you do have it, is MPV the player under options->media? If your hydrus does not have libmpv access, or is failing to import it, it will fall back to my native viewer, which has no audio support.

Fixed. Thanks a lot.

I'm new to Hydrus. Using Linux MX.

Before I was using Tagspaces which is a sluggish piece of trash. Looking for a more suitable alternative I bumped into Hydrus; so far I'm very pleased by its performance and rich settings.

>>15118

Is there a way to get the blacklist back to working? There's usually a cookie for you blacklist on gelbooru but for some reason the downloader doesn't use it and stopped working for me a month or so ago. I can't remember if it was something I did or if the site changed or what.

The plan to move to Endchan only has changed!

Codexx on 8chan.moe has kindly offered to host me on /t/ over there. As I have never been a comfortable BO, this is a much better solution for me as the primary place to talk Anon about hydrus.

I will be maintaining a Hydrus Network General, the first instance is here: https://8chan.moe/t/res/2219.html

The links in the release tomorrow will reflect this. Endchan will remain as a bunker.

>>15141

If you happen to see this in the next 18 hours or so, I think you want this: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Downloaders/Gelbooru

test

>>11542

join

>>15029

I am sorry for the trouble here. The blacklist should work on the tags parsed from the page. So if you tell it to stop 'guro', then any page with that tag should get the 'ignored' status with an appropriate note in a downloader using those 'tag import options'.

One user has been telling me about this not working for him, but I couldn't figure it out. Can you give a search example on danbooru and a blacklisted tag that aren't applying for you?

>>15039

I am sorry, I do not know what is going on here. Sounds like it could be an ssl issue or similar, maybe if your OS is old, although I'd guess that sort of thing would be more likely right on Jan 1st 2021 as some certificate timeout ticked over.

If you can paste the whole error, I can have a closer look. If it is moaning about ssl certificates, one way around it is to uncheck the 'verify regular https traffic' option under options->connection.

>>15071

>>15089

If you mean 8chan.moe, I believe they did some direct linking referral trickery on their spicier boards to make sure direct links from external sites don't work. There may be a solution here with some forced hydrus referral urls on the URL Classes, but I do not know the details and have not looked into it myself. I think you'll have to browse these boards manually for now.

>>15074

Any site that doesn't operate like a normal booru is tricky. Sometimes you just have to babysit them.

I assume you are downloading through nitter, right? I think the twitter API that nitter hooks into does not deliver tweets beyond about 3,000 tweets into the past, so if you are hitting that limit somehow, or if nitter has its own limit, that may explain it.

I have been getting reports recently that nitter has been giving '429 - server rources limited' or similar recently as well. This may be related, I am not sure.

>>15088

I am not sure if the existing pixiv login script works right now, it has always been temperamental. If not, I think you'll have to use Hydrus Companion or similar to log in by copying your browser cookies to hydrus. Then pixiv will think your hydrus is you and it'll have access to private stuff.

>>15095

Support here is not yet great. If you are importing from hard drive, they should import in filename order. If you are importing from a site, it will come in the order the site's parser pulls the URLs, which is usually good, sometimes reversed. The main problem is hydrus currently has integer timestamp for import time, so if two files are imported in the same second, they will sort randomly when you sort by import time. If this is what is hitting you, I apologise. At some point there will be a conversion to sub-second times, although a retroactive fix would likely be impossible.

I am hoping the eventual file alternates work will provide a nicer system than page tags. I agree they are clunky.

>>15115

>>15116

This is some developer bullshit unfortunately. You don't need mpv, but 'libmpv1'. You should be able to get it with apt.

This thread is now archived at https://archive.is/0WoQk . I will not post here any more, as the board will be deleted next Wednesday. Please move to the Endchan thread at https://endchan.org/hydrus/res/10.html , thank you!

The plan to move to Endchan only has changed!

Codexx on 8chan.moe has kindly offered to host me on /t/ over there. As I have never been a comfortable BO, this is a much better solution for me as the primary place to talk Anon about hydrus.

I will be maintaining a Hydrus Network General, the first instance is here: https://8chan.moe/t/res/2219.html

The links in the release tomorrow will reflect this. Endchan will remain as a bunker.

>>13543

Once a month, I make an encrypted 7zip archive of my entire database while Hydrus is closed. Then I slap it on a blu-ray and store it in an undisclosed safe location. This gives me an incentive to go through my inbox and keep my archive clean, because splitting archives to put them on separate disks is a pain in the ass and waste of money.

I also have an automatic backup that does basically hourly backups, but it's to a drive that's always connected to my PC so I only consider it slight fuck-up protection.

>>15078

If you did it on the backup, I don't see why it would cause problems. It wouldn't be encrypted though, unless the device you're syncing to stores it on an encrypted partition.

>>15081

>Then I slap it on a blu-ray

Good luck when it reaches a bigger size.

>t wouldn't be encrypted though, unless the device you're syncing to stores it on an encrypted partition.

Always encrypt your backups in some way. You could even just rSync it to another part of your system locally, then encrypt it alongside anything else you want to backup, then have that encrypted archive be transferred remotely.

I'm sure there's other easy ways, I just haven't dealt with backups much. I just backup to an encrypted drive every now and then locally.

>what other methods do you use to backup your Hydrus files remotely?

run a hydrus client on a server and set it as a remote file repository, then you can tell your local instance to upload to it.

>>15078

I'm the anon from the OP, the main issue with this approach is that without deduplication of some kind, you'll end up having to send your (rather sizable) database file with every update all over again every time you rsync with the remote folder. As I mentioned last year, storage space isn't so much a concern as it is upload speed, as even with a mild recent improvement they still top around 2Mbps.

I had to stop using Duplicati as it was incredibly fragile, I'm now using borg and rclone to send my local repo to a Backblaze B2 bucket (even cheaper than my previous option). Borg handles the deduplication and encryption.

>>15120

With something like borgbackup/rsync you can do it in a way that it only updates changed files I believe.

So you're not backing up the entire db over and over, only changes.

Sankaku is now hiding lolis. Is there some way to get around this?

I'm not sure if GUGs can make these, but anyone have a module for setting up Youtube subscriptions?

>>14782

They're not hiding lolis. I don't understand why I keep hearing this. Did you check the mature content option in settings and clear your account blacklist? Do you have an account in the first place?

Can someone help me understand what parsing scripts are for, and how to use them?

Are they to improve the amount of tags that are found for images? Like a reverse search?

>>15020

The site isn't even working yet so why would there be a working downloader?

>>15022

What do you mean? The site is up and there is content on it. I only found out about it very recently and I don't know about the situation except for what I can infer from anons' posts.

>>15022

It was broken for a while but it seems to be working now.

>>15020

The site has changed the URL schemes for posts since that was put on Cuddlebear's repo. I would wait until the site is more stable and/or the API is finished before trying to find/make a downloader.

>>15024

I see, that makes sense. Thanks.

I made that WIP kemono parser, its specifically for kemono.party/files and was made when the actual kemono frontend was non-functional. It was meant to do the bare minimum of scraping their bulk file CDN but I don't even think they use the same URLs for file downloads anymore.

>>15075

The main benefit is everything is shared in one client database. As well as basic simplicities like only one client to backup or have options for, you will also be able to search in more than one local file service at once, store files in more than one services at once (and only need one copy of the file), only have to sync to the PTR or do other large tag/db maintenance for one database. It may open up some interesting new processing workflows as well.

>>15087

Then you should provide a way to import one database into another without losing/resetting any metadata like "time imported" for example. Or else this feature is completely useless to me because I don't want to lose that from my multiple databases. I'm not going to create a new database and spend hours or days setting everything up, then importing all the files from my old databases and lose all of their metadata in the process.

>>15090

Yes I agree, I will have some sort of client-to-client transfer as part of this, so users who have been using multiple clients will be able to merge neatly and preserve time and archive metadata. Around that time I'll likely have to introduce rating export/import too, and perhaps URLs as well.

The good news is now we have the Client API, this should be neatly possible as a direct client-to-client connection.

>>15092

Thank you.

I had a good week. I was not able to fit much interesting fun stuff in, but I fixed the slow tag autocomplete search in the manage tags dialog, sped up tag processing, fixed some 'ghost' tag bugs, and reduced some wasted CPU in the network engine. Overall, the client should be a bit neater and faster in 425.

The release should be as normal tomorrow.

>>15077

It already does tag stuff based on the file, see the system tags. There is 'system:has audio' and 'system:no audio', and then there's system:filetype which lets you choose either the type (image, video, etc) or exact MIME (i.e. mp4) of files to seach.

There are plenty of others too, like dimensions, time imported, hashes, filesize, etc.

>>15082

>It already does tag stuff based on the file, see the system tags.

I was suggesting it automatically applies these tags when it detects them.

For example if you import a file that has audio, it knows that so it just applies the tag.

Unless I'm missing something, it does not seem to apply automatically.

>>15084

Is that not what it does? Unless you mean applying a non-system tag like "video with audio" or something, it does do that automatically.

>>15099

Just tried adding a new url with a bunch of videos to the watch, some have audio some don't.

None of them have been autotagged with any tags, except for the few that have a hash matched.

So it does not seem as if it is auto-applying tags for stuff that are videos, and stuff that has audio.

>>15101

do a search for "system:has audio" you absolute retard

>>15036

You can set tag import options by url class in network>downloaders>manage default tag import options

Unless I'm missing something, this seems to be missing:

I've been trying to figure out, is there no way to have stuff autotag per watched threads? Didn't see anything in the docs about this.

I know you can set tags to be automatically applied on the watcher, but can you not do it PER watched thread?

So if I add a watcher url, and want all files that get fetched from that to be auto-tagged with series:pokemon, then I add a second watcher url, and want all posts from that to only be tagged (automatically) with series:kingdom_hearts is there no way to do this?

If I am correct, I would love to see this. It's a complete PITA to have to do it manually for each thread for basic tags.

I'd also suggest this be done anywhere else it is not possible on the other download tabs, although I rarely use anything besides the watcher so not sure if they are missing this.

My other suggestions are:

Under manage tags, the recently added tags only save about 22, while there's an entire vertical space for it to save around 60. This means you constantly have to keep doing a search as you go through tags. Would be nice for this limit to be bumped up a lot higher.

And lastly, the ability to shift click items, or shift hold and drag to drag highlight items. When dealing with lots of thumbnails, this would save so much time. Currently it seems it's only possible to Ctrl + A to all, or ctrl click each individually.

Thanks

>>15091

If you set tag import options in the watcher page it is applied to the threads that are queried while those are set, so you could change it to have your tags for that thread then change it again for another thread.

You can change the amount of recent tags in options, options>tag suggestions>recent.

Shift clicking to select a thumbnail range should work already.

>>15094

>If you set tag import options in the watcher page it is applied to the threads that are queried while those are set

It would be very nice to have a prompt or option area to add a default tag if the user wishes. Then it just resets after the page is submitted. The user could also right click the watch and edit the tags and it would easily update them for all the previously grabbed files in that watch and any newly tagged files. The way you explain is a bit inefficient imo.

>You can change the amount of recent tags in options

Ah, a bit hidden, thank you.

>Shift clicking to select a thumbnail range

I was behind like 2 Hydrus updates, but just updated and that works now. Maybe it was a bug then. Works now though. Thanks.

Does Hydrus have any support for multiple audio/video streams in one file?

Say I have 4 videos with the same audio, just slightly different video (say different levels of clothing). You can use ffmpeg or what have you to combine them into one file with multiple video streams and one audio stream, which mpv can play (at least externally, cycling with the _ keybind).

You could do similar things for multiple videos which only differ in audio, say language (keybind #). You could even have both multiple video and audio streams in one file, if you really wanted.

I could have each video+audio stored separately in Hydrus, but to me they're the same video, just different variants. Being able to combine them would help reduce clutter in the search and lets you be sure just from the thumbnail that the variant you want is available.

I think it would be cool if Hydrus could switch video/audio streams without having to launch in an external program. Don't know the technical details on whether a keybind or right click menu or both would work, though.

I have an extremely small amount of 3D in hydrus, I tag it with "series:real life". I have to manually tag it, but since I can count the amount of 3D I have on one hand it's not too much of a pain.

I manage my anime, manga, and books outside of Hydrus, but I keep doujins in it. I tag doujins differently from images, kind of like how exhentai has different tags than boorus do.

One of the best parts of Hydrus is the archive/delete filter. I regularly go through my archive and trim the fat, there's a ton of images that look good the first time but that you realize you would never actually jerk it to. If you have tons of files that cannot be automatically tagged, I recommend doing so to make it easier on you.

>>15055

The nice thing about it vs tags is it will be a cleaner compartmentalisation, particularly for the nsfw/sfw/'my sister is looking over my shoulder' problem. When a file service only stores sfw files, then, as an example, you can type whatever into a tag autocomplete and you won't get undesired results. This would be true for many other situations where recommendations and 'similar to' and random 'system:limit' searches may provide surprises or just a wave of false positive results on your little art reference collection.

For vs multiple clients, this allows you to sync with the PTR (or have shared tags of any sort) on only one client, which saves a ton of space and CPU. All other stuff like backup routines and bandwidth options etc… are shared too, and you can have files on multiple domains as well without needing more space.

I'll be interested to see what new workflows emerge as well. Maybe in a couple of years we'll be putting all 'inbox' files on their own file service or something.

>>15056

That google doc is really interesting. Seems like JPEGXL with the right parameters is great across the board.

The issue of dupes and their metadata propagating to the PTR or other shared resources is a worsening and unavoidable problem. It has only worsened as the big CDNs have yeah started optimising content. I am not sure if there really is a nice solution, and it may just be an architectural problem with the 'tag repository' idea. Perhaps a future version of the repo will also include hash-dupe pairs, but I suspect that would produce its own set of management problems as it would require user agreement on dupes, and not all dupe merges want all tags copied.

My secret dream is that the huge 'corpus' of tag metadata we have built up with the PTR so far lets us train ML in the next five years and the scope of the PTR is greatly reduced. Rather than sharing 'file x has tag y', I'd love to share 'tag x looks like y visual data', and then clients will run ML on their own files for common/simple tags. Whether the tech can ultimately do this to a high enough confidence to apply a certain subset of tags automatically isn't obvious yet, but the feedback from hydrus-dd, which uses the DeepDanbooru ML, is promising.

I had a good work week. I did a variety of small fixes and quality of life improvements, and I finished redesigning the part of the database that does wildcard tag searches. Autocomplete lookups and file searches that rely on complicated tags are running faster across the board, and the old design's sudden lag spikes (e.g. with namespace:*anything* searches) are entirely eliminated.

The release should be as normal tomorrow. There will be some database work on update. I will have a better idea tomorrow, but I estimate users who sync to the PTR with an SSD can expect it to take 5-15 minutes.

>>15059

I did already try that, however it seems to only accept individual galleries (and at 'browsing' quality, at that). If I had to seek out each gallery and check for new ones myself, it's not really much of an upgrade.

I expect to get something downloading via the archiver would be a pain if for no other reason that there's multiple pages and you would need to check that the user has enough gp/currency (people might also get upset if gp gets wasted). Not to mention the archiver downloader will be in a compressed filed that will need to be extracted.

Still, in principal it should be possible. I and I'm sure other people can help answer questions about exhentai. It would be a big feature IMO.

>>15062

>Seems like JPEGXL with the right parameters is great across the board.

Exactly. Matches my own tests on the current Jpeg XL reference. Exciting times. ImageMagick got jxl in 7.0.10-54

>My secret dream is that the huge 'corpus' of tag metadata we have built up with the PTR so far lets us train ML in the next five years and the scope of the PTR is greatly reduced.

Yea, that would be nice. Still might leave artist name, url and so on.

Plus just imagine how many more images people might produce via AI crunching / pre-producing images. I wonder if the db of known images (plus known tags - future training data?) will be obsolete all that soon.

It looks like 'namespace:*' searches, despite now working, may be running super slow or doing too much database work, in IRL PTR-syncing situations. I will look into it next week! If you are hit by this, please turn them off for now under tags->manage tag display and search.

>>15014

Two issues I've noticed network-wise so far.

Issue 1 Disclaimer: no idea if this is a regression - first time I've tried this on any version.

On v421 I'm running into an error when querying for IDs with specific filetypes through the API.

I'm trying to get a list of all files of type webm, wav, etc. but I can't figure out the proper way to do that in the "get_files/search_files" endpoint. Stuff like "meta:webm" works in both the client and API but it's missing a lot of files and seems to be manually assigned; on the other hand, using the query "system: filetype is foo" in the client seems to pull directly from the local DB of files and doesn't miss any but when I hit the API with "?tags=%5B%22system%3A+filetype+is+webm%22%5D" I get no results.

I'm not sure if this is user error, by design/wontfix, or you just haven't gotten around to handling/exposing it yet. Is there currently a way to filter on the filetype over all local files? If not, could you add that to the suggestions you're polling on?

Issue 2 Disclaimer: I have a janky workaround for this

I'm working on a browser-based frontend for Hydrus, and one of the problems I've hit is that safari for iOS (and maybe Mac too, I don't have one to test) won't download media like mp4 files because it tries to sample the file with a range download, and if the remote server doesn't support byte ranges then it bails and the file doesn't load. My current workaround is an nginx reverse proxy with the proxy_force_ranges option set, but it's kind of janky and I can't really ask people to run an nginx server to use the webapp if I ever get it to a releasable state. I'm not sure what library you're using to serve the API but if there's an easy way to enable byte ranges that would be great, and if not no worries, it'll be quite a while before it'll matter.

I had a good week making some small fixes and improvements to finish up the year. Autocomplete works a bit faster, and some quality of life is improved.

The release should be as normal tomorrow. It will have the 'next big job' poll and be the last release for the year.

>>15016

Hey, unfortunately system predicates do not work on the Client API yet. They aren't text-based in the actual client, so I can't just pull text in the query to accept them (yet). They are top of the list to add for Client API features.

Here's the master job for Client API by the way: https://github.com/hydrusnetwork/hydrus/issues/656

Thank you for the note about file ranges. I have not run into this need yet, so I will research how to do it properly. I use twisted as my main server engine, so I know this is possible, I just think I have to link some things together.