/hydrus/ - Hydrus Network

Is there a good place to discuss tagging standards for the PTR? There has been plenty of talk on what tags and namespaces we should use, and I think there should be more. Maybe it's worth its own thread.

>>12638

Siblings are useful, but being able to right-click a tag in the manage tags window and select "replace tag with…" which lets you type in a tag to replace it with would be great. This would only affect the selected files that have that tag. I might still want to keep that tag on other files I haven't selected, so a sibling isn't helpful. I could manually remove the tag from the selection, but then I can only add the new tag to all of them.

i'm working on tagging approx. 18k files right now with rating tags, i.e. rating:explicit, rating:questionable, rating:safe. i'm trying to figure out if there's an effective way to do this automatically.

I've set up obvious tags as parents for rating:explicit, so explicit is applied automatically with practically no problem. is there a way to say to the client "apply tag rating;questionable IF file does not have rating:explicit AND file has tags (tag1 OR tag2 OR tag3)"

the reason I want to do this is because if i set a parent relationship for let's say tag breasts->rating:questionable, then i'm going to end up with a load of files with both rating:explicit and rating:questionable since the parent relationship does not check for the rating:explicit tag.

also, mr dev man, i love your program. thanks a ton

>>12632

>sounds like a very subjective description of namespaces to me.

Tagging is entirely subjective to begin with.

Programmatically, namespaces are used for disambiguation to prevent naming collisions. What this means for tagging is that in practice you should have namespace_A:same_character_name and namespace_B:same_character_name to disambiguate two characters. People do shit like "character:Yuno (Mirai Nikki)" and "character:Yuno (Black Clover)", cluttering the tags with two variants of Yuno instead of providing a namespace for each of them (eg: "black clover:yuno" "mirai nikki:yuno"). This is probably done to avoid cluttering namespaces with is often its own namespace (series: or copyright:). In practice, almost nobody uses namespaces solely for disambiguation. They use it for meta information like "series:" or "artist:".

>not sure what the difference to you is between having two namespaced tags "hair:short","hair:long" and having two regular tags "short hair","long hair". Either way you have two tags.

I explained the difference. Information about the image (meta) vs information that describes what the image is (descriptors). This is also how every booru I know of works and how I choose to tag. Boorus will use the tags "black panties" and "black dress" and not namespace them under "black:" like "black:panties" and "black:dress". Mixing descriptors with meta information is messy as many things can be swapped with their namespace and still make sense. For example, which is more valid: "dress:black"/"dress color:black" or "black:dress"? Why would you namespace one and not the other? Would you use both namespaces?

"hair:ponytail hair:ribbon hair:black hair:short" is clutter and even if you break each "hair:" down into more specialized namespaces like "hairstyle, hairaccessory, haircolor, hairlength" you only worsen the clutter.

Compare: "ponytail ribbon black_hair short_hair" vs "hair_style:ponytail hair_accessory:ribbon hair_color:black hairlength:short"

>>12647

Character: Yuno Gasai is different to Character: Yuno, the Meal-Saving Prince. Clutter is a buzzword to excuse minute and less important tags from being tagged on images. A dress being black doesn't mean it's not in the style of Dior's iconic Little Black Dress, and so a Dress:Black and Black:Dress can be applied to the same image for the same garment and mean two completely different things. If a user believes that Hairstyle:, HairLength: or HairColour: tags are too specific, they can simply not use them. Hydrus allows certain namespaces that the user deems unimportant or uninteresting to be left out when viewing a certain tag or groups of tags. It's called Censoring. Hydrus does NOT allow for tags to appear on images where there are none. Therefore, it is the duty of any user of the PTR to overtag images and allow the user to pick and choose which tags or namespaces they see fit to read or discard according to their preferences.

All of this relies on people actually using namespaces, though, which people still don't.

>>12650

>Character: Yuno Gasai is different to Character: Yuno, the Meal-Saving Prince.

No shit, that's the point of differentiating them. I excluded last names for the sake of example but had a feeling you'd be insufferable enough to ignore the point. So here's a more solid example of two people with the same first and family names.

Michelle Williams (All the Money in the World) and Michelle Williams (Destiny's Child)

Both would be tagged "Michelle Williams". This is what namespaces are *meant* to resolve and where the name "namespace" comes from as a programming term. You'd provide a namespace for each Michelle Williams as a way to disambiguate between two otherwise identical names.

>Clutter is a buzzword to excuse minute and less important tags from being tagged on images.

If you noticed, I didn't remove any "less important tags". I simply didn't give them redundant namespaces that provide no additional value over a plain tag and only serve to add clutter the search terminology.

>Therefore, it is the duty of any user of the PTR to overtag images and allow the user to pick and choose which tags or namespaces they see fit to read or discard according to their preferences.

I don't use the PTR and am not speaking about public choose-your-own-tags-and-censor-stupid-tags group tagging. Most people are retarded when it comes to tagging things which is why once boorus grow past a point their tag quality goes downhill fast with very poorly or even misapplied tags.

>>12652

>which is more valid: "dress:black"/"dress color:black" or "black:dress"?

If I wanted to namespace this tag (I don't), "dress:black" makes the most sense. "dress color" is unnecessarily specific, leading to more namespaces than needed.

However, if I cared for searching by/adding/removing dress colours often, then I could more easily find all dress colours in a long list of tags if there was a "dress" namespace. This applies to a list of tags on an image, as well as search results for tags having entered only part of the query (e.g. just the namespace, or just the colour).

This philosophy works well with Hydrus, and does not increase the number of tags. More namespaces are introduced, but only for what you are interested in.

The only problem is that one has to add local siblings for all tags they wish to namespace, and possibly add the reverse siblings to the PTR if they want to easily mass upload tags.

>had a feeling you'd be insufferable

Now, now

>>12658

>If I wanted to namespace this tag (I don't), "dress:black" makes the most sense. "dress color" is unnecessarily specific, leading to more namespaces than needed.

Not saying you wanted to use these namespaces - but I was pointing out that black:dress and dress:black are both equally valid ways of tagging the same thing using namespaces. "dress color:black" was just used as a more verbose "dress:black", the comparison was more between "dress:black" and "black:dress". Further examples of these namespaces, you might have "dress:wedding" and "black:shoes".

>However, if I cared for searching by/adding/removing dress colours often, then I could more easily find all dress colours in a long list of tags if there was a "dress" namespace.

I don't believe this is a common enough scenario or a significant enough increase in quality of life from scanning an alphabetical listing of tags for "color_dress". It takes maybe an additional second, if even that, to see if an image is tagged "red_dress blue_dress black_dress" instead of "dress:black dress:blue dress:red" even on my images with ~70 tags. It provides that small amount of QOL with the tradeoff of either increasing the complexity of the tagging system (rules about what belongs in the "dress:" namespace) or increased verbosity or redundancy when searching.

Complexity: "hair:long black_hair ponytail ribbon (Complexity is defined as having to clarify "Which values belongs in the 'hair:' namespace?")

Redundancy/Clutter: "hair:long hair:black hair:ponytail hair:ribbon" (Reduced complexity, "All hair related values belong in the hair namespace")

Verbosity: "hairlength:long haircolor:black hairstyle:ponytail accessory:ribbon" (Removed redundancy and reduced complexity by being more verbose)

No complexity, no redundancy, no verbosity: "long_hair black_hair ponytail ribbon"

>…as well as search results for tags having entered only part of the query (e.g. just the namespace, or just the colour).

"hair:" returns "hair:long" and "hair:black" (and all the other length/color in the hair: namespace) and "*_hair" returns "long_hair" and "black_hair" (and all the other length/colors tagged as adjective_hair).

>>12643

>Maybe it's worth its own thread.

Sounds good to me, it's a bit messy to mix it in with Q&A

How do you get Hydrus to download Explicit tags on rule34hentai where you need an account to see? The Hydrus login doesn't do anything and Hydrus still skips these when downloading from that site.

How do do you undo an imported downloader from an png? I tried importing a downloader from an png and I don't really like how its getting tags so I want to go back to its default settings.

>>12667

Is namespaces the only way to sort tags?

Is it possible to add them to sorting list to have them presented in a way that looks logical to humans?

If it was possible and if coloring a tag was moved from namespaces to these list, I don't think we would need so many namespace wich essentally multiply the tag in the database.

dog and species dog both exist.

This may be stupid, though, I'm no programmer.

Has the Linux version of Hydrus gotten any more stable? How do I migrate my DB from windon't to linux?

Will it ever be as stable, or will this always be a primarily windows-focused application?

>>12673

Im running it from source, so far only once hydrus crashed.

>How do I migrate my DB from windon't to linux?

Same as in Windows.

>Will it ever be as stable, or will this always be a primarily windows-focused application?

idk seems pretty stable to me.

My database sees only 21 images out of few 1000. Even though they are there. Everything else is considered orphans. Am I fucked?

>>12674

That is nice to hear, last time I tried it on ubuntu it would crash constantly.

Maybe there was an error in my setup.

I'll try it soon!

Do you think there is any way to easily remove the more smutty compnents of the program (explicit references to hentaihaven/gelbooru etc.) becuase I want to use it in a way where it'll be relevant to cataloguing photos of family and friends? (i.e.: father, mother, brothers will see the program).

How come the installer fails everytime I try to download it?

>>12671

I used that example because the tag "dog" is aliased to "species:dog" in the ptr.

I too would prefer it not to be a defacto alias.

I wager you either don't use the ptr or found a way to blacklist alias locally?

I wonder if the way we use namespace is necessitated by boorus compatibility rather than being the most efficient way to sort tags.

Using a list system would however requires more user involvement though, which doesn't seems a good idea for such numbers of files.

In the same vein I wonder if affix and suffix are possible in lieu of those qualifier, by which I means linking up to tags togethers.

"character name" is affixed to "series it come from".

It's 100% not doable now but may be technically possible, perhaps.

I'm getting a filenotfound error even though the file exists on my system, not sure why this is happening? db diagnostics say nothing is wrong

I created the db on MacOS and have since imported to Windows, exact same db works on mac but fails on windows… did I miss something during my migration? (db was fully portable)

>>12688

I don't use the PTR because people tag like retards, so you're right. :)

>I wonder if the way we use namespace is necessitated by boorus compatibility rather than being the most efficient way to sort tags

As I explained somewhere, namespaces are used to differentiate two things that have the same name. It comes from programming, especially object orientated programming, where you'll have many objects that may share the same name but aren't the same thing. For example "second" and "second". "time:second" and "position:second" provides disambiguation. ( See: https://stackoverflow.com/questions/4211827/why-and-how-should-i-use-namespaces-in-c )

People don't use namespaces properly and end up with "(clarifications)" in their tags which, in my opinion, is quite ugly. Because disambiguation is rarely needed, it's boorus often use it for *meta* information like who drew the image or what the image is a drawing of rather than what it contains.

>"character name" is affixed to "series it come from".

This is already double by making use of child-parent relationships and if you don't tag like a retard. Unless I'm misunderstanding you. :)

I'm still kinda new to all this, but im wondering why theres a image limit on the twitter downloader? It only seems to stop at a point. Also will it ever be possible to download gifs and videos off from twitter?

Has anyone ever noticed booru subscriptions consistently failing to pull certain tags? I am suspecting that any tags which are replaced by a local sibling are not getting added to the image at all.

For me, this is a Sankaku subscription. I'll look into it a bit more.

Where can I find other downloaders? Or are all of them already installed?

>>12684

>Do you think there is any way to easily remove the more smutty compnents of the program

At the very least you can hide them from view from network -> downloaders -> manage downloader and url display. I think you may be able to entirely remove references to them through network -> downloader definitions and clicking on each submenu and deleting everything in there.

>>12731

>If so, can there be changes where copying "imported" files to its own directory is entirely optional?

Yes, but that would fuck with the entire program. Hydrus depends on files being in it's own folders in order for half of its features to work.

There might be some workaround in the future but no plans yet AFAIK.

>>12732

The only thing I really want is some aggregate, search, catalog, tagging features. Should I be looking for another (simpler) program?

Not quite sure how to ask this, but is there a way to purge all PTR data from my db?

>>12684

>That is nice to hear, last time I tried it on ubuntu it would crash constantly

Sorry for late reply, DO NOT USE THE PORTABLE VERSION, RUN IT FROM SOURCE.

I could swear I asked this already but maybe it didn't send as I can't find it.

Following scenario:

One PC acts as a local server, nine others use that local connection to access files on the server. Would it be possible to put the DB on the server and access it from all other PCs?

What happens if several clients try to change things?

>>12734

have you deleted the ptr from the services menu?

How should I tag variations/revisions of a illustration?

I found using a combination of page and chapter (what should the chapter be even called?) with collections produces the desired effect, but it seems like a terrible solution.

>>12740

Just going to have to wait it out. The public tag repo is unaccessable until the bandwidth limit resets at start of next month. In the meantime check out https://cuddlebear92.github.io/Quicksync/#two and use the services -> import repository update files menu to get a head start on updates.

>>12739

That was the first thing I tried, but it didn't seem to do anything. I somehow got client.mappings.db from 15GB to 3.5MB (I think I exported my local tags, deleted client.mappings.db, and re-imported my tags), but client.master.db only dropped from 4GB to 3.5GB after vacuuming

>>12740

Use >>1914 or do the individual file lookup. The problem with that is if the hash doesn't match then you'll get nothing. That's why I run everything through IQDB and use the script to collect it all for import

>>12737

>What happens if several clients try to change things?

I'm not sure which thread it was in (it was sometime within the last month or two) but I'm pretty sure Hydrus Dev said not to do this. Something about SQL not being able to handle multiple programs trying to lock/unlock the db.

>>12644

Nice idea, thank you. I know what you are getting at with the current workflow not being able to do this for a multi-file selection. I think I can do this, but I can't promise it super quickly.

>>12646

Great, I am glad you like it!

Best way to do that for now is with a search page that you hit F5, ctrl+a, F3, ctrl+v, enter, enter, ctrl+r on every week. A search like:

[ (semi-lewd tags), -rating:explicit, -rating:questionable ]

Then hit F5 to refresh and see all the files that apply. ctrl+a, f3 to bring up manage tags. ctrl+v, enter to add 'rating:questionable', then enter to close manage tags and ctrl+r to remove all selected files from view.

There isn't a way to do kind of 'logical' tag siblings/parents that rely on more complicated tag states and implications. I have thought about it several times in a 'wouldn't it be nice to add tag x if both tag y and z were present' way, but that is a lot of power and I can see how it could go extremely wrong if I did not wrap it behind two advanced modes. So I am a little wary of it atm and am ok that human eyes go over complicated jobs like this for now.

rating is a very subjective namespace. I agree that explicit is going to be easier to set up than questionable or safe.

>>12666

I have heard the R34H login script can be finicky, just failing sometimes. If you are absolutely certain you have the credentials set up correct in manage logins dialog, and that the login script is active, try hitting the 'reset login' and 'do login now' buttons on the same dialog, just to retry.

>>12667

Hit the manage dialogs under network->downloader definitions menu and delete the new objects. You are probably talking one GUG, a couple of url classes, and a couple of parsers. If they overwrote old objects, those dialogs should have 'load from default' buttons or similar. Once you are all set up, hit the manage url class links dialog and make sure the 'url classes' you want are connected to correct-sounding parsers. You can click the 'try to link…' button on that dialog to try to auto-fix the links here.

>>12668

I would like future versions of the tagging system to support metadata about tags to permit custom sort orders. Future tag siblings will allow you to better control your various dog/species:dog preferences locally just to your client.

>>12673

If I could wave a magic wand, I'd love for the Linux version to be more stable. I'll be putting some time into improving multiplat stability in about three or four months. Unfortunately I am a Windows guy so just have less experience and time spent on different Linux flavours and window managers.

The stability problems on Linux are mostly to do with my shit UI code interacting with stricter/different window managers that want things in slightly more proper ways. I'll keep pushing on the problem, but I can't promise excellence for all systems in quick time. That said, some Linux users experience no problems at all with source or the built release. I recommend you try it, see if your situation is good enough for you, and plan to be ok with staying on Windows or whatever.

To migrate your db, the basic rule is just to install on Linux and then copy your install_dir/db folder across. Don't do this if you are uncertain–feel free to ask me for more details if you need help. Make a backup beforehand. Background reading here:

https://hydrusnetwork.github.io/hydrus/help/database_migration.html

>>12676

This sounds like a serious problem. Please check the 'help my db is broke.txt' document in install_dir/db. If the files are legit still in your client_files folder, but just seem to have disappeared from your client's records, there are ways to recover. I am happy to help with all of this, but first you need to make sure your hard drive is healthy, which that document will walk you through. Feel free to send me an email if you want to work more one-on-one with this.

>>12684

>>12727

Yeah, delete the stuff from downloader definitions. Bear in mind this is imageboard-centric software. Anyone who googles 'hydrus network' is going to end up here pretty quick, so I recommend you sit in the cockpit and just drive some slideshows if you don't want to be asked questions about the INTERNET HATE MACHINE.

>>12687

Have you got an anti-virus that doesn't like something in it? This has happened a couple of times before. The 'upnpc' executable file in install_dir/bin are common culprits for false positive detections, and I know some scanners can see inside the archive of the installer exe.

>>12689

Can you say a bit more about your migration situation? If you open up database->db migration, does everything appear to be in the right place there? What's the first two hexadecimal characters of the file's hash, and is it in the right 'fXX' subfolder for that? If the client gives you path details in those FileNotFound errors, do they line up with what you are looking at, or are they old/incorrect to what you expect?

Also, if this is a video, is the actual error about not being able to find ffmpeg, which hydrus uses to render video? Do any videos work? What is ffmpeg's apparent version under help->about?

>>12696

There are two limits with twitter–one is a ~2,800 tweet limit caused by their API, and another is if the user has a series of 20 or so tweets without media content, the downloader bails early.

Twitter videos is tough. twitter deliver videos as streamed mp4s, so no easy URLs for hydrus to latch on to. I know one user is working on this problem. Apparently twitter can give you direct vid URLs from the API, but it takes an API key.

>>12700

Can you check your options->tags for 'Apply all siblings to all services'? If it is checked, would unchecking it help? I expect to extend this system in future so you can control which services can apply to which others, but for now it is all or nothing.

>>12721

Lots here:

https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System

Check the 'all-in-ones' folder for easy-import pngs. Bear in mind the quality of these downloaders varies a lot. Test them a bit to make sure you only get the tags you want, and don't import a thousand all at once or you'll fuck up your UI with gigantic lists of shit you don't care about.

>>12731

Here are my current feelings on 'external file' management:

https://hydrusnetwork.github.io/hydrus/help/faq.html#external_files

It is technically possible, and if it were a simpler problem all around I'd do it, but I just don't think I can put the maintenance time into this sort of system. If you only want easy-external access to a few thousand files, you may be able to cobble together what you want with an 'export folder' that maintains a synced folder of a search and gives the files nice filenames based on tags.

>>12733

ACDSee is good. If you only want to catalogue a few thousand files and external management is critical for you, that might be a better fit.

>>12734

>>12739

>>12744

Unfortunately, my current 'clear orphans' tech doesn't do master definition recycling. I plan to add this, so it will go down eventually, but we just aren't there yet.

>>12737

>>12750

Yeah, there aren't great solutions yet for a single client accessed from multiple locations. I do not recommend you run a client.db from a network location, as the locks the network drive provides are usually fake and not good enough for what SQLite needs to be safe.

Running multiple clients at once from multiple network locations will cause your db damage.

The Client API is likely to open things up here. I know users who are working on phone/tablet/web browser wrappers that present you booru-like interfaces.

>>12740

Yeah, sorry, PTR bandwidth is tight atm. It is now free again but will likely block again for the last week of the month. I have a plan to fix this in 2-3 months. You can just leave it to sync and it will catch up when it can.

>>12809

I don't know. Most boorus share the same software, so if you know Gelbooru 0.2.0 supports it in some way, any booru that runs that should be good. Doing a search for '(booru) advanced search' or '(booru) cheatsheet' can usually get you to a nice page like this:

https://gelbooru.com/index.php?page=help&topic=cheatsheet

https://danbooru.donmai.us/wiki_pages/43049

https://e621.net/help/show/cheatsheet

https://konachan.com/help/cheatsheet

So looks like gelb 0.2.5, danb, e621, and moebooru are good with a flat 'md5:hash' search. I'd bet Gelb 0.2.0 can do it. I am not sure about Gelb 0.1.1.

>>12809

Check out the last few posts from the Hydrus Scripts thread:

>>12604

>>12631

Does gallery downloaders check for newly added posts periodically or do I have to create a new query to fetch new posts?

>>12822

The downloader page just does one job once. If you were to manually create a search for the same query you put in before, it would start a new queue starting at the current new beginning. Obviously you do not want to do this yourself every week or so, so if you want to have a downloader that periodically re-checks and keeps up with a particular query for a site, check out 'subscriptions':

https://hydrusnetwork.github.io/hydrus/help/getting_started_subscriptions.html

PROTIP: Don't go fucking nuts with this. Start slow and figure out a workflow that won't nuke your available bandwidth and hard drive space.

>>12829

Forgot to mention, It's a danbooru/gelbooru subscription.

>>12814

Nice, thanks.

>>12816

I already use all of those. I want more tags.

>>12829

Ig means ignored, so 27Ig means 27 ignored URLs.

>>12833

Got it, thanks.

>>12834

Ignored tends to be either 404 or tag blacklisting (say, the file had 'scat', and that is in your blacklist). If you zoom into the individual sub query and order its big file import list by status, you can review the ignored reason in the 'notes' column.

>>12835

Type in 'character:' in the autocomplete entry and wait a moment, and a new special tag should appear in the dropdown list called 'character:*anything*'. This is a special tag you cannot type in manually that uses cleverer db code to speed up the namespace search.

>>12837

Nice! Any reason why "character:" doesn't default to this?

>>12839

That's a good idea, thank you. I think it is probably reasonable to say that 'text:' is not a tag people will typically want to attach to anything and search for regularly, so I'll swap in the 'text:*anything*' version whenever this is added.

>>12803

database -> db migration said everything was in the right place, all the files existed on my system and were in the correct subfolders. Client wouldn't cite a path in the errors, not sure why. ffmpeg is also installed locally + system wide.

I fixed the issue by importing a new file; not sure what this changes but after that I didn't get any more errors. not really sure how to reproduce this, sorry.

for posterity: what i did was move a fully portable ~45gb database from mac to windows, using resilio sync (home version)

Is there a way to search for files that have mutiple tags in the same namespace?

Main scenario i am thinking is searching for files that have mutiple 'series:' tags for crossover images but dont have a 'crossover' tag.

i have two files that i want them to appear as a group when i select sort by collection

what do i do?

Will using socks5 with your application result in the DNS lookup also being routed through the proxy? E.g. querying safebooru will be resolved through the proxy setup in the settings?

Is there a pane with tags that I can click to add to an image? I hate having to right click>manage>file's tags>type tag name in just to add tags to images in a page. It isn't much better in the media viewer either from what I can tell. It's super inconvenient.

How would I go about recording a file's original creation/modified time as a tag? Maybe a script to record it in a text file? Like the ones the booru tag parser makes.

>>12855

Potentially answering my own question:

Looking into the source code on page

https://github.com/hydrusnetwork/hydrus/blob/master/include/ClientNetworkingSessions.py

line 126, it appears you merely set the proxies to be the necessary value. Presuming a sufficiently up-to-date version of Python is installed with requests[socks], just making sure to use socks5h in the manage_options->connections should do it. In short, the user does have power over this.

Consequently, one can easily run hydrus over TOR. I did notice socks5 being quicker than 5a, presumably because naming resolution can be slow. In addition, since you'd be using the proxy's DNS, you can't really cache results. It'd be interesting if you could use your cached DNS results from the host machine, but for new lookups use the proxy. However, that's a separate thing all together.

tl;dr you can easily proxy over tor with hydrus.

If you could confirm BO, I'd be grateful.

Quick question, should I be seeing saved webms as zips of images in Hydrus? Sort of like how you can unzip an epub and get a bunch of HTML pages. Or, should the saved webms be *.webms and viewable in Hydrus? This has probably been answered at some point or another, so, if so, sorry about that.

This could either go in the bugs or this thread: Does the Hentai Foundry downloader support fetching tags? I can't get it to work with subscriptions, nor with download tabs.

>>12852

Hmm, I am not sure what this was. I wonder perhaps if the OS X legacy db info had slashes in some path instead of backslashes, and one part of the files manager was unhappy with that? It is possible that by looking at your current paths and ok-ing the dialog put those unusual paths through a normalisation routine that cleaned them up for Windows, but it could also be something else.

If you go to install_dir/db and check the relevant 'client - 2019-xx.log' file and scroll to the error time (I think you'll be able to ctrl+f for 'missing' or similar to find it as well) and check the exact path that was logged in the error, is there anything funny about it? Maybe like 'E:\my files/ab\abcde….jpg'? If you have the whole error traceback, that may be helpful to me, but if so make sure you either email it to me directly or redact the path and file hash if you post it publicly.

I'll keep this problem in mind. If you aren't getting it any more, even on the same file, it sounds like it fixed itself. Please let me know if you start getting it again.

>>12853

Not yet. I'd like to add an option namespace component to system:num_tags.

>>12854

Collection works by namespaces right now. If the files have a shared 'title:' tag, you can collect by 'title' namespace and that'll put them together. If they have page numbers, you can sort by page to get them in the right order.

I overall do not like how paged-content is handled in the client. Page tags are clunky to work with, and grouping many sorts of comic and mini-comic is a pain. Adding cbz support and the associated hydrus database backend to deal with file groups/pools with indices is probably the silver bullet here.

>>12855

>>12859

I know very little about proxies, so I cannot comment very intelligently. I just pass whatever you put in those input fields to requests, and it handles the rest. A bit of searching lead me to this:

https://stackoverflow.com/questions/12601316/how-to-make-python-requests-work-via-socks-proxy

and this:

https://github.com/urllib3/urllib3/blob/a019c5e9e8ac461111552fdf9aa97899977cccbf/urllib3/contrib/socks.py#L141-L166

So yeah, it seems like using 'socks5h://' does DNS through the proxy, but 'socks5://' does not. I'll update the UI on that panel to suggest 'socks4a' and 'socks5h' as well.

>>12856

There are a few tag 'suggestion' columns you can turn on for the manage tags dialog under options->tag suggestions. Favourites/related/recent can be useful depending on your workflow and can be double-clicked (and in the 355 I just put, you can keyboard-shortcut focus to these) rather than having to type every time.

The manage tags dialog that opens from the media viewer is actually a window not a dialog (i.e. it does not prohibit interaction with the parent window while it is open), so you don't have to close and open it again to edit other media. Also, by default, hitting page up/down on an empty tag autocomplete input should send a previous/next command to the media viewer parent, so you don't even have to change focus to tag and move on.

That said, I would like to add some quicker 'remove'-style actions to the normal taglists when looking at thumbs or the media viewer. Something like right-click->remove this tag, which could spawn a little yes/no dialog that would handle the more complicated back-end for you without having to make the whole manage tags dialog.

>>12857

I have a long-time job to cache this information and make it searchable and sortable. I know another user wants this very much, so it will eventually happen.

If you do not want to wait, yeah I think a script would do the job best. Do a stat call on each file and then output to a (hash).jpg.txt file that you could reimport with 'parse neighbouring .txt files' checked. You'd have to reimport these beside your files, which might make for a pain in the ass workflow though. You might be able to wangle it by reimporting from your actual client_files directory, with the .txt files embedded beside, but that is probably asking for trouble. If your collection is relatively small, maybe something like:

system:everything

export (copy) everything to a separate folder

generate your .txts

reimport that folder, setting to delete original files.

It'll crunch that import pretty quick, since it already has all those files.

The files will get your 'creation-date:2015-02-01' whatever.

But that would also be a pain to keep up with. I'll add this eventually (another thing the new file mainteance manager will be good to retroactively fill in) and handle it all automatically, so if you can wait, I'd advise that.

>>12869

Are those from danbooru? They are probably ugoiras imported from Pixiv, which danbooru offers both in native zip format and a converted webm. If you would rather the webm, please hit network->downloader definitions->manage url class links and switch the 'danbooru file page' entry to 'danbooru file page parser - get webm ugoira'.

If you want to reimport the zips you already have as webm, do a search for 'system:has url:danbooru file page' and 'filetype:zip', and then ctrl+a and right-click copy danbooru file page urls, and then queue those up in a url download page with the 'file import options' set to 'do not skip downloading because of known urls'.

Native ugoira support remains a long-term hydrus goal.

If you are getting weird zips from somewhere else, please let me know. If the zip contains a list of jpegs or pngs, it is probably an ugoira.

>>12875

It only does 'creator' and 'title' tags. The first HF parser I made grabbed the artist tags, but they ultimately proved too low quality to keep. Too much #i_feel_good_today #lewdtuesday and #the #party #is #in #my #pants bullshit unfortunately.

Alternately, are you not getting the creator and title tags? I just did a test here and it seemed fine, but let me know if you aren't getting anything at all.

>>12884

Yeah, I just wanted someone else besides me to try it out. I believe it works for me, and based off of function definitions and descriptions, I'm pretty sure it's acting as expected. However, it's not like I've done some absolute test like using Wireshark to see that no DNS lookups are being done locally, but, rather, are all being done through port my proxy lives on.

>>12887

The website is ATFbooru. I created my own parser/gallary downloaders ect. because I didn't see, at a glance, an existing one that suited my needs. It works perfectly for images–including gifs–however, despite the data being in .webm format when I go to the page in my browser, the download is a .zip. As you said, the zip contains a list of of images, therefore, it's probably that thing you called an "ugoiras".

I can look into the 'danbooru file page parser - get webm ugoira' downloader and add it to my own script.

However, I do want to be certain, the webm conversion wont lose quality right? They're equivalent? I ask as a comparison between differing video compressions (e.g. Xvid yields a loss in quality when compared to H264).

>>12893

That's the tricky thing with ugoiras–they are a pretty weird format, a bunch of images rendered by javascript according to variable framerate. There is no excellent lossless format to convert them to. The sites you are getting webm conversions from may be using some clever encoding options to preserve lossless and frame timings, but they may not. Webm has a lossless mode, but this is not typical–the filesizes can get pretty gigantic. The problem is aggravated by most artists putting jpegs in their Ugoiras rather than nice pngs, which can look beautiful in an ugoira when presented right, so you are already taking the compression hit from jpeg.

Ultimately, if I can write a native hydrus ugoira detector and renderer (although we'll have to figure out how to attach the JSON frame timings, which aren't guaranteed to be in the zip), then we can just keep the original files. Or figure out the right params for lossless variable frame rate conversion using ffmpeg and do that webm/whatever job 'properly' ourselves. Whole thing is a bit of a shitshow with no ideal solution at current.

If you want to copy the danbooru parser, I think it basically detects the webm embed with a higher URL priority than the normal file download link. Then, when the page provides zip on the download link and webm embed, it goes for the embed, and when there is a jpeg or whatever in both, it won't find the video embed and falls back to the non-sample-guaranteed download link.

>>12886

Well it's not really that tedious….if I knew how to script.

I thought once Hydrus imported the files, it gave them a new creation date matching their import time though

>>12894

>filesizes pretty gigantic

not too important.

>sites use clever encodes ect.

>artists putting jps in their Ugoiras

This is where my lack of knowledge of pixiv comes into play, but what is it that exists on the artist's computer that is uploaded? Does the artist upload a zip of images and then pixiv determines how to render it? If so, then the zip is the "loseless" original since that's exactly what was on the artist's PC.

As a completely separate aside it's a tad sad when you notice something, but it doesn't work out. E.g. there was an attribute field which held all the tags for the image, delimited by spaces. Consequently, if you HTML parse your way into that object and get the attribute, proceed to split (via regex option in Hydrus) you can get a list of all the attributes. However, although one will have a list of tags, they wont be counted as distinct results. I guess it's where optimality and usability are traded.

>>12896

Ah shit, yeah, sorry: I had forgotten how the copystat worked. It copies modified date, but a new creation date is made. I use this, if you are further interested: https://docs.python.org/3/library/shutil.html#shutil.copy2 Some permission bit stuff is copied as well.

This was not true in early versions of the program, I think maybe four or five years ago, so files imported before then will have modified dates the same as creation dates.

>>12897

I don't use Pixiv, so I don't know how you make an Ugoira. The uploader sets the framerate I think. I presume they have an upload form for all the images and you add ms timings beside each file, and then they create the zip and the accompanying JSON or whatever to deliver the timings to your browser. But maybe they have a program that generates that info clientside and you upload it all together.

Yeah, I'm sorry about the inability to split strings at that last stage. When I next put a good amount of time into the downloader, untangling the rats' nest of the edit parser ui will be a top priority along with fleshing out some needed parsing functionality like 'split this one string by whitespace and return multiple strings'. iirc, the current pipeline can't deal with n strings appearing from that space atm, so I can't just add a checkbox or a 'split this by xxx'. That might be wrong though, so I'll make a note to look at it.

>>12885

Thanks for the detailed response. I don't understand what you meant by this because no matter what I do, I have to switch focus from the tag input window to the media viewer to go to the next image:

>Also, by default, hitting page up/down on an empty tag autocomplete input should send a previous/next command to the media viewer parent, so you don't even have to change focus to tag and move on.

Btw, would you be averse to fleshing out the hover tag pane on the left of the media viewer with more functionality? I picture a possible workflow inside the media viewer being:

1. text input gets captured automatically by a new search box added atop the hover tag pane when pane has focus.

2. The hover tag pane shows tags already on the image when the search box is blank. When search box contains text, it's populated with autocompleted tag suggestions, and any autocomplete suggested tags that are already on the image have a highlighted gold background.

3. If there's no autocomplete tag suggestions, then the tag doesn't exist and therefore the Enter key should create and append the typed tag to the image. Idk what the performance implications of this would be, so maybe namespacing can be used to cut down the tag pool so it's less laggy?

3. up/down arrows can navigate autocompleted tag suggestions without needing to use mouse input to change focus from the pane. Enter key appends tag to image, delete key removes tag from image. Shift/ctrl click to select multiple tags perhaps?

4. Left/right arrows can navigate to the next image without needing to change focus via mouse click.

5. right click>mange>file tags for more advanced tagging

>>12796

Still doesn't work. The rule34hentai downloader still ignores Explicit rated images on rule34hentai by saying it couldn't find file or post URL to download.

Is it possible to download a pixiv tag starting from the oldest images through the gallery > pixiv tag search function or do I need to go through page by page starting from the first/oldest page in url download instead?

>>12923

Another bug I've discovered: if I go via the individual page links, after page 10/11 it keeps giving me the same set of images no matter which page I go to afterwards. Very odd!

>>12897

To better explain, if I open up the media viewer and then hit F3 to open up a manage tags window on top of it, I start with the keyboard focus in the autocomplete tag input text box, so if I were to start typing a tag, results would load in the list below. If from there, instead of typing regular text, I instead press page up/down, the underlying media viewer will browse previous/next. Do you not get that?

If you have multiple tag domains, like 'local tags' and 'public tag repository', then pressing up/down arrow in the same situation–focus on the tag input text box, and no other content in that box–should switch between them. Do you get that at all?

I am open to allowing easier tag editing in the tag list on the left, but the simple view it presents makes it tricky. It presents the 'combined' tags from all tag services in the client, so if you add a new tag, should it go to 'local tags' or 'public tag repo' or something else? I suspect my next step there is to add simple right-click menu options for 'remove tag' and then refactor the tag processing code, which is currently all bundled up into the manage tags dialog, into a shared space, so that taglist can be clever enough to say either 'remove this tag from x yes/no?' to 'this tag is in two services, remove it from x or y?' and so on. Once I have that code refactored, I am open to playing with more quick workflow here.

>>12922

Hmm, I am not sure then. If you open network->downloader definitions->manage login scripts, go into the 'shimmie login' and run a test on the domain with your credentials, it should list the results of each page hit. You should be able to copy the result body the server returned. You could try to copy and paste those pages into some new .txt files and see if there is a 'bad login' or cloudflare-related error that the login system isn't picking up? I am not sure if I wrote the shimmie login script or someone else, so I do not feel very confident talking about it. I understand it does work for some users.

>>12924

>>12923

Yeah, the basic hydrus gallery downloader works on the same principle of starting at page 0/1 and walking through each 'next' subsequent page, adding results, just like you might browse. Since most boorus present images in descending upload order, you will be getting newest first. I know some users have played around with adding a booru-specific param to the url to change the sort order, but this has mixed success.

Can you say more about the same results after page 10/11 issue? Is this for pixiv, and is it in your browser or hydrus? Can you give an example URL that does this? Several boorus cap their results to 500 or whatever, even if they suggest something big like 152 pages, and after page 25 or so they'll deliver a blank page, 404, 403, 'please login to see more', or the same last page results again. Does this sound like that?

>>12930

The pages on pixiv in the browser are fine, they continue as normal. But no matter how many times I refreshed the client, reloaded page 12 in the browser and copied it again, it gave me the previous pages' images over and over, which obviously didn't import because they're already in the db.

Here's the URL I was copying:

https://www.pixiv.net/search.php?word=%E3%81%A9%E3%81%86%E3%81%B6%E3%81%A4%E3%81%AE%E6%A3%AE&s_mode=s_tag_full&order=date&p=12

The previous URLs had exactly the same layout except the number was just lower. 11 is the last one to load properly, after that it just returns page 11 irrespective of the page number.

>>12932

I'm dumb and I figured it out. It needed a login to complete more than 11 pages it seems.

Just started using Hydrus, and I'm trying to get it to autotag an existing collection, but I'm a bit confused.

>Tags are rich, cpu-intensive metadata. My repository has hundreds of millions of mappings, and your client will eventually download and index them all.

Seems way overkill that the user should have to download all that information if they only need tags for the specific files that are part of their own collection. Isn't there a more efficient method?

I'm aware also of the tag archives, but it seems that's also a broad and inefficient approach containing data for the complete file set from a booru/gallery website as well. Is there no other way except to maintain a massive database of all the tag associations of all files ever seen? If downloaders could be used to query downloading tags only for the user's existing files (it may be possible, but I am ignorant), that would be very useful and eliminate the need to waste HDD space/ processing power.

>>12946

Only if the hash of your file exactly matches the hash used by the booru of choice. Protip: they usually don't unless you got it straight from that booru.

Use >>4475. And read the thread. It'll take some light editing to get it working due to a recent hosting issue.

>>12946

As I understand it, main reason is privacy.

Under this system, you have full access to all those tags without having to tell a tag server anything about which files you do or do not have.

And you can just sync once in a blue moon, instead of having to phone home every time you add a new file.

>>12948

That's a very good point I hadn't considered.

>>12947

Having a real bitch of a time messing with scripts. I got a few from that furfag dr guy. Got one of them to work for one image that I manually selected, and then everything took a shit and the scripts just hang at the fetching stage now.

https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/wiki/2.-How-to-fetch-tags-in-bulk-with-md5-hash

It seems this method would be the absolute easiest but there is no longer anyway to get the md5 file hash in the options menu, and the default file names are the same as the sha56 hash. I think I'm going to just throw in the towel and try the public tag repository.

Not sure has this been asked before: How to minimize the active subscription windows? They are blocking the thumbnails and are very annoying.

I ctrl+f'ed the threads, so hopefully this hasn't already been asked.

I created a default sankaku channel tag subscription with a test query of "tifa lockhart" (without quotes), but I'm getting a network error.

Specifically, bad gateway 502 along with some html shows up at the top of the edit screen for the subscription. When I first performed the search, it asked me for a user agent string I believe which contained firefox and some other stuff. I did a similar subscription for gelbooru which worked fine. I'm using tor but can access the site just fine in my browser.

>>12932

>>12935

Thank you for the update. Pixiv seem to change their access rules every few weeks, it is a constant battle.

>>12933

Ah, thank you for this update. I think it is supposed to be default for new users–I think the shimmie login script has an example URL that does the link-up automatically on client first-boot initialisation–but I wonder if you got caught up in some combination of version updates that meant this did not apply? I apologise if so. The login system, as I am sure you now know, it pretty a scrappy v1.0, barely out of prototype.

I do remember other users telling me R34H had funny login rules anyway, so I recommend you keep an eye on this in case it starts giving you more trouble. A future version of this whole system will be able to detect if you get suddenly logged out for some reason and will try to recover and give human-friendly errors and so on.

>>12934

I apologise, I fucked up the clipboard watcher! It is spamming those messages false-positively on certain kinds of URL that are not matched. It is fixed today, please let me know if you have any more trouble with it.

>>12946

>>12947

>>12948

>>12950

Yeah, privacy and low server CPU are the top reasons. I prefer that your personal CPU cycles happen on computers you control, so the tag repo is really just a clearinghouse of data. The PTR is getting huge now though, 580 million tag mappings, so after this duplicates work I am going to put some time into figuring out better solutions here. One top priority is adding a tag filter to the incoming stream, so you'll be able to say "I don't want filename: or title: tags" and so on.

This is very much on my mind. The PTR has been a great success, but it was always a crazy prototype and we are hitting some limits.

>>12952

No excellent way yet. I think each sub has an option in its edit panel to not show a popup, but I generally advise against this as you often then don't notice it at all, even when it is having problems. This is on my mind, and I have a fairly high priority job to figure out an up/down arrow button or something so you can collapse and expand them. For now, best solution is to hit network->pause->subs and then just unpause them when you are happy for them to run.

>>12953

502 is a server-side error. It could be that Sank was overloaded at that time and could not respond to your request, or it could be CloudFlare giving the 'we tried talking to the site but it timed out' similar message, or they may have blocked Tor. Is the browser Tor as well, or just hydrus?

If you run a test download and check the gallery log button (the little icon button on the gallery search section of the downloader panel), it should bring up a list with the first URL it tried to hit with more error info in the 'notes' column. Do right-click->copy notes and put it in a text viewer–that should be the html page that was returned, which should, if you scroll down a bit, have some more info. It might be like 'Tor is blocked m8' or something else. I'd be interested in knowing what you find, but be careful just pastebinning that html and linking it here, as it may have personal info.

I just clicked here from /tech/, shithole that it is. I have no idea what this project is. Could we have a sticky?

>>12973

>New user? Start here -→ http://hydrusnetwork.github.io/hydrus/

lrn2read

>>12966

Thanks, seems to be working now.

>>12966

What do you guys think a good max allowed bandwidth would be? The default 64mb for sankaku is too little.

>>12965

That'd be nice. Or make each of them a tab would be nice too.

I noticed some of the tags from the PTR are just flat out wrong, such as pictures in which a girl is standing and facing the viewer so that her ass isn't even visible being tagged ass, and so on. Is there a way to deem certain public tags as less important and hide them, or a way to ignore public tags and copy only a portion of them to local tags?

I suppose I could always just add a local alias and search by the alias instead of the public tag.

>>12986

I added this after being contacted by a top person at Sank who was concerned about their bandwidth costs, which I sympathised with. After some discussion, we compromised that I would set a restrictive bandwidth limit by default. You can of course edit your client's rules to be whatever you want. If you want to do them a favour, you might like to try to find what content you can on other sites.

>>12995

There are not excellent ways yet. The PTR has plenty of weird and incorrect stuff, and there are so many new tags every day that it is completely beyond me to moderate effectively (I am also not a natural janitor or site-manager). I will be doing substantial work on the client-repository relationship after this current duplicates work. You will have more control over what tags you get and where they go. I expect to add multiple 'local tags' services and easier ways to migrate your tags around so you can better handle all this stuff.

Anyone else having problems with the deviant art downloader, first the login script stopped working, now it can't seem to download any page content to parse correctly.

I created a gallery download page and it stopped after 19 images with a message about 502 bad gateway. I figured the booru was having issues, but no, I can access it just fine. In fact, a new gallery with the same exact query is downloading right now just fine, making the message below confusing:

expires in 50 minutes: 502: <html>

<head><title>502 Bad gateway</title></head>

<body bgcolor="white">

<center><h1>502 Bad Gateway</h1></center>

<hr><center>nginx</center>

</body>

</html>

Is it possible to have two clients share a database? I keep my data on external drives and want to be able to work in both Windows and Linux.

>>13018

I think the answer is yes, they just can't be open at the same time.

>>11122

Sankaku gives you at most 1000 results from any query, but you can circumvent that to some extent if your initial query was at most 3 tags by looking up query + order:filesize and query + order:filesize_asc.

Is there a better solution to this? Finding a custom query to get as close to 1000 images without going over and then having to do this every 1000 images is a pain. Even 2000-3000 images would be a massive improvement.

>>13029

Nm, I happened to be checking the bugs thread and noticed someone made a parser that solves this supposedly. Just out of idle curiosity, why are parsers shared as png files and not text files? Also could a parser be malicious?

Databases work cross-platform, right?

I want to try out hydrus through wine because of all the linux version's issues. How do I go about making the switch? Do I simply copy and paste the linux hydrus network over the windows one like I was simply updating? Or do I restore the (linux) backup I have through windows?

just started using this a few days ago, really liking it for sorting my giant folders of booru rips, but one thing I can't find is parent child picture relations, like danbooru has.

I want to be able to set one picture as a parent, and then another as a child picture, for alt versions and small doujins/image sets. Is there a way to do this or am i fucked?

>>13031

I wrote that parser. It's a png because Hydrus lets you export them as .png files for easy importing. If you're that worried just copy and paste this:

[58, "sankaku gallery page parser fixed", 2, ["sankaku gallery page parser fixed", "b29f9b48f7c6374d12e7d4cbb527f16fd3c725946dd6479c0684e5e0e1dea15f", [55, 1, [[], "example string"]], [], [26, 1, [[30, 4, ["Next page from rel link", 7, [27, 6, [[26, 1, [[62, 2, [0, "div", {}, 22, null, false, [51, 1, [3, "", null, null, "example string"]]]]]], 0, "next-page-url", [51, 1, [3, "", null, null, ""]], [55, 1, [[[1, 7]], "parsed information"]]]], 0, false, [6, 50]]], [30, 4, ["Thumbnail URLs", 7, [27, 6, [[26, 1, [[62, 2, [0, null, {"class": "content"}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]], [62, 2, [0, null, {"class": "thumb"}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]], [62, 2, [0, "a", {}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]]]], 0, "href", [51, 1, [3, "", null, null, "example string"]], [55, 1, [[], "parsed information"]]]], 0, false, [7, 50]]]]], ["https://chan.sankakucomplex.com/?next=3794306&tags=krystal"], {"url": "https://chan.sankakucomplex.com/posts/index.php?id=123456"}]]

And I suppose a parser could be designed to work in such a way it makes a shitload of network traffic to get you IP banned from a booru, honestly, mine might because sankakuchan hates people using their bandwidth yet has the most expansive collection of images imo. Also it's not too hard to make a image file into a malicious script, that used to/might still be an issue on Linux, it just requires you to execute the file which isn't really a concern with Hydrus.

>>13033

You can set images as alternates, and you can click an images and find all of its alts, but not much more. It is something the dev is working on currently.

Is there any way to print a list of tags and their occurances? It'd be really helpful to weed out the typos and stupid non-tags artists insist on using in their galleries

>>13033

This is a highly requested feature, which I suspect may win the 'next big job' poll that the dev does every few months. IIRC, the duplicate system overhaul he's doing right now is basically a precursor to being able to overhaul the 'alternates' system.

For now, I'd suggest setting those images as alternates of each other.

I imported the latest Quicksync update files (not the full db), and it's taking forever to process now, like 1 entry per 10 minutes, then cancels out after around 15 minutes. Is there something i'm doing wrong or do I have to let this run for a year?

Is there a way to set multiple tags with one shortcut?

>>13034

But doesn't hydrus execute the code from the png?

Also the downloader still finishes at only 1k even though I've set it to stop after 2000 and changed the bandwidth restrictions. Is sankaku channel tag search the right downloader to choose?

Is there a simple way to set up a blacklist for subscriptions?

I have a subscription setup for sankakucomplex with a few hundred artist queries in it, and while I do want Hydrus to download anything tagged with those artists, I don't want it download anything tagged with, for example, "holding_hands". I know could add "-holding_hands" to each query, but I don't want to have to do that for every single query.

>>13014

It seems ok to me here. I just did a login test and download and things came through ok. Are you getting any errors, or are the URLs just not appearing when you do a gallery search? If they have any errors, can you look at the list of file import statuses (hit the little icon button on the downloader page) and right-click->copy notes on the bad results, so we can see more of what went wrong? I wonder if they turned on CloudFlare protection or something briefly.

>>13016

Thank you for this follow-up. After a bit of playing around, I was able to reproduce this problem. There is a certain unusual parsing type that was kicking in here that was not sending tags to the next stage down. This is fixed for 358. Please let me know if you have any more trouble here.

>>13017

If you turn on help->debug->report modes->network report mode while this works, does that give you any more information? Is there perhaps a url class that is redirecting the URLs to a different URL? I've had that sort of problem before, where I was accidentally hitting the wrong URL despite the URLs in the file list looking correct.

It could also be some CloudFlare-style protection that is blocking anything that doesn't look like a browser with a BS 502 error. Usually you get more info than that brief html document though.

If you have some example URLs, I am happy to try from my end in browser and hydrus.

>>13018

>>13027

Yep, the db is platform neutral so you can have two installs point at the same db and they should be fine. Only rule is only one client install can be open on a db at a time. Normally this is impossible to violate since the client will detect if it is already open on the same computer and complain, but if you have a funky network setup this check can be skirted.

If you just want to have a db on a usb drive and move it back and forth between two machines or boots, Linux and Windows, you should be completely fine. I recommend you extract two portable installs and put the db proper in its own directory, something like this:

E:\hydrus_db (db and client_files folders go here)

E:\hydrus_linux (linux install extracted here)

E:\hydrus_windows (window install extracted here)

And then launch Windows/Linux with shortcuts with '-d="..\hydrus_db"' or '-d="../hydrus_db"' as appropriate. But you know your situation better. Here is some more help about all this:

https://hydrusnetwork.github.io/hydrus/help/database_migration.html

Let me know if you would like any more help.

>>13032

I'll just point you up to my post above: >>13065

Basically there is no trouble migrating the db around–it is completely portable and will work on any install from any location, so you are good to copy or move it around to test in other locations. As long as you can see the main db dir with your hydrus client exe or script, you should be good to go and if everything is working, it'll launch.

I still recommend making backups and so on before you try anything too fun, just in case.

The help file here talks about the different components of the hydrus install:

https://hydrusnetwork.github.io/hydrus/help/database_migration.html

I think in your case you'll be wanting to move around your original install_dir/db folder. Extract hydrus to your wine folder structure and then copy that db dir straight into the install. I am happy to help if you run into trouble.

>>13051

I am no cyber-hacker-3000, so I cannot promise anything iron-clad, but my serialised png system shouldn't execute any external code. The data embedded in the png is zipped up JSON, stored as grey bytes at the bottom of the png itself, which I read and load. It unzips to the same as in >>13034 's post. There's a couple of data bytes in the top-left of the png to say where the data starts and how long it is. If what is read and unzip parses as a hydrus parser object, it is good to import, otherwise it will throw an error. At no point are arbitrary lines of python code intentionally executed.

If someone has an attack vector for https://docs.python.org/3/library/zlib.html#zlib.decompress or https://docs.python.org/3/library/json.html#json.loads, they could make up a malicious png. My Windows dev machine here says the current zlib version I am on is 1.2.11, which the docs suggest is currently OK.

If you are very concerned about your privacy, I recommend you only import by copy/pasting the json text and giving it a sanity pass with your eyes just to make sure it isn't trying to api-url-redirect your gallery queries to "NSAHydrusUsers.ru.cx" or anything.

>>13037

Not at the moment. Many clients have way too many unique tags to show in one place in any kind of reasonable wait time, so figuring out show-all UI has been tricky. There's a 'big job' to figure out a tag cloud control which would likely have this tech.

If you only have, say, 2,000 unique tags, you might have some luck messing around with wildcard searches in the autocomplete tag input. I have some blocks to stop simple '*' queries, but someone was telling me the other day that '*:*' works and will show basically everything you have. Be warned it may take some time for such a query to return all results and may be hyper laggy.

Otherwise doing a search for a whole bunch of random files and then sorting the results' tags by incidence is a decent way to find (1) and done stupid tags.

>>13043

Try forcing an analyze under database->maintain->analyze->full. It will take a few minutes, but if your problem here is due to too much data at once, this should fix it.

>>13050

Not yet. Future shortcut system should be able to do this, but I will likely not work on that until I have caught all the old code up to what is current.

>>13058

Sure. You do this through tag import options of any downloader context. You don't want to set this for every individual subscription, so I recommend you go network->downloaders->manage default tag import options and then set them either for the global default or per domain. If you haven't overridden this info in your subs, your subs will inherit from these defaults at runtime.

Under your tag import options edit panel, check the button for 'set file blacklist'. Add your scat/gore/whatever tags that you don't want, and any downloader using those tag import options that finds a page with one of those tags will skip the file download and give an 'ignored' status.

It is worth pausing your subs while you do this and testing this with a manual download page, just to make sure you have it set up how you expect.

What was the procedure for manually copying login cookies into the client?

>>13067

Thanks for the breakdown, and I'm not too concerned about it. It was an idle thought that arose from thinking how this png system weirdly obfuscates the downloader code, so I wondered what the security implications were.

>>13034

I looked and I didn't see any new downloaders added, so I assume this was added onto the sankaku channel tag search downloader. Is this intended for use with the gallery downloader? Maybe that's why it halts after 1k downloads.

>>13072

This should be explained in >>11696.

>>13072

>>13075

I have a lot going on at the moment, but in the 'to-do fairly soon' list is adding cookie access to the Client API, which will make this whole thing a lot simpler. The guy who writes the browser plugin is keen to add a button that'll just transfer login cookies to hydrus in one click.

>>13073

I don't know exactly how or what you imported in your situation here, but if you added this, you likely also need to update the 'links' to make sure it is applied to the right 'url class'. Check network->downloader definitions->manage url class links. Find the sank gallery url class and make sure it is matched up to the new parser you imported.

Anyone else having issues with pixiv? I keep getting an error that "The parser found nothing in the document!" Only started happening within the past 48 hours.

>>13083

see >>13057

>>13078

I imported the png version of >>13034 via network>downloaders>import downloaders. It's supposed to allow the sankaku gallery page to exceed 1k items.

Interestingly, after I did network>downloader definitions>manage url class links and applied `sankaku gallery page parser fixed` (which I assume is from the png I imported) it consistently stops after only 24 items.

>>13085

Hmm. That is the correct procedure for switching over. I expect you'll want to switch back until this gets figured out.

I am afraid I do not know anything about the 'fixed' parser other than it is supposed to pull a different gallery page with some 'startswith' id or something, so I can't talk cleverly about debugging it. I assume you are logged into sank if you were originally getting 1k results, otherwise I would say 'try logging in'.

Is there a way to use Hydrus with sites that make you click "Agree" or "Enter" before you can use them, e.g. giantessbooru.com?

Hydrus doesn't download anything from DeviantArt, no matter if I log in or not. I understand that you're supposed to have the client download something without logging in before you can do it with an account, but how?

Gelbooru added some gay default tag blacklist you have to modify to access certain tags. Any way to remove them from hydrus?

Here's an example.

https://gelbooru.com/index.php?page=post&s=list&tags=loli

>>13114

Also, when you go into the options you can enable all site content and it gives you a special key you should be able to throw in with searches to get the full results like "&api_key=anonymous&user_id=9455", but I couldn't get it to work with anything.

Speaking of tagging systems, what are idealistic searches you wish could be usable on boorus?

>eg. "on-model"

>eg. "girly/femboys but no masculine males in the image" (possible solution would be having negating tags for like "non-femboy")

>eg. "horsecock but only on horses"

As a side note, I found an imageboard which used anime girls for an image captcha. I wonder if that could be utilized for tagging boorus, like how Google uses theirs for AI training.

>>13111

Never mind, I figured it out. I needed to clear cookies first, through the login menu, then try again. Now that I have downloaded some pictures without logging in, I should be good to go.

>>13107

Yeah, should be able to. In the new downloader system, that process would count as a 'login'. You can create one of these yourself, although it is not trivial to do. You also need very basic web development knowledge, such as what a cookie is.

The help for all this starts here:

https://hydrusnetwork.github.io/hydrus/help/downloader_intro.html

There is supposed to be some login help here:

https://hydrusnetwork.github.io/hydrus/help/downloader_login.html

But I just discovered that I never wrote any! If you would like to write your own login script, skim some of the downloader help to see what my approach is and then check out network->downloader definitions->manage login scripts. Your client should have 'hentai foundry guest login click-through' as one of its defaults, which basically does the same as what you want here: if it doesn't see a session cookie in the HF session, it simulates hitting the main page and clicking-through.

I think if you duplicated the HF login and then edited it to your other site you might have some luck. That particular HF script also posts an HF content form to unhide spicy content, which you can remove. I am happy to help you if you run into trouble, and if you figure something out, please post it so other users can benefit!

>>13108

Thank you for this report. Please pause the repo if you have not already. Can you then go to your log (install_dir/db/client - 2019-7.log) and scroll down to this traceback and copy the last bit? (If you still have the popup, you can do the same thing with the 'copy traceback' button). I think I know what is causing that error, but I'd like to check the whole error be certain.

It looks like your files manager is/was unable to find one of its subdirectories. Have you ever moved your database around with database->migrate db? If you have, please go to your moved 'client_files' dir. If you have not, please go to the default at install_dir/db/client_files. Are there 512 directories in there? There should be 256 in the format fxx and 256 with txx (files and thumbnails, respectively), with the xx counting in hex from 00 to ff. Do you have the 'f4c' directory? I suspect you do, because if it were missing, hydrus would complain on boot. If so, it sounds like your db has somehow lost its internal store of db locations.

If you do have the f4c folder, please make sure your client is closed and then jump up to your base db dir, normally install_dir/db, and run the sqlite3 executable. Copy paste these lines into the terminal that pops up exactly:

.open client.db
SELECT COUNT( * ) FROM client_files_locations;
SELECT COUNT( * ) FROM client_files_locations WHERE prefix LIKE "f%";
SELECT COUNT( * ) FROM client_files_locations WHERE prefix LIKE "t%";
.exit

You should get 512, 256, 256, but I assume some will be less. Please let me know what you get. We can fix it manually if only f4c is missing (i.e. if you get 255), but if there are more missing I won't walk you through manually fixing this issue, since it is a pain in the ass going through 256 entries to figure out what is missing. If this is your issue, I will write a new repair routine on boot that will catch this unusual problem and try to fix it automatically. I am 95% certain I can have this in for 359.

You should not import any files to your client in the meantime. That missing directory issue will affect any file imports. The cause for this issue is also a concern. I cannot think of any way that one of those directory entries would be removed. Anyway, let me know what numbers you get, and the full traceback, and we'll get further into this.

>>13111

>>13121

Thank you for this report. I have a report from another user that suggests the DA login script is broken, and I am due to give it a look this week. I am not sure what your cookie issue was, but please keep me updated if you learn anything new.

>>13114

>>13115

I don't have a gelb account so I can't talk too cleverly, but I think your best bet here is to have hydrus login to gelb for you. I recommend you use a new throwaway account. Am I right in saying that gelb has a blacklist tied to your user account? I assume you can set your throwaway account's blacklist how you want, and then hydrus will inherit that when it logs in.

To add that api_key to searches in hydrus, you'd have to alter both the gelb GUG and URL Class in hydrus. It might be a bit of a pain if you aren't used to the system, so I recommend trying the login first. Let me know if that works for you. Related help at the bottom here:

https://hydrusnetwork.github.io/hydrus/help/getting_started_downloading.html

>>13121

>>13124

OK, so I can download stuff if I'm not logged into DeviantArt, but once I do log in (and it does say I'm logged in just fine), it doesn't download anything. The login says "valid", and I get no error messages; it just doesn't import any files.

>>13122

Thanks! I'll look into it and see if I can make it work.

I actually sought help from that downloader_login page before posting, but it wasn't too helpful, as you can imagine. :p

>>13117

I don't tag on boorus, but for hydrus I am all for longer/combined tags to clarify these situations. I think tag parents in hydrus can help tag quicker too. For instance:

horse penis on horse

could have all these parents, some being grandparents:

horse penis

equine penis

animal penis

penis

animal genitalia

species:horse

species:equine

whereas 'horse penis on futanari' or '-on human' or whatever other pseudo-suffix would have a different set of nice parents. It is neat as your brain can work on this higher level of recognition and it lets you add a bunch of valid tags in one action. Same with something like "girly/femboys but no masculine males in the image" or the various "small female big male" stuff you come across sometimes. With these specific tags, you can define precisely the core of what you are interested in in the image without being awkward or overbroad in the search. Also it lets you quickly and precisely dismiss what you don't want with a single negation tag like ['horse penis on horse', '-horse penis on human' ], rather than ['horse penis on horse', '-species:human'] or ['horse penis', 'species:horse', '-species human'], which are not the same thing.

I think you could easily go crazy trying to define these forever, however, so I will fall back to one of my main tagging rules: only tag what you are personally interested in. If you want to see horse penises only in their natural God-given state, then pursue this 'horsecock but only on horses' path, and perhaps the '-on intersex' companions you might want to exclude with a negation tag. But don't try to tag everything on everything on everything or you'll go nuts.

I have spent thousands of hours creating an engine that burns trillions of cycles of CPU to precisely categorise and search different types of bepis. Didn't expect I would be here ten years ago, jej. I am certain that within ten years we will be feeding this info into AI tech of our own with some fun results. :^)

>>13118

Yeah, when you start typing a tag, put a hyphen "-" before the tag, so like "-footwear". This will discount any file with that tag from the search. I think all regular tags will work this way, including special tags like "creator:*anything*". If you have some tags in the 'selection tags' area on the left, you should also be able to right-click them and say something like 'exclude this tag from the current search' and some other options.

Note that 'system:everything' is secretly implied to any search that has >0 tags, so if you just enter for '-footwear' or whatever, you don't need to add the 'everything' yourself.

Also, when you do big searches like this, try hitting the 'searching immediately' button (ctrl+i shortcut as well, I think), which will pause the search-on-new-tag and lets you queue up a bunch of tags before actually firing off your big search.

>>13126

Thank you for this info. No idea what's going on here–maybe the Eternal Cloudflare has struck again, or maybe once you log in they now deliver differently 'shaped' gallery page data or something. I'll look into it.

>>13085

That parser was faulty, I believe. I made it work for me some time ago, should be at >>13138 – please tell me if it works.

To hydrus dev specifically: how do you manage all the tasks involved with a project of this scale? I find it staggering that you're able to keep on top of it all so well, presumably as one person. Apologies if there's already an answer around.

I asked some time ago about how to restore old downloader functionality(e.g. "get X more new files"), and I tried simply increasing the "stop after this many files" values(search and gallery parser) and resuming the search, but Hydrus doesn't do anything with it.

Is there anything else I can do or must I switch to searching with a high value initially and pausing when I'm satisfied?

I've been working on creating my own parser recently, and have been successful at the parser bit, but it only downloads the first page of a gallery link. How do I make it download the entire gallery?

My gallery parser correctly returns a "gallery next page url", and I have a url class of type "gallery url", yet only the first page is being downloaded.

Am I missing something stupid?

Whoahhhh so what the fuck am I looking at here? What exactly is hydrus? Can someone please give me a run down

Is it possible to have the tag autocomplete enter existing namespaces when tagging with a word that isn't a non-namespaced tag? e.g. entering "splatoon" automatically tags as "series:splatoon" since I don't have a "splatoon" tag in my tag services. I know it's possible with tag siblings but adding an additional pair for each tag is a lot of extra work.

>>13132

>>13170

Hey, I am sorry I was late to this. I think another user (sorry if it was you somewhere else, and I am forgetting) was telling me that this cookie is not tied to the account, but the session. I still have a job to add cookie export to the Client API. This will make this operation a one-click action for anyone with the browser add-on and clear the hassle from this shit.

>>13137

>>13136

Hey, I am sorry getting back to you late here as well. I believe this now auto-heals on boot in 359, so have you had luck with that?

For some reason, six entries have been dropped here. You might like to check install_dir/db/help my db is broke.txt just as some background reading. It might be worth running a db integrity check on client.db at least, just to see if there are any other dropped rows in that file.

If the new version did not fix this automatically for you, you can fix it manually by using this line:

INSERT INTO client_files_locations ( ?, ? ) VALUES ( "fXX", "client_files" );

For each dropped fXX prefix. fb9, f4c, and four others that I imagine you can figure out by booting and running some more file integrity checks or similar. If your actual location for the files is not install_dir/db/client_files (as per the "client_files" parameter there), the client will ask you to relocate to the correct place on boot.

>>13159

Yes, unfortunately there is no way at the moment to get that old queue-building method back.

>>13168

>>13170

>>13191

I guess since this is divorced from needing an account, the cookie is not actually remembered account-wide. Thank you for this information.

If this is a tick-box with a typical html POST form to save it, I should think it would be possible to both write an account-less login script and extend the existing account-credentialled gelbooru login script to also do this step, if they user wanted. The Hentai Foundry scripts do something similar here, to do the initial home page 18+ screen click-through.

>>13171

>>13183

If you are testing the URL in the normal URL downloader, this page doesn't have the pipeline tech to do a gallery walk, so it will only ever parse the single gallery page you drop/paste into it. If you want to test a full gallery walk with 'next page' stuff, you will need a GUG for it and do it in the gallery downloader.

>>13184

>>13185

Yeah, my first siblings system proved only half the job here. I want to make sibs and parents much more portable so users can just grab like a 500KB text file from somewhere and just pull a whole bunch of decent 'series' siblings from the PTR to their local tags in one go. Doing it as one person by hand is a huge pain, and the current system does not resolve subjective disagreements well (e.g. is "bodypart:", though technically accurate, actually 'good'?).

The next step of tag sharing will be better en masse definition curation, and making it more shareable.

>>13193

Thank you so much for this fascinating insight. I've been struggling with organisation a lot recently, so I thought I would ask you how you do it, as you're clearly able to continually deliver despite all the challenges you mention. That's where I'd like to be, and I have every respect for you. That software looks like just what I've been looking for (and not finding) - thank you again.

I thought that keeping up with messages might be difficult, as you have so many scattered locations to check. I expect it to be a shortcoming that imageboard communication is naturally ephemeral, so you may feel pressured to follow up with people in a timely manner. Perhaps that helps you keep on track though. I expect I've taken quite a bit of your time at any rate, for which I apologise. Certainly don't feel like you should respond to me.

I have held off on suggestions for the software in respect of the fact that the userbase is something worth preserving, as you say. Any improvements to the UI, for example, would probably lower the barrier of entry and result in an influx of lower-skilled users, causing more hassle for you. I had wondered, though, if it was possible to write hydrus in such a way that frees you up to work entirely on the backend, leaving frontends to interested users. But I'm sure it's a lot of work for not much payoff.

Am I required to run hydrus as an administrator? I don't recall needing to several versions ago.

>>13206

Thanks m8. I appreciate your comments. I like to insist I am just a lad though, no one special. I don't want to get a big head.

It has taken years for me to figure out a good schedule and attitude to doing this work, and there is no magic trick I have found except keep working every week and recovering over and over from failure. If you want to figure out your own system, my best advice is to keep at it, and if you are young, don't expect it to come as fast as you want. Stress is part of the whole package, like any other work, and another resource to be managed.

Stoicism works for me. It is a little cliche in some imageboard circles, but if you haven't read it, I recommend Marcus Aurelius:

https://archive.org/details/meditationsofmar00marc

Now that hydrus is functional and does a whole bunch of things, my overarching plan is to move myself and the hydrus code to be more platform-focused, providing a base that users can make things for, rather than pushing towards a complete package I have to continually tinker with. The downloader overhaul last year was the first really big stab at this–moving the whole thing to be completely user-configurable so regular users can populate and maintain it, rather my having to hardcode and update all download info myself. I now want to extend the Client API, maybe play with add-ons of some sort, and 'generalise' the code to allow more user-configurable behaviour. This lets me worry less about infinite specifics and spreads the workload without my sperging out at working in a team.

I'll be doing some PTR/tag repository work after this current duplicates stuff. Main push will be not dissimilar: giving clients more control over what they pull from the PTR, and starting work on improved local tag siblings management.

Completely decoupling the UI from the db is probably too big a job to consider at the moment, but I know some users are planning pseudo-booru wrappers using the Client API. We'll see how that goes and iterate on from there.

I am all for new suggestions, btw, so please don't be afraid to say something. I just have trouble keeping up with them a lot of the time. I'm not about to add a Facebook plugin, ha ha, but thoughts on improving and cleaning up workflows is always helpful. As the guy who understands the cogs behind everything, it is often difficult to picture what is confusing or unhelpful to users.

>>13209

Shouldn't do, unless your db is in a protected location like 'C:\Program Files'. The client needs to write to its db dir, which is by default in install_dir/db, so if that is in a protected or otherwise read-only location, it might not work as a regular user.

What error messages are you getting, if any, that drove you to try 'run as admin'?

Here's some general info on the db and how to move it about, if that is helpful:

https://hydrusnetwork.github.io/hydrus/help/database_migration.html

>>13212

Ah okay, the install directory got moved recently after discovering Chocolatey and installing hydrus through it. I'll see about moving it back to the root of C:\ where it was before.

>>13211

Yes, that makes a lot of sense. I agree about stoicism. I happened upon your recommendation of Marcus Aurelius to another guy, and it's good stuff. Some self-discipline is certainly in order, which cannot be provided by software.

I certainly intend to give some feedback after I've had a chance to organise my thoughts. I'm sure most of it, if not all, will have been said already though so hopefully it's not too tiresome reading duplicate suggestions. Can't really see a public to-do list (or the like) around to help prevent that, though.

>>13192

Updating to the latest version seems to have solved the issue. When i started the client i got the prompt that the db entries were missing, and i could then point them to the folder where my files are stored. Will do some integrity checks now to see if anything else breaks.

Thanks for the help!

>>13221

Ah, never mind. I found the option to import.

Did the dev ever add the ability to manage multiple databases? I know there was talk of it at one point, but I haven't kept up with development in a while. I like the idea of having separate databases for SFW and NSFW or some similar scheme.

Why is there still a search plugin for the Mishimmie? It's dead, it was never exported and Dreamhost has no interest in restoring it.

>>13217

Yeah, I'm afraid my to-do is too big to reasonably convert to a public system. It has personal info in it too, from user submissions and so on, so I'd have to sanitize rather than just copy/paste. I also haven't found an online issues system that doesn't make my head explode either–for a while, I had github issues activated, but for me it just resulted in dozens of long discussions about how potential new features might behave, and I lost more and more work time to it until I couldn't keep up and abandoned it. All that shit is driven towards boosting social media engagement, which is the very thing I hate, kek.

I know some users on the discord tried to make some sort of wiki to act as a 'frequently asked issues' or something, but I don't know how it ended up.

>>13220

Great, thanks for letting me know.

>>13226

Yeah: turn on help->advanced mode and then check the repo under services->review services. Hit 'export updates' to pull them out, and then on the new (also advanced mode) client hit services->import repo update files to push them in. Then do one sync with the PTR on that new machine so it knows the updated files it wants, and it should recognise it already has them and be able to start processing right away.

Be very careful doing this. It is a lot of CPU work all at once, and I didn't design the processing pipeline to deal with it. I recommend you let it run for x minutes processing on a shutdown maintenance run (which will have strict timing rules and be able to cancel itself eventually even if it runs super slow), and then restart the client and run database->maintenance->analyze->full. This will make the db do some optimisation stuff on the new populated db tables so it can do the rest of the processing at decent speed.

I overall do not recommend hitting 'process now', if any user has told you about it, on a fresh sync. Some users have tried to do a full processing run in one go from nothing, which usually locks up the client due to mis-optimisation and slow work.

Let me know how you get on!

>>13238

Not yet. It'll be a pretty huge job, unfortunately. I'd like to have multiple local tag services coming in after this duplicates work as part of a significant en masse tag management overhaul.

>>13241

Thank you for this report. I'll take it out of the defaults.

Is there a way for me to filter by filetype? I know you can sort by filetype as an option, but say if I wanted to only see .jpg files of a certain tag, is there a way for me to do that besides manually adding filetype tags to every image?

Guys, how do you set up a Derpibooru login? I do not know what kind of login script it uses. My grabber just uses the API key.

>>13258

yes you can filter by filetype same way as you apply other filters to a search

what happened to the downloaders here

https://hydrusnetwork.github.io/hydrus/help/adding_new_downloaders.html

is there a way to know the last time hydrus was open? I went to a repair center and I'm afraid the saw all the nasty shit I have.

>>13273

Maybe the log files? Check last time modified, or if there are any dates written to them.

>>13279

fuck, they did.

fuck fuck fuck fuck

>>13281

Holy shit, there were looking at your stuff for like 7 hours? Maybe they want a copy

>>13281

F

This is why we use passwords

(And not repair centers)

Hopefully ther's an easy way to do this.

I have two seperate instance of Hydrus with different files, tags etc.

Is there a way for me to merge them into a singular Hydrus instance with all tags intact? Could I start the server on one port and connect to it from the other instance and grab everything?

>>13307

Hey, I am afraid I have a ton of IRL shit going on today, so I can't catch up with my messages properly until tomorrow. I will try and check in again later this evening. Please feel free to make a fresh thread. There is certainly a lot to talk about on this subject.

If Anons can get use out of hydrus as an archive tool, I am happy. As I say a bit here >>13305 , to me this episode is a reminder that you can't rely on content, particularly spicy content, remaining in a location you do not control. Hydrus was in part designed to solve this problem by being a 'local' booru.

That said, hydrus is not currently well geared towards paged content like comics. Increasing support is planned, and I am in no way opposed to adding cbz/cbr import support and improved paged browsing, but at the moment the overall hydrus workflow works best for single images like you would find on most boorus and imageboard posts. Any Anons who would like to know more, please feel free to ask other users here or check my 'getting started' help here: https://hydrusnetwork.github.io/hydrus/

In regards to IPFS or using Hydrus File Repositories or something to effect better sharing on large scale, I think this will be increasingly possible over the next few years, but hydrus is not currently there. The IPFS integration (which also lays the groundwork for other small-file p2p support) is under active development now and is still mostly for advanced users who are already familiar with how hydrus works. There isn't an easy switch to throw and have doujins flow nice and easy again. In future, I'd love to do eventually add IPFS/hash/tag cross-referencing to be able to search IPFS or other file sharing domains using tags and file identifiers harvested from other sources.

I am open to more opinions and thoughts. I am not big into the doujin scene, so I am largely ignorant of the actual mechanics and the overall size and shape of the problem to solve. I saw some posts saying Sad Panda was about 50-200TB. If there is a db out there with all the file hashes and tags and so on, it wouldn't be super difficult to convert it to hydrus tags and import it to a client. Current PTR is 620 million tags covering 31 million files, so if those are the sorts of numbers we are talking about, I think the current hydrus db can handle some jobs related to this at least.

>>13265

>>13258

Yeah, try system:filetype in a regular search page. You can mix it with any other tags or whatever.

>>13263

I don't use derpibooru–I assume there is a derpi login script on the github that another user made? If you have imported it ok, go network->downloaders->manage logins and find the entry and click 'edit credentials'. Put in your username/password and hydrus will try and login as you the next time you make a request on the derpibooru domain.

Most of the downloaders in hydrus download as if you were browsing. Again, I don't know if the derpi login script (and accompanying downloader you have) works on an api key, but I imagine it just uses user/pass and simple gallery/file post page requests like most of the others.

>>13269

What's the problem? Did some go missing?

>>13307

ipfs is the solution, and it's being developed, but the timeline is very long and not likely to come out any time soon.

>>13281

If you like, close the client, go to install_dir/db, and enter this:

.open client.db
SELECT dump_name, datetime( timestamp, 'unixepoch' ) FROM json_dumps_named WHERE dump_type=13;
.exit

That'll give the timestamps of the x most recent 'last session' and 'exit session' timestamps, just in case any are left over from that 07/23 period. It probably won't add anything more useful than that log, but you never know. It depends on your idle settings, but since tag repo processing kicked in two and half hours after boot there, my guess is they opened the client and then for a period they didn't move the mouse, probably starting about 1pm or so, then moved it again at 1:30 to cancel processing, then left it alone again. I assume your client had caught up with the PTR by the 3:41 processing job since it stops there, or maybe they were continually moving the mouse until 18:40 looking through all your stuff, maybe, and then closing the client safely.

If your session does not seem to have changed in your boots since then, the worst the guy probably did is click through your open tabs. If the files (or empty pages) are the same as before, he didn't hit f5 on anything.

In an unrelated note, hydrus runs well from an encrypted partition. Ciphershed is an easy-to-use solution:

https://www.ciphershed.org/

>>13333

Shit, should have said, 'run the sqlite3 executable' in the db dir. That'll give you terminal that you can paste those lines into.

>>13299

It is generally a pain in the ass every time, since it depends on what the exact file format is, and regex is not the most pleasant tool to work with, even if it is excellent.

For commas, something like this should work:

(?<=parentfoldername\\).*?(?=,)
(?<=, ).*?(?=,)
(?<=, ).*?(?=\.)

But I am a bit rusty.

>>13301

No simple merge system yet. If you only want to move files and tags, it isn't super difficult–basically, export all the files and reimport them to the new db, and hit help->advanced mode and use the advanced service-wide update button on both clients' appropriate local tags panels on review services to export and import a Hydrus Tag Archive of your tags.

If you need to move ratings or complicated (i.e. other than 'everything is archived) archived status, there aren't easy solutions. For inbox/archived, you can split it into two jobs, but ratings have no import/export yet.

Let me know how you get on. I can figure out a hacky fix to transfer the data or maybe some simple import/export finally.

Subscriptions and several other objects should be exportable and importable from their dialogs, either to clipboard or png files.

>>13302

The menu that has file | pages | database and so on? That should not be removeable in hydrus, but perhaps your OS or Window Manager allows you to clear/hide it? What OS are you on? Can you take a screenshot of what you see?

The shortcuts system is in a mess of several different systems. Some is hardcoded. The new system is available under file->shortcuts, which you will be able to check out once we get your menubar back!

is there a requirement needed for ubuntu lts minimal to get the server to run? the client appears to run fine but getting an unhandled error on the server when i close it from the command line. end goal is to move the ever growing db off my pc and onto this box that has way more storage.

when its running i'm not able to connect from another computer and the firewall is disabled ubuntu.

Unhandled Error

Traceback (most recent call last):

File "threading.py", line 914, in _bootstrap_inner

File "threading.py", line 862, in run

File "site-packages/twisted/internet/base.py", line 1267, in run

File "site-packages/twisted/internet/base.py", line 1276, in mainLoop

— <exception caught here> —

File "site-packages/twisted/internet/base.py", line 902, in runUntilCurrent

File "include/ServerController.py", line 407, in TWISTEDDoIt

StartServices()

File "include/ServerController.py", line 333, in StartServices

HydrusData.Print( 'Starting services\u2026' )

File "include/HydrusData.py", line 970, in Print

print( repr( text ) )

File "include/HydrusLogger.py", line 159, in write

self._log_file.write( message )

builtins.ValueError: I/O operation on closed file.

Hi to all,

There are any bad side effects by installing hydrus (and it's archive) into a share program folder like mega or dropbox?

I wanna try to do this in order to access and update my archive from multiple computers

>>13312

Feeling sad about this, also https://8ch.net/animu/res/111544.html#q112288 (not using crosslinks to prevent bumplocks)

>>13347

Threw a bunch of python modules that it was missing. still not run able.

Unhandled Error

Traceback (most recent call last):

File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner

self.run()

File "/usr/lib/python3.6/threading.py", line 864, in run

self._target(*self._args, **self._kwargs)

File "/home//.local/lib/python3.6/site-packages/twisted/internet/base.py", line 1272, in run

self.mainLoop()

File "/home//.local/lib/python3.6/site-packages/twisted/internet/base.py", line 1281, in mainLoop

self.runUntilCurrent()

— <exception caught here> —

File "/home//.local/lib/python3.6/site-packages/twisted/internet/base.py", line 902, in runUntilCurrent

call.func(*call.args, **call.kw)

File "/mnt/hydrus/hydrus network/include/ServerController.py", line 407, in TWISTEDDoIt

StartServices()

File "/mnt/hydrus/hydrus network/include/ServerController.py", line 380, in StartServices

HydrusData.Print( 'Services started' )

File "/mnt/hydrus/hydrus network/include/HydrusData.py", line 970, in Print

print( repr( text ) )

File "/mnt/hydrus/hydrus network/include/HydrusLogger.py", line 159, in write

self._log_file.write( message )

builtins.ValueError: I/O operation on closed file.

>>13352

Nevermind i'm just going to move it to an external drive

>>13337

Worked like a charm, thank you! I only needed the tags and the file associations so that was very easy.>>13337

>>13352

>>13347

>>13357

Thank you for this report. Another user told me that 361 is having trouble starting services correctly–they only seem to start for him when he hits ctrl+c, which then quits the program. I did change this this past week, and it was all fine on my test machines, but something is messed up with the initial event queue for certain platforms. I am going to give it another go this week. Please let me know if 362 works for you, or, if it is convenient, see if 360 works ok.

>>13350

Yeah, I generally recommend not putting the actual db on a remote system. SQLite likes to have some clever hard drive locks, and these remote locations tend to offer 'fake' versions of the lock commands if they even offer them at all. It might eventually cause malformed errors, or it might just not work for you at all.

It is also good to have the actual db on a local system for latency purposes.

You can try it if you like, and I'd be interested to know how it goes, but make sure you have backups of everything, and don't try to run two separate clients from two separate computers on the same db at the same time.

I know several users who enjoy putting just their media files on a NAS or even in the cloud. Keeping the db files and thumbnails locally but putting files on remote storage tends to work ok. Here's some related help talk about your options:

https://hydrusnetwork.github.io/hydrus/help/database_migration.html

>>13353

I'm afraid it is just me, one lad. I have trouble working in teams, so I have a strict policy of working on my own for hydrus. I am happy to help other Anons with their various situations, but I am always drowning in work, ha ha. I can never get things done as fast or well as I'd like.

That said, the overall push of the past couple of years has been to make hydrus more 'platform' based with the completely user-configurable downloader system and the new Client API. This lets other users fix their own problems without having to wait for me to hardcode-fix the danbooru downloader or whatever. I'd like to continue this trend going forward.

>>13360

Thanks. That's odd. On my end I just make a MenuBar object an attach it to the main frame, and then wxPython and your Window Manager are supposed to handle the rest according to native controls. Some Window Managers put it up on the main OS frame. My best guess is 2bwm was trying to put it into a location wxPython couldn't handle and it bombed out. Increasing compatibility for different Window Managers is a constant fight, so please let me know if you discover anything new here.

>>13359

Great, thanks for letting me know.

If I were, for purely cosmetic reasons, to change the current media destination's location drive letter (it's the same drive, but now E instead of L) how would I go on about that? I can't really migrate the database as it's the same drive. Just tell Hydrus where to find it again via adding a "new" location?

Hey,

Thank you for your hard work.

This is an amazing tool.

My question: why not upload the tag database to something like megaupload/mediafire and lets us import it that way, instead of through chunks of 64mb/day?

Is there something preventing you/anyone from doing this? Of course it wouldn't be up to date but that would eliminate a large portion of the waiting from our side /bandwidth from your side.

(I also asked this through tumblr, feel free to ignore my question there)

>>13375

Already been done here:

https://cuddlebear92.github.io/Quicksync/index.html#two

In Hydrus use services -> import repository update files

Oh, got another one:

Is there a faster way to get a single file out of hydrus than right click > share > copy > image (bitmap) if I want to post it somewhere?

>>13374

Ideally no. Theoretically, it should be fine™. A bunch of stuff will change, like all those thread watchers will become multiple watcher pages each with one page, but nothing should break, hopefully. There's a max page limit of about 145 or so, now, and I am not sure what yours currently is, so pare down your session if it is larger than that.

The main thing is make a backup before you do it. If it all goes wrong, you can always roll back.

I think we are talking more than 100 versions. The client won't let you update more than 50 versions at once due to bitrot (I can't guarantee the update will do anything). I recommend trying 10 versions at a time. If you are on 250, try 260 and boot it. If no errors, try 270, 280, and so on. If it breaks, try 275 or whatever, half-way between your jump. Let me know if you run into trouble.

If you are a madman, you can try a couple of 50s or whatever, but whatever you do make sure you have a backup before you try. I know one user just did like 320->360 and he said it was ok, but this is a complicated subject and every client is different.

>>13375

>>13376

Yep: https://hydrus.tumblr.com/post/186658823864/hey-there-thank-you-for-your-amazing-work-i-was

Let me know if you have any problems.

>>13377

I think you can set up a shortcut for that, but it might only work in the media viewer. Try file->shortcuts->media and add a new shortcut for 'copy_bmp'.

However, if you are uploading to 8chan or any other normal site with an upload box, just drag and drop the thumbnail to your browser. It is a lot nicer to send the real file rather than a bmp/png copy. If you are trying to DnD to discord, try turning on the discord BUGFIX option under file->options->gui.

>>13376

>>13379

I see, thanks!

For someone with a ton of images is there some way for me to limit my inbox search to, say 1000 images so everything doesn't slow way down trying load all the images in the inbox?

>>13403

someone had requested a mascot for the software.

>>13401

>>13400

just do the option on the help menu to set it up automatically for you

>>13405

I did for both before coming here, and it still wasn't connecting.

>>13407

I figured it out, I was using an older build. New version works just as advertised. Also seems to be downloading only 6000+ tags, will that number increase?

>>13419

Also, I have all of my images sorted into fairly general but still useful folders, is there a way to have Hydrus keep those images assigned to those folders while uploading?

I am very slowly bumbling my way through learning the program and figuring out how to use it properly, and only seven hundred or so images in did I find the automated tag lookup script in the options menu. I noticed, however, that the script does not also pull the urls from the pages of the art, as the more roundabout way I was using does. Is there any quick and easy way to make it so the urls get pulled and attached to the image along with the tags as well?

>>13420

Nevermind, figured that out too, turns out it's an option in file>import folders.

>>13421

That's a good idea. That tag lookup script stuff was the prototype for my later downloader engine overhaul, and the way it fetches and parses its results is not great. I'd really like to rework it to use the newer parsing system, at which point it could definitely associate URLs in a sane way. That whole system is also pending an eventual throttled 'auto-check' system that you'd be able to queue up and have work in the background in some sane way that didn't end up eating bandwidth and CPU.

Best way to associate URLs right now is to run a gallery download. A bunch of the site parsers now pull md5 or similar hash and recognise when the client already has the file (and hence don't end up downloading it again, but will get tags etc…), so they work a bit like the lookup scripts anyway. You might like to do a test on a site you like for an artist you like and see if that solves your tag&url problem quicker. Help on this if you are new to it:

https://hydrusnetwork.github.io/hydrus/help/getting_started_downloading.html

>>13431

Thank you for your attention! That is what I have been doing, feeding the hashes of the existing files I'm importing from my own folders through the downloader to get their tags and urls, the only problem for me is that the process for that is being excruciatingly slow for me, taking upwards of two hours to get through about 150-ish files, meanwhile the tag lookup is nearly instant.

I should assume that is purely a connection problem on my end, yes? And also thank you for all your documentation, it has been an immense help so far.

Is there a way to delete unused tag info? Hydrus's two .db files sum to 19.5gb filesize, while my actual images are only 4.4gb. It seems like it's storing a ton of info about tag associations for images I don't actually have. Is there a way to tell Hydrus to go through and prune all data that doesn't match to an image on disk?

>>13432

Ha, sorry for insanely late reply. Since you see normal tag searches go fast, I think your connection is fine. There is a built-in limit for 'gallery' URL hits in hydrus, which might be what you are running up against. Normally, I strongly recommend keeping this limit in, as it makes regular booru searches work at a sensible pace, but if you change the setting under options->downloading, under 'gallery downloader' section, the 'additional fixed time…' setting, which is likely 15 or 30 seconds, take that down to 3, and these url queries, if they are indeed waiting on that, should go through quicker.

I wouldn't recommend taking it down to 1, but you can try if you want. Gallery hits tend to be CPU heavy for servers, so hydrus tries to also be polite by not hitting them too frequently. Hash lookups may be ultra-fast serverside, but if they have set up bandwidth banning based on frequent heavy requests to the gallery script, you might get hit without meaning to.

>>13445

If you are synced with the PTR, unfortunately you need to get all the PTR definitions and data as well, which is about 20-odd gigs now. As you guessed, it gets all the mappings (~700 million at this point!) the server has, so all computation happens clientside and with complete privacy. This tends to be less of an issue if you have a 1TB+ total collection, but for your situation, it is not great.

If you wish, you can delete the PTR, which will clear most of that data after a vacuum call. It will not delete master records, which I have a planned solution for as per here, which I answered earlier today:

https://hydrus.tumblr.com/post/189534434774/ive-been-using-hydrus-and-trying-out-the

>Unfortunately, there is no way to clear out ‘orphan’ master definitions for hashes and tags yet. This is becoming a bigger problem with various larger dbs, so I expect to have it in in the next couple of months. I know what I need to do, I just have to write a decent maintenance routine that won’t gobble up too much CPU. The ideal is to have it work in the background, so you won’t have to do anything–it’ll just shrink your db naturally over time.

>>13446

I haven't got one in by default. The main repository for user-made download scripts is here:

https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System/All-in-Ones/Single-Sites

Unfortunately, I do not see one for Pillowfort there. I just checked out the site and saw it is in beta and needs a registration fee. Let me know if I missed a way to view as a guest. If you have a login, do you know if they have an API to talk to? Is their browsing experience similar to tumblr? I do not make downloaders myself any more, but if you could provide some example html/json/xml data for the Pillowfort equivalent of a booru's gallery tag/username search and post pages, perhaps the guys who do make downloaders now can figure something out.

If this is a bit technical for you, feel free to get into the discord and see if one of them will walk you through it.

So im getting disk I/O errors because my hdd is presumably dying on me and I neglected to make any backups of my database. I move the entirety of the db over to a new drive successfully except for client.mappings, I get errors whenever I try to move or copy it. Am I fucked?

Also the captcha on this chan is fucking cancer holy shit

>>13450

Also I have tried cloning it in sqlite, seems like its getting stuck at current_mappings_10.

The program also still works fine, no tags seem affected though I dont really know what client.mappings handles. The only reason I noticed something was wrong was because I suddenly couldn't process repository updates.

My yande.re tag parser has disappeared for some reason. Can anyone post an import file for me?

Is there a way to re-import files from my collection at mass in order to change where their tags are stored? I made the mistake of saving all import tags to just the public repository, now im noticing now that a huge amount of images are getting tagged completely incorrectly by it so now everything is jumbled. Therefore I just want to make a new local tag system for specifically the imported tags but re-importing everything manually would take weeks.

>>13450

I responded to you on Endchan.

https://endchan.net/hydrus/res/10.html#432

>>13455

Try this:

https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/raw/master/Download%20System/All-in-Ones/Single-Sites/yande.re-tag-search-2018.09.22.png

>>13459

I am not totally sure about your problem here. When you say to re-import the files to get their tags again, did you add tags to your files when you imported them? Was that from regex filename parsing, or neighbouring .txt files, or something else? Or were the files downloaded, and these tags are parsed from boorus? If you were to do this the slow way, what would you be doing?

services->tag migration may be of use here, but if things are too jumbled, maybe not.

>>13464

>>13463

>>13465

d8ca57 is me, fuck this multiple imageboards with different BO trip systems, ha ha ha.

>>13465

Parsed from boorus mostly. The slow way would be to manually download the images again using the urls that are saved with them, one at a time.

>>13464

Thanks! This works when I do a test parse, but when it comes to actually importing a new image from yande.re it's not pulling any tags for some reason. I double checked that the url class link was correct, but still no luck.

anyone recommend a good external viewer for mac? don't really know whats good

How could I go about pulling "date taken" data from my pictures and get that converted to a tag?

I'm going to merge two different databases together with one being on an old hydrus version.

Is just exporting all the files them importing them with the tags enough or is there a better way to go about it?

>>13480

that's probably the most straightforward way of doing it

>>13471

I got it. It turns out that my default tag import options had been reset at some point.

I wonder if it would be difficult to create a thread image ripper like the ones for 4/8ch for other *chans running tinyboard + vichan or whatever 7chan is running? Could one just edit the 4chan one or is it a more involved process?

Anyone have a GUG for 4plebs? It seems similar to an imageboard setup but watcher, URL, and simple downloader all fail, yet the full images are there when I click manually and are just stored on its own CDN like on 4chan.

Or is there some workaround?

>>13467

Ah, yeah. Hydrus does not remember the origin of every tag, so you can't do a local-only retroactive mining of tags based on tag source. Best way is to set up a new url downloader page and feed the urls in again (doing a right-click on a big ctrl+a of the files you want should let you copy all urls that match a certain site, and then you can paste back into the url downloader). In order to get the tags again (normally hydrus will just skip urls it has seen before), you'll want to set the tag import options to 'get tags even if urls/hash already known'.

Do a test of a dozen files first here, to make sure you have it all set up how you want, and then do a bigger batch.

>>13479

Hydrus does not read EXIF data at the moment. I plan to add this sometime and add search predicates and tag conversion for it.

I do not know when I will add this, so if you want to do it now, you will have to use an external program to pull it before import and then convert the tags you want to a newline-separated 'sidecar' 'filename.txt' file to import beside the file. I know some users who have done this, although I am afraid I have forgotten the name of the EXIF extracting software they used. I think they had a system where the software converted to xml files, and they used a script to convert that to newline-separated as hydrus understands for import. If you are not very technical, it may be a pain in the ass, but I am happy to help if you want to give it a go.

>>13480

>>13481

Yeah. You might want to do it in two batches: one for inbox files, one for archive, so you can quickly regenerate that information in the destination. If both clients have services->tag migration, that's the best way to import/export tags atm, using Hydrus Tag Archive files.

>>13484

database->migrate database has good estimates of db size, ok estimates of media size, and ok estimates of total thumb size. If the SSD is very tight, you might not want to move the db files–they need a drive with free space to work well. I recommend 1-2 times as much free space as the size of the largest of your four db files. This is no big deal if you don't sync with the PTR, but if you do, your client.mappings.db is probably 18GB or so.

You can now create multiple local 'my tags' services. Hit services->manage services to add more. The new services will appear in any place you can add/edit tags as separate panels or tabs.

services->manage tag display is a new-ish panel that lets you hide tags for single services or all services.

There is no way atm to apply a tag to all files in a file domain. I recommend you set up a page with:

system:limit=1024

-the_tag

And then do several rounds of f5-ctrl+a-f3-ctrl+v-enter-enter. You can push the system:limit a bit higher to do it in larger batches, but you'll take an inefficient CPU hit dealing with too many at once. If you have 100k+ files, this may be too much work.

>>13485

I keep meaning to knock up some quick parsers and url classes for the bunkers, and then I do not find time for it. It would not be super difficult for anyone familiar with the download system. Most imageboards offer good APIs or nice html, and as you say, if you have a parser for one vichan board, you probably have support for 95+% of them running the same software.

8kun is difficult because of the DDoS stuff.

>>13486

I don't know anything about it, I am afraid. How do they fail, is it a 503? If they have aggressive CloudFlare protection, this is a common cause of hydrus not being about to get files when a browser can.

>>13488

- Not yet. I would like to add this some time.

- Try database->migrate database to set up multiple storage locations. It allocates files across multiple locations randomly according to set location 'weights', so if you just have a couple of multi-GB files, you can't guarantee it will be smart about putting them in one place, but as your collection increases in size, it will be more accurate.

- No support for this, unfortunately. I am not sure there will be a neat solution in the near future. You are free to do what you want, but I recommend you not use hydrus for movies. I designed it for imageboard-style files, where you need to keep track of hundreds of thousands of smaller files, so most of the workflows and maintenance timers are built around that. Files that have sidecars just don't work well with the hydrus storage system.

is it possible to delete the deleted files record for specific files, rather than the entire database? I know I can do the latter via `services > review services > local > files > all local files`, but I want to keep the whole deleted files history, except for maybe ~120 files worth.

Alright I think I'm an idiot. I'm trying to download the images from an artist on pixiv, and despite having other pixiv downloads that work this one just doesn't and I can't figure out why.

I would really like to use hydrus, but I feel I'm losing the browseability of folders. How do you "browse" certain core tags easily?

Like, right now, one folder I browse frequently is based on specific characters where the directory structure is: "my-images/series-name/character-name/". I've imported a bunch of these folders and given them tags, but now I miss the ability to see, at a glance, which series I might want to browse.

I figured out that in hydrus I could mark certain tags as "favorites", which is a start. With my old folders I could quickly navigate the hierarchy using the enter+backspace keys to go up/down and arrow keys for moving around, and I can't do that with hydrus tags. It's also clear that the list of favorites would be unwieldly at some point.

Let's forget the hierarchy for a moment. Each character is a nice conceptual "album" and it'd be great if I could just browse my "character" tag and easily switch from one character to another. I can double click on a favorite tag to bring up that tag search, but if I double click another tag it does an "AND" search…I first have to navigate to another part of the window to remove the first character's tag.

So I'm stuck. I really want to move to tags though so I can easily browse by artist or other tags which cuts across my "/series-name/character-name/" folder structure. But I want those tags to provide the same browseability had I used a "/artist/character-name/" folder structure.

>>13504

I personally have a bunch of tabs set up with tags to organize some my db without having to search every time. For example, I have a blanket tab for all 2d art which then has other tabs in it that are tagged for whatever else, including an Artist tab that then has a bunch of tabs that are organized by artist. It wouldnt be hard to do the same for what you want, though it is maybe a bit less convenient than a simple folder structure though you gain way more than enough to make up for it.

Speaking of, a feature i'd really like to see is the ability to add a "master tag" to a tab so that that tag is required for all searches in that tab and all others that are in that branch. Would make it a bit less tedious to organize that way.

>>13507

And though im not sure its within the scope of the program, actual folder-like functionality would be great to see. The way I would imagine it would essentially just be creating tabs and giving them a presence in the actual gallery screen instead of just the tab bar, while being able to navigate through them like an actual file explorer.

Organizing everything through tabs is kind of hard on the eyes and awkward to use if you have a bunch of them, so this would be a godsend. I'd probably lose some time going full autist and setting up my own folder structure to organize absolutely everything, though.

>>13510

The way I would imagine it is that you can specify a folder structure in a separate input box, below the input box for the search string. So you might say:

folders: <namespace1> / <namespace2> / <namespace3>

which would automatically populate a virtual folder structure with the results of your search. Items that were returned by the search but don't have a tag for namespace1 would be put inside a 'misc' folder at the bottom. (and recursively, there would be such 'misc' folders inside each level of the hierarchy)

I accidentally deleted a tag, my intention being to replace it. Using the show deleted option in the tags menu, I can still see the tag, but it isn't applied to the files it once was. Is there any way to re-apply a deleted tag to files?

new to hydrus, is there a way to change a namespace's color? I added two new namespaces recently and they both were the color of one I already had.

I am unsure on whether to post this here or in the bugs thread, but after updating the program to the latest release, I noticed that I no longer can press Ctrl+C to easily copy the file's path. However, the hardcoded shortcut section still references that as the shortcut to copy files into the clipboard. Is this change by design, or is it truly a bug?

>>13552

If I'm not mistaken, you can do that by going through File > Options > Tag Presentation

Where is the place with the most current info on getting downloaders to work? I am searching but I keep finding outdated info.

I am not good with computer. I just want to download every single picture of my waifu on the entire internet.

Hey lads, apologies again for lateness. New schedule calls for me to check this four times a week, let's see if I can stick to it this time.

>>13496

Under options->files and trash, turn on the advanced file deletion dialog. It provides an option to delete a file without leaving the deletion record.

If the files are already deleted, there isn't a way to do this I don't think. Maybe you can do something clever like searching for the files in 'all known files' domain and then re-deleting them. You know what, I'll make a job to add a 'clear deletion record' thing to the right-click menu for these files.

>>13497

As I understand it, Pixiv now prohibit nsfw results from non-logged-in searches, and the hydrus pixiv login script silently broke some weeks ago, I think due to them adding captcha. Please use Hydrus Companion to log into pixiv now:

https://gitgud.io/prkc/hydrus-companion

>>13499

Yeah, that md5 lookup process is decent. The other solution is the Public Tag Repository, which I specifically designed to solve the problem, but if you are new to hydrus, you might want to hold off on that for a bit. The PTR is now a sledgehammer, and if you only ever want to tap out tags for a couple thousand files, it might not be right for you.

>>13531

I folded that into a changelog a little while ago, I think. I basically just exposed the internal values I was using for the score to users.

I will make it so you can value pngs over jpegs. Out of interest, can you talk a bit about your situation where that is useful? Is this for, say, a set of patreon rewards, like 'raw' ultra-high-quality png versions? In general, most users I have talked to about this tend to have many more situations where the png is some clipboard.png imageboard copypaste post of the original jpeg, so they generally prefer jpeg to be the higher weight be default.

>>13546

Some deleted tag storage and workflow is borked, so let me know if this doesn't work:

Open up the file(s) in manage tags dialog, set to 'show deleted', and then double-click the 'deleted' tags that have '(X)' after them. The double-click should apply a re-add action.

I think you can also just try to add the tag as normal, and the delete will be overwritten.

I am assuming this is for 'my tags' tags, and not something like the PTR. Undeleting tags for a tag repository is more complicated and really needs a janitor.

>>13553

Ah, thank you for this report. I am pretty sure that is old–I think all the copy stuff is now under file->shortcuts->media, with action names 'copy_(bmp/file/path/sha256_hash)'. I'll clear out that old text.

>>13555

Here's the main repository:

https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System/All-in-Ones/Single-Sites

Open up Lain in network->downloaders->import downloaders and drag those pngs on her. It isn't always perfect when there are object conflicts, but it generally will import everything automatically.

So I have a couple questions here, first I was wondering if its possible to automatically add a specific tag to every imported file that DOESN'T have another specific tag thats parsed from wherever im importing the file. For example, I'm trying to categorize my collection by 2d and 3d files, 3d is usually tagged on boorus but 2d usually isnt so its kind of a pain doing it manually.

Second, how do I completely delete the ptr so that I can re-download it? Im fairly sure my ptr is corrupted because a lot of tags are mixed around where they shouldn't be. I've tried deleting it from the manage services menu and making a new one but that just processed the repository updates that ive already download again and doesn't fix the jumbled up tags. Dunno if re-downloading it would even fix the problem but I figured i'd try.

Is there a way to clear all previous deletions without reason?

I've done some fuck ups in the past and now that there's deletions explanations it's a good time to unfuck my hydrus.

>>13569

I don't think you can do this 2d/3d tag logic atm. My best suggestion for now is to have a search page set up with:

-2d

-3d

(possibly system:known url=somebooru.org)

i.e. 'all the files that are from site x that don't have 2d or 3d)

And then once a week or so go:

F5

Ctrl+A

F3

Type '2d', hit enter

Ctrl+R

On it, to do this manual job in a single batch.

For the PTR, I am generally confident that your PTR is not corrupt, but the tags were applied to the file incorrectly due to human error or a mis-parse. Sometimes, for instance, a website changes its layout and a bunch of 'favourite' or 'users also liked these tags' get sweeped into a bunch of files, which slowly gets corrected on the hydrus side from user petitions and janitor approval. Simple mistakes are fairly common, and the chance of a PTR file being corrupt and that not being noticed by the client during processing is infinitesimal, so my bet is on the former. The client knows exactly what the update files should be, and if there is a single byte difference from what the server has, it throws all sorts of errors.

If you notice completely the wrong tags, with no correct tags, on many many files, particularly if it is the same kind of swap (like everywhere you expect to see 'series:touhou', you actually see 'character:samus aran'), it is rarely possible that your db has switched up some master tag definitions, possibly due to a hard drive fault. Can you provide an example hash of a file that has bad tags, and what tags you see on it? Right-click it and hit share->copy->hash->sha256. I'll check it my end and let you know if I see radically different tags.

>>13570

I do not think so. If you are feeling brave, and you are not connected to any file repositories, and you absolutely have a good backup in case this goes wrong, try running the sqlite3 executable in your install_dir/db directory and copy/pasting each line here in turn:

.open client.db
delete from deleted_files where hash_id not in ( select hash_id from local_file_deletion_reasons );
.exit

Then boot the client. Do a sanity check, a test import on a previously deleted file or something, just to make sure everything seems good. Hang on to the client.db backup just in case this screwed something up and we need to reimport its old data.

Oh, and run help->debug->data->clear db service info cache, which will force your client to recount deleted file numbers next time you look at them.