/hydrus/ - Hydrus Network

>>2232

>>2233

hydrus is written in python and uses the stock re library which doesn't support a few things, including \K, that some other regex engines do. For a overview of python regex see here: https://docs.python.org/2/library/re.html

hydrus_dev it might worth looking at the pypi regex library that aims to replace re it supports some nice features like \K and capturing groups in lookaheads/lookbehinds which would be nice if you ever add support for capturing groups in resulting tags.

https://pypi.python.org/pypi/regex

I'd provide you with a working regex, but i'm not very good at crafting them :/.

>>2234

Closest option to this is file->options->media->Zoom smaller media to fit media canvas. Other than that not that I know of.

>>2235

This boils down to hydrus's inbox concept, essentially the inbox is a sort of staging area for files. It's used for files that have just been imported and have no tags, that way you can sort what you have and have not tagged. The little mail icon is just to indicate whether a file is in the inbox or not. A file being archived meaning it's NOT in the inbox.

This is not the only way to use this flag, but I believe it's the intended use.

Hopefully this helps a little bit.

>>2258

Drag and drop the image thumbnail in hydrus onto a file select in your browser (or anywhere actually).

You're also able to export images from hydrus through the share option in the context menu.

>>2234

As >>2237 says, the fit/don't fit checkbox is the only zoom option right now, but I would like to add per-mime zoom settings, and more options (so you can say things like 'fit a png unless its fit size is less than 110%, in which case just show it as 100%').

>>2237

That regex library looks good, thank you. I would like to overhaul the path tagging dialog, not only to add groups and other advanced regex stuff but also adding better gui or maybe even a wizard to make it easier for people unfamiliar with regexes to use it. A big limitation though is fitting those advanced controls into an already cluttered dialog. I may just have to have multiple pages.

When I focus on that, I will probably update the regex library as well.

>>2238

That regex site is great, and your example is as well. I will add it into the program and the help.

>>2258

When you import files, make sure the 'delete files after successful import' checkbox on the first import path selection dialog is set how you want. If you would like to keep your original files as well, that is a completely fine way of running the program.

Otherwise, as >>2260 says, you can just drag and drop thumbnails out of the program to do a quick export. This works very well for upload boxes like here on 8chan. Due to technical limitations, they retain their ugly hash filenames. If you would like to export en-masse to the filesystem with nicer filenames (e.g. 'creator - title.jpg') you can do that through the more complicated file export dialog. Let me know if either do not work for you!

>>2280

Not really. You could try giving all your files a local 'nsfw'-like tag and then including '-nsfw' in your searches, but that's a bit awkward and prone to mistakes. For now, if you want to divide your files into multiple file domains, then I recommend you run multiple clients, one for each purpose. I may extend the client to support multiple 'local files' services in the future, but I am still thinking about the best way to do it.

>>2282

Most of the time, you probably want to stay synced with a tag archive. Most use md5 as their file hash, while hydrus uses sha256, so only mappings for local files (where both sha256 and md5 is known) can be cross-referenced and imported to the db. A few tag archives use sha256, so all data is imported in one go, but this is rare.

Either way, staying synced to a tag archive only adds very small lag, usually a handful of milliseconds per file import, so you don't have to worry about performance loss.

I usually leave my main IRL client on my laptop open all the time with about eight tabs for different ongoing purposes. I have it set to run maintenance jobs on idle, so it generally stays synced to the PTR all the time. It stays open through computer sleep/hibernation without problems.

I also have a file-empty admin client that I use to process PTR petitions, that I only have open for a few minutes a day, just to get that job done. I have that set to sync on shutdown, which usually takes ten or fifteen minutes.

Both scenarios work ok for me, but it took some tweaking on file->options->maintenance and processing to get what I wanted.

>>2283

Here is probably best for now, but perhaps there could be a more permanent place. I assume you have read my rough schema at:

http://hydrusnetwork.github.io/hydrus/help/tagging_schema.html

In general, I am easy-going about this stuff. If you start tagging files with good and appropriate 'medium:' namespaces, I figure other people will start using them as well. If your tags are less helpful, they might be revoked or altered with siblings. Remember that most files in hydrus have metadata that allows for 'system:mime is video' and 'system:duration>0'-type searches, so adding things like 'animation' may not be as important as you expect.

Anyway, I don't have a specific place to discuss this, but feel free to create a wiki.

>>2288

That seems exceptionally slow, especially from an SSD. Having it on your system drive's disk might slow it slightly, but I would expect that to be slight and occasional.

I have written a general help page on this (although a lot won't apply to you, because most slowdown is usually due to fragged HDDs) here:

http://hydrusnetwork.github.io/hydrus/help/reducing_lag.html

At the bottom of that page are instructions on how to produce a profile. If you like, please generate one or more profiles of your slow file results loading (and anything else that might be slow) and send me what you get.

Also, some general questions:

Are you synced with my public tag repository?

How large is your install_dir/db/client.db file?

What OS and CPU do you have?

If you are running windows, and you open task manager (Ctrl+Shift+Escape), what sort of idle CPU and HDD usage is your computer using?

If you create a search that always produces the same small set of files, and you press F5 to refresh it, does it load faster on the second or third try?

>>2290

I think you should stay synced, because most of the regular boorus are md5 tag archives, so you likely won't have imported all the info.

Since you have mentioned this, I will make a job to include a little check when you sync with an archive that will tell you if the hash type makes an ongoing sync redundant, to clear up future confusion about this.

>>2296

As I say at >>2306 , I may have fixed this problem for tomorrow's release. Please let me know if you still have lag after you update.

How exactly do I instantly synchronize tag databases, apart from closing hydrus (which will interrupt the synchro after 30 minutes)?

>>2320

Thanks!

Is there a way to delete tags that aren't in use by any image, or was that one of the "coming soon" features? I recently fixed a spelling error for a character name in all my files, but I'm worried I'll mess up again since the misspelling still comes up in the tag suggestions.

>>2327

>>2329

I am sorry to hear this. How long does it seem to lock up for? I have noticed some extra lag on the initial set-up of tag processing recently, sometimes up to a couple of minutes on my slower machines. It may be the client was still working in the background, but was not updating the gui.

I've reduced this lag a bit in today's release, so please check if you still get it in the new version. If it still lags for, say, ten minutes, or you don't notice any hard drive or CPU activity from the client in task manager, let me know.

>>2331

What's the count on the misspelling? Is it something low like 'series:ebangelion (1)'? If so, the tag is probably applied to some files you have since deleted, or someone else made the same misspelling. Tags that aren't applied to any files at all (i.e. their count is 0) shouldn't appear in autocomplete results.

Try opening a new search page and changing 'local files' to 'all known files' (click the button on the autocomplete dropdown). Then also change the tag search domain to whichever tag service you care about, and then try to search for the misspelled tag. Hydrus can return search results even for files that aren't on your hard drive–it just won't have the thumbnails or be able to show them in the media viewer. But you can manage tags for 'remote files' like normal and remove/petition the bad tags in the same way. If you clear up all the instances of the misspelling, you shouldn't see it appear in autocomplete results again.

Let me know if I haven't explained that well.

>>2350

Actually, nevermind!

It's working now, and writing content rows. Used to crash after a while on the previous version, but it's fine now.

Thanks for the help, anyway!

>>2347

That's an odd error–thank you for reporting it. I think the client is failing to correctly determine the correct thumbnail for some non-local files. When you load these files, do they all get the hydrus ψ symbol black/grey thumbnail, or are they blank, or do they have their original thumbnail? I might have fixed this for next week, but I am not sure.

>>2349

>>2350

>>2351

The recent change in v199, which splits the client db into smaller pieces, should reduce future vacuum time to about thirty seconds. I hope you don't get this awful lag again.

>>2353

It takes a bit of effort to add each new file type (hydrus stores and searches files' resolution and duration and so on, which means each new type needs a parser that can query that information), so I have only added the most popular ones so far.

I would like to add text, but it is difficult to automatically determine those sorts of files' mime as they typically have no file header. They would probably need a different import window, and I would likely want to add full text search and internal rendering and so on. I think text file support was on my last 'what big things to work on next' poll.

I expect I will push for more audio and video types when I have internal audio rendering done.

>>2355

Thank you for reporting this. While recently improving 'fit zoom' calculation, I accidentally removed it from the list of possible selectable zooms–this is fixed for next week!

>>2370

Doing:

multiple clients->one NAS location

is not possible, but:

one client->one NAS location

Should work fine.

I would like to permit multiple clients per db, particularly for LAN situations, in a future version of the program, but it is not currently possible–a client assumes it has exclusive control over the file storage, so if another client deletes or adds a file, the original will not be notified and will get confused.

>>2374

That does sound possible, and it might be something I add in future. Please bring it up again when I open up a new poll for 'what big things to work on next'.

So I'm still kinda learning. Whenever I start up, everything is blank. I have to specify "everything" first, and then it gives me access to my tags.

Is there a way to make it such that it always defaults on "everything" on startup? The first time I fucked with this, it kept crashing and what not. Ended up reinstalling and seems to be running smoothly, but I'm wondering if I just gave it too much and it crashed or something. Shoulda saved the logs but completely forgot (m-my bad ;_;)

So I really like what I've played with in this program. I think it's great if you have a metric fuckton of images.

However - I was wondering if there is a way to preserve the filename to some extent. I like this as a general way to keep my folder sorted, but if it comes to an individual basis, there's two things I have that I enjoy.

1)Being able to identify something quicker by filename. This is more important to things like .webms, but certainly can come in handy with gifs and even .jps. I would imagine that there is a way to import the file name as a tag if you are importing via folder, but it wasn't an option as near as I can tell.

2) m-muh funny filenames. Which sounds trivial, but it's nice to be able to handle that. I assume that the reason the files need to have their own folder structure is for speed sake - which I suppose makes sense. But I would be curious as to how being able to keep filenames and folderstructure would affect performance at all.

NOT THAT I'M KICKING AND SCREAMING DEV DO WHAT I WANT, if a lot of people like it and it werks, I'm just curious as to if these are features that are plausible or not. This is an A+ program and it's awesome to see a dev take so much initiate in their software.

>>2389

I am glad you like it!

Thank you for this suggestion. For v201, I have added two checkboxes under file->options->gui to hide/show the thumbnail 'title' banner and the thumbnail page indicator, which appears in the bottom right. Let me know if they don't work for you.

>>2398

Typically, using system:everything on its own is not something you want to do. When you have 100,000 files imported, this sort of search will likely crash the program.

Showing all of your current tags takes a lot of cpu time to figure out, so I have reduced it to now to the autocomplete entry searches. If you have not tried it yet, when you type into the search box, the client will look for tags that match your input. It can get complicated, but simply, if you type in 'eva', the database will list everything like 'series:evangelion'. You can select your tags from that list just like those listed in 'selection tags'. This is the primary way of navigating at the moment.

Having a 'tag cloud' control would be nice, though, to let people quickly search by the most popular tags in their collection. Several people have asked for one. I have a plan to do it, but it'll be a slightly larger job, so I will do it when people vote for it on the next 'what big thing to work on next' poll.

By the way, you can set hydrus to continue where it last left off by selecting 'last session' from the session startup dropdown in files->options->gui. This can get laggy, though, if you close the client with a 5,000 thumb page open, as the session loader isn't asynchronous yet, so it will pause a little while on client startup as it rebuilds that large page. This may have been what you saw before, trying to preserve system:everything.

>>2399

I vacillate on filename (and external file/folder storage) support. I can appreciate that some people still want to use them in some way, but tags are so much more powerful, and that gap only grows with the number of files. I think the ideal solution is to move towards using 'title:' namespace more, although I don't want to lock anyone into anything. I'd like to have mouse hover trigger animated thumbnails in future, which I hope will help identifying movies better, and I'd like to add an explorer-like 'details' view for search results, which could figure out some useful identifying info to put in the filename-analogue column.

It is not uncommon that new users want to keep filenames, and it also isn't uncommon for them to change their mind once they are more comfortable with the software. I still think my help files could do with a better 'here's where I'm trying to head' section, but I'm still figuring that out myself, jej. You may or may not find your feelings changing as you do more searching inside the program.

If you have any programming/regex experience, you can import filenames as tags while you import using the pathname regex tagger (file->import files->add tags based on filename). As a new user, this dialog may be too complicated for you right now, but it is possible to easily set up 'filename:blah.jpg' on your local tag service as >>2401 suggests.

My chief regret about ditching filenames is your second point. My attached image has been amusingly named dark_souls_lore.jpg before, and I have tagged it 'filename:dark_souls_lore.jpg' on my ptr, but there is as yet no neat way to propagate that to Firefox as I upload it here. These jokes are goofy but important to our culture, and I don't want to nuke them. I think I will add an option for this in future, to autorename drag and drop exports based on the presence of a filename namespace.

>>2405

>>2399 here.

Yeah, the more I use it, I think the tag system is great. Once I learn to import by filename, I think it will go a long way once I learn to use regex a bit more. (Those are more for my .webms, since it's easy to forget what they are looking at just a frame.)

Yeah, I can see why you did it from a programming standpoint, considering how many files we are working with. It's certainly something I would LOVE to see, because silly enough, those trivial little jokes always give me a smile, and will honestly be one of the things that prevents me from using it to organize all of my files for now. Being able to export with a filename parameter would be honestly an important feature for me.

But thanks for the hard work, this is an AWESOME software and I'm glad you put in the amount of work that you do. Cheers devbro

>>2408

Being able to do this is actually a complicated problem, and something up for discussion for a while. A future version of the program may allow something this easy, but not yet.

The best way we have figured out so far is hydrus tag archives, which I describe more here (scroll down a bit):

http://hydrusnetwork.github.io/hydrus/help/advanced.html

The exact way that tags get applied in this way may or may not be confusing to you as a new user, so you might want to use the software normally for a bit first, tagging some files on your 'local tags' service and figuring out what namespaces are and so on. If you don't have a slow computer, you might want to sync with my public tag repository, which will eventually apply many of danbooru's tags without you having to do anything else.

Or if you are feeling brave, you can jump right in to tag archives. A slightly old danbooru (as of 2014-11, I think) tag archive is here:

http://www.mediafire.com/download/fffxc49x52cnuxd/danbooru.7z

Syncing with that will take a while but add a whole heap of tags. Let me know if you would like any help.

>>2410

Try going system:limit, which will randomly sample from the results (defaulting to an implicit system:everything if there are no other search predicates).

The db is ignorant of how the gui wants to ultimately sort the results, so I can't do 'get me the newest 1k files' yet, but you can add a 'system:age' predicate to emulate that sort of behaviour.

Also, you can set an implicit system:limit on all searches at file->options->speed and memory.

>>2421

>Try going system:limit

This changes everything!

>>2415

Oh yeah, this is perfect with a filename: category.

This is something I could easily see replacing most of my folder management. Yeah, I think in a future version when you have the time, adding the ability to export files based on a filename parameter (or any parameter somebody chooses, for that matter) Is an excellent feature idea, considering how messy the filenames get in general. (Big ugly string of numbers with meaning only inside of the hydrus client)

Guess I can use it to sort some of my other folders for now though. Cheers, everybody!

I've noticed that images don't really look that good when viewed in hydrus, compared to other image viewers like irfanview. It seems to depend on how zoomed in I am.

Is there a setting or something I can tweak to improve it?

>>2435

>>2455

Import folders can't do regex stuff yet. I'd like to add it, but wrapping all that complicated info in the regex dialog into a serialisable package will be a pain in the ass, so I haven't found time. For myself, I save the paged webcomics I like to a separate folder, and then once every two months or so, I do a batched regexed import that does creator and series under the 'tags just for selected files' bit as well.

>>2451

Or is not possible yet, but it is planned. Please vote for it the next time a 'big things to work on next' poll comes around. For now, please do something like:

1 girl

solo

* background

-complicated background

or whatever fits your purposes.

>>2466

Support for IQDB and other similar services is planned. I'd like to add it to the new 'suggested tags' control I'll soon be adding to the manage tags dialog.

>>2467

>>2468

Check out file->options->tags->apply all parents to all services, which I think defaults to off now.

>>2475

It goes 'xdg-open path' in a normal subprocess call. I really don't know about any of this stuff linux-wise–would you like me to add an option to customise the call? I could even eventually add per-mime options.

>>2476

I've looked into this, and I couldn't easily figure it out. When hydrus catches a DnD event, it asks it if there are any filenames, and firefox says 'yeah this bmp in your tempdir that'll expire as soon as the DnD event is over'. I presume there is a way to ask the event for the actual file stream, because Explorer can figure it out, but I think either wx (my gui library) isn't clever enough to get it, or I am missing which DataObject I need to be using.

I'll have another look anyway.

>>2489

>>2491

I'm sorry about that unhelpful error message. I'll improve it for next week–please let me know what it says in v203.

>>2493

>>2494

>>2495

That's frustrating, because the recent change was supposed to improve things! Do you find the ugly zoom tends to occur on particularly large images that are scaled down, or smaller images scaled up? Is it only when they first load, or after you zoom in and out a bit? Does it tend to be worse for jpgs or pngs?

If you still have that image and don't mind posting it, please do–or email it to me–and I can play with it my end.

>>2504

Here's the image. It seems to be caused by zooming out. 100% zoom looks right, as does zooming in further. It seems to be about the same for both jpgs and pngs

If I choose the zip version, can I run it directly from a USB-drive? Are there any downsides in doing so?

>>2506

>>2509

>>2510

I did a little work on this this past week, fixing a little zoom calculation bug and improving resize quality in certain situations, but I think the real fix here–including for the zoom-in lag/instability–will be when I upgrade to 'tiled' stream-rendering, where I'll always render from source (rather than sometimes rescaling an already scaled bmp in memory), and I'll only render what is on screen. This has been planned for a while, but it'll be a big job. When I get around to it, please let me know how it works for you.

>>2515

I'm not sure I completely understood what was deleted and what is still showing for you. I assume you are seeing autocomplete counts for a tag even though you removed it. Let me know if I got that wrong here:

If the tag doesn't exist for any files (i.e. its count is 0), it shouldn't appear in autocomplete results. If you do a 'local files' / 'all known tags' search for the tag, do any of your files still have it? What if you change the search domain to 'all known files' / 'your tag service'?

You may still have the tag on some deleted files. You can still search deleted files using 'all known files', so try searching for them and then removing the tag.

Or, if you are feeling brave, you can try going services->review services->your tag service->perform a service-wide operation and then selecting:

delete – specific tag's mappings

And then entering the tag in the search box and hitting enter to set it, and then clicking Go!. Make absolutely sure everything is set correctly on that dialog before you click Go!.

If you still get counts even though nothing is tagged, it may be that my autocomplete cache is miscounting. Let me know if this seems to be the case, and we'll look closer.

>>2520

>>2531

Great!

>>2532

It is absolutely designed to do that, so do try it out, and let me know how you get on! The only problem is if you plan to use a flash thumb drive, which will be unbearably slow at doing anything complicated or large on the database, so I wouldn't advise storing more than say 1,000 files or 10,000 tag mappings.

But if your drive is a usb 3.0 hard disk or ssd (like a WD Passport), then it should work basically as good as a regular internal hard drive, even if you encrypt it.

>>2542

I plan to rewrite my downloader engine to make it easier for users to create their own gallery streams to download from, but for now, if you are brave, you can mangle your way through the ugly dialog I created at services->manage boorus. I mostly wrote it for my own debug purposes, but feel free to look at how the existing boorus work and add your own. Feel free to ask for help.

Http/https usually depends on the gallery/thread url given. The client tries to stay on whatever you provide, but it will switch if the server asks it to.

>>2546

It is not supposed to increment its download count unless it gets the file 200 OK and it all parses from JSON and saves to disk and so on.

The service sync process does split downloading and processing into two steps, though, so are you maybe missing the 'soft' error in the first step, which will say something like 'could not connect to service to download stuff, will now continue with processing'?

If you go to services->review services->the tag service, what does it say there about this? It should say 'downloaded a/b - processed c/d'. Are the numbers there sensible? If you go to install_dir/db/client_updates/random_hex_which_you_will_have_to_infer, are there a timestamp_metadata.json files?

I would like to add a 'check all the current updates are good' button, but the current way services store updates doesn't neatly support this. I would like to improve the network protocol in the next few months, to save bandwidth and allow for signed updates, and this will be more possible and sensible then.

>>2547

You can't do it yet, but it is on my list. It'll be a slight pain in the ass to add, so it will probably have to go on the 'what big thing to work on next' poll.

This frustrates me as well–right now, I batch up my saved webcomic pages into a separate folder and import them with regexes every two months or so.

>>2561

Also, a lot of my pictures aren't tagged when I import them. Why is that? For example, that picture of the character "Palutena" should definitely be tagged, but it's not, as with most of my files. I am going through them putting local tags by myself, but a lot of them aren't even tagged from the tag repo.

Also, is this place dead?

For the second reply, I am too basic to understand, is there a way to reverse this? How can I understand these hashes more?

For the third reply, I don't think I understand what I'm doing.

Pic related. The bottom is what I did, and the top is the result. All of those tags show up in my 'local tags'. I don't get why I have all of those weird tags. Could you go through this once more? Maybe you can see what I did wrong. This is just a comic I'm practicing with.

If I understand this correctly, after I'm finished, I'll have a bunch of tags with 'page:' in my local tags? That seems annoying, but as you said those would be tagged with whatever I have and if I don't want to see them I'd search '-comic'? I don't know.

>>2577

>For the second reply, I am too basic to understand, is there a way to reverse this? How can I understand these hashes more?

Unfortunately, there's not much you can do. Even a tiny change to an image will change its hash by quite a bit, and it's impossible to track these changes. For me, the best solution was to make sure I always download the right images. For instance, where possible try using hydrus's download feature to rip images from boorus directly instead of saving them from a chan. This should guarantee you'll have images tagged.

>Pic related. The bottom is what I did, and the top is the result. All of those tags show up in my 'local tags'. I don't get why I have all of those weird tags. Could you go through this once more? Maybe you can see what I did wrong. This is just a comic I'm practicing with.

You did the regex and everything correctly. And if you look, all the images DO have the right page tags applied. Those garbage tags must have been added in the past for some reason. Just delete those.

>If I understand this correctly, after I'm finished, I'll have a bunch of tags with 'page:' in my local tags? That seems annoying, but as you said those would be tagged with whatever I have and if I don't want to see them I'd search '-comic'? I don't know.

Yup. Pic related is what it ought to look like.

>>2588

My current plan is to store it in a new class of metadata that'll have its own db tables. It would probably be hash pairs with a subtype of 'a is a higher quality version of b' or 'a is in the same image group as b' or whatever you want, and then when the db can produce dupe pairs in quick time, I'll throw the comparison up to a human who will be able to discern which relationship best applies. I'll then upload that metadata to a new repository and distribute it to anyone who wants it like tags in the ptr.

I can write new display gui or search system: tags or collect-types or 'please merge the tags for all files that have dupe relationship x' or whatever is good based on that metadata.

Anyway, as you say, it needs discussion, and we'll see how it looks after I get better search going. I suspect the scope of what we want will depend on whether dupes are ultimately one-in-fifty or one-in-fifty-thousand.

>>2582

Unless you have a special situation, I recommend you use a program like FreeFileSync to dupe the whole installation folder to a backup disk. (shut the client down first!) Then you have a completely running copy in case anything goes wrong. Updating the backup before you install a new version (I do it weekly, before I test the new release) is a great idea.

I killed my hydrus db, should I try to recover it or just move it to the side and start over?

I was updating the db from v202 to v204 when I had to reboot (nouveau's support for my video card isn't great, some times it locks xorg/the entire system) and now hydrus show pic related except the traceback isn't output to the log and the traceback dies if I close the "a serious error occured…" window.

fwiw: I have no local tags, and one local rating (a binary porn rating) but I've been trying to figure out how to get what I want without it.

>>2650

I think it is recoverable. Because of the unusual nature of v204, I split the update into two transactions, and I think the second 'ok that's all great' transaction didn't happen for you. Please go to install_dir/db and make a full backup, or at least a backup of your client's .db files, and then run the sqlite3 command line program. Then type:

.open client.db

SELECT version FROM version;

This is probably '203'. If so, type:

UPDATE version SET version = 204;

.exit

Then try to run the client as normal.

>>2682

Alright I got to ask a few questions before I bang my head against the wall some more.

1. Does Hydrus support multi-image posts on pixiv and tumblr?

2. What does pic related mean with the unrelated mimes? Does it have to do with issue #1?

Help would be greatly appreciated since I am hoping that this Hydrus Network program could save me the 3-4 hours I spend a day checking on artists.

Does anybody maintain a repo that is synced with the public repo, but perhaps keeps tags a little more general?

I've been having to petition even bloody filenames from the public repo :DDDDDDDDDDDDDDD. Honestly, I like the option and I contribute to the best of my ability, but it really does make me want to just remove the public repo from my system, and go through to organize them all myself and call it a day.

>>2670

>>2671

Matching booru tags to images is an ongoing and difficult problem. Boorus don't really offer a way to say 'here's and image, what tags do you think it has?'.

Although the PTR is a general purpose solution, when you are more experienced with the software, you may want to look into 'tag archives', which is some users and my attempt to solve the problem for specific boorus. You can read more on this page (scroll down a bit):

http://hydrusnetwork.github.io/hydrus/help/advanced.html

Basically, we've made some databases of scraped md5->tags for several boorus that you can sync to your hydus db. It works bretty gud, but can be complicated.

At the moment, hydrus can't track files outside its own db. It has come up a few times before, so I think I'll write a help page on it rather than repeat myself. tl;dr is that hydrus wants to get away from filenames and folders, so I don't want to write and maintain a heap of code to keep supporting the old inefficient way of finding stuff. Some people's opinions about this have changed after using the software for a while, so I suggest you try managing a smaller copy inside hydrus, 5,000 or so files, and see how you feel after a month.

>>2682

>>2694

The pixiv issue is manga/ugoira as others have said, but the tumblr one is new to me. Do you have any example links that are failing? Or just a tumblr username that you know produces these errors? It might be some weird video format I don't support yet, but if there are legit regular images that are getting their urls chopped or something, it may be an easier fix.

>>2683

Ah, when I changed the time delta control to a dialog-spawning button, the 'change' event got swallowed! Sorry, this will be fixed for next week! If you set the default in file->options->downloading, then any new thread checker pages will be correct on init.

>>2687

>>2689

>>2690

Better tag censorship/filter/hide-from-this-control options are planned.

You can't hide by number of tags, but I think it has been discussed before–that might have been where you saw it.

The biggest lesson of hydrus for me has been that the difficult part isn't getting things, it is the managing them. Weirdass tags pop up all over the place, and there aren't many good ways to yet deal with them, but it is definitely something on my mind.

Does anyone know of a way to save notes/annotations on booru images that have been translated?

>>2701

There's no way to do this yet.

>>2702

Unfortunately, I could not repeat this. I tried the download and everything downloaded and imported fine for me.

If you go to your subscription or gallery page and open the import status window like I've attached here and you did in >>2685 , what errors are you getting? If you get 'failed', try right clicking on some and selecting 'copy notes', which will copy the longer error message. Paste that here, and we'll look closer at what is going on.

You said you had slow internet, so perhaps that is doing it somehow. You might want to try right-clicking on the failures and selecting 'try again'.

>>2704

At the moment, I think it gets the two comparable lists, sorts them with ascending integers at the front and then words by alphabetical, and compares the lists one by one until there is a difference. So, for instance:

'series':

A = [ stargate, stargate sg1 ]

B = [ stargate, stargate atlantis ]

Would sort to B, A, as 'stargate atlantis' < 'stargate sg1', while:

'page'

A = [ 0, cover ]

B = [ 1 ]

Would sort to A, B, as 0 < 1.

I have thought about changing the word part of the ordering to be by word length rather than alphabetical, as supersets tend to have shorter names, like the stargate example above, or 'marvel' having higher search precedence than 'iron man' or 'captain america' despite being later in alphabetical order, but it hasn't been enough yet an issue to make me look at it closer.

Do you have any thoughts how you would like this sorting to work?

Has anyone added Derpibooru to their managed boorus yet? I'd like to copy the stuff to be entered so my client can scrape it properly.

How would I go about cleaning my local/personal DB of tags and mappings that aren't unique to my local DB/are already in the PTR or one of my synced tag archives? I imported a lot of tags from booru's to both my local db and the PTR before realizing this is generally redundant, but I also did some local tagging on these same posts I want to keep…

>>2707

Great! Let me know if it happens again.

>>2708

I'd love for hydrus to support everything any booru can do, but that sort of specific and semi-complicated thing is something I don't have time to add any time soon.

Floating boxes over other boxes is something html (i.e. webpages) does very well, but applications not so much. I don't know of any program that does it, but other Anons may have recommendations.

>>2714

I threw together this a while ago for someone else:

http://www.mediafire.com/download/ezrh9ibrn4ng6ee/derpibooru.yaml

Go services->manage boorus and then drag that yaml file onto the dialog–it should import.

As I remember though, derpibooru has unusual search url format and html/css, which means I couldn't figure out how to support multiple tag queries, nor how to parse tag namespaces.

The planned expansion of the hydrus gallery parser engine will support more kinds of queries and parsing rules, which will increase support for sites like this.

Have a play with it and let me know how it goes.

>>2718

>>2719

>>2722

This issue comes up often enough that I have written an extra bit in my help about it, which will be up in tomorrow's release. >>2722 has the main of it, in that human-readable files and folders are for many reasons generally bad for keeping track of 10,000+ files, so the program abandons the attempt.

Please try hydrus for a few weeks with a few files. Many people have found they don't miss files and folders once they are familiar with the program.

>>2720

Shit, thanks:

>>2683

Are you backing up or restoring a backup?

The 'make a backup' job is supposed to throw up a popup message saying 'backup up db', 'copying files directory' and so on, but I think the db backup thread is choking or blocking the gui thread, so it doesn't have time to respond.

Some of the larger jobs are still bad at yielding time, and the gui accidentally locks itself until the job is complete–I'll give this code another pass, see if I can reduce the hanging.

For now, the solution is unfortunately just to wait for the backup job to finish. If you want to make sure the client is still doing work, you can watch is do CPU and HDD in task manager or just watch your chosen backup folder populate. Although the initial backup run will take a while, updating it in future should be much quicker.

>>2723

There are some ugly advanced tools for this under services->review services->your tag service->perform a service-wide operation. You can copy or delete certain tags or namespaces, do one-time union or differences with tag archives, and create your own tag archives from services.

You sound like you know what you are doing, but obviously be careful with this dialog, as you can make mistakes easily. The big operations it can do can also take a long time as well.

I think what you want to do is:

Open the advanced dialog for the PTR.

Export everything to a pretty huge hydrus tag archive somewhere.

Open the advanced dialog for your local tags.

Do a one-time delete using the HTA you just made.

This should do:

local tags = local tags - PTR

Which is I think what I understand you want to do. Although if you don't have any local tags you want to keep, you could just go [delete all mappings] for your local tags.

>>2724

Thank you

>Are you backing up or restoring a backup?

restoring, the pop-up messages only show up when I create a back-up, but not when I restore it.

I ran a couple of small-scale tests, and I think that the Restore Back Up works normally, but it doesn't tell me that it ended and everything stays locked until I end the client with the task manager

>>2724

Much obliged, good sir.

It works with single-tag queries, and I think I see what you mean about the multiple-tag query difficulty. If I get something to work, I'll tell.

Derpibooru's API doc is actually on their page https://www.derpibooru.org/pages/api , in case you didn't see that already.

>>2726

Is there a way to explicitly tell hydrus what the url search string is?

For example, I want to search for the artist kevinanso and only see safe results. The textbox-input would be

safe,artist:kevinsano

and the url

https://www.derpibooru.org/search?sbq=safe%2Cartist%3Akevinsano

So could I tell hydrus to input

safe%2Cartist%3Akevinsano

as an entire string into the url and advance through the results otherwise normally?

That would allow me to scrape all results of a single search, which is currently good enough for me.

>>2725

Ok, thank you, I will check it out for next week.

>>2726

>>2727

Yeah, you can fudge the url in the manage boorus dialog to do a bunch of funny stuff. I made a 'gelbooru favourites' parser a while ago that did that. You might or might not be able to put 'safe%2Cartist%3Akevinsano' through, as I think the % symbols might get url_encoded. You might want to try something like:

export the booru to a yaml file

rename it as booru_safe.yaml

import it to make a dupe

change the search url to be something like:

https://www.derpibooru.org/search?page=%index%&sbq=safe%2C%tags%

and then load it up in a page and type 'artist:kevinsano'. That page arg might not be correct, though, so you'll have to play around.

%index% is replaced by the page number as the booru parser progresses

%tags% puts in whatever you put in the search box, with spaces replaced by the choice in the 'search tag separator'

It keeps incrementing %index% until no results or 404 is found, or you cancel it.

The new version of the parser/downloader engine will allow more characters as the search tag separator and so on, so for now you'll have to hardcode them in the url.

Note if you have really weird tags, you can also do:

https://%tags%ww.derpibooru.org/search……. (complete url with %index% somewhere)

Where %tags% is just a single 'w' character in www. Put 'w' into the search box, and your tag-hardcoded url will fire through the pipeline as if it were legit. Again these are just hacks until I can add better classes of downloader like search-string-less queries and so on.

>>2735

In your install_dir/db folder, do you have a client_thumbnails directory? And under it, are there 256 folders from 00 to ff?

If so, you should be able to go database->maintenance->regen all thumbs. It'll take a while, but I think there is a progress popup.

If you don't have that directory structure, check out your client_files so you know what you are looking for and then extract and run a fresh client somewhere else to generate the base thumbnail directory structure (you don't want to manually create 256 folders!), then copy those folders over to your real install.

You also want client_archives and client_updates. If you sync with a repository like my ptr and don't have anything in client_updates, you'll want to go services->manage services->your repositories->reset processing and download cache. I think that'll recreate the complicated folders in client_updates and repopulate them with update files. If it doesn't work, you'll have to completely remove and then re-add the services manually.

Let me know if it looks like you need to do anything else.

>>2739

>>2741

Yeah, check out manage services again. You can now set number of 'stars' and change whether 0-out-of-x is permitted. Your original scores will be preserved and scaled if you change the rating (2/5 will become 4/10, for instance), because the db stores it as 0.0->1.0. I call it a 'numerical rating', under local services, as opposed to 'like/dislike rating'.

And use system:rating to filter your search results based on rating.

>>2732

Sweet, I'll try that. Thanks for the detailed answer!

>>2754

Both of these things are planned, but they'll be a little way off, the second more than the first.

When I next overhaul the downloader/parser engine, it'll be better able to parse non-tag info from boorus like rating and source and so on.

File-relations is likely to be the next type of metadata the server can store, as I'd like to have everyone's 'A is a dupe of B' and 'A is higher-res than B' and 'B is A but has a watermark' and 'B is an alt of A' comparisons shared efficiently. The first step in this will be a more efficient dupe detecting algorithm, which I hope to be working on in a month or two.

>>2771

thanks

>>2823

The client can't do single booru page parses yet, but I would like to add it in the next downloader engine overhaul.

I'd like it so you just put the url in, and the client goes 'ah, I know how to parse that site' and figures out the full-size file and offers you the tags and ratings and whatever.

>>2830

The client can't do OR searches yet, but I would like to add it in the future.

There is basic 'session' support to save and load your current pages. It isn't perfect, but check out pages->sessions, and also check file->options->gui->default session at startup, which can be set to 'last session' to automatically carry on from where you left off.

>>2840

This isn't currently possible, but it will be when I next overhaul the downloader engine to allow any user to write a parser/downloader for any site.

>>2841

In hydrus, tags are separate from files, so they still hang around even if you don't have the files in your db. This is how tags from the public tag repo appear when you import files, for instance.

I started writing out a way to do what you want manually, but I realised there isn't a neat way to do it, so I've added one for next week. In v208 go:

services->review services->local->tags->local tags

perform a service-wide operation

[delete] [all unnamespaced mappings] (make double sure you have this set correct)

Go!

And that should do it. It may take a while to complete, and you'll have to refresh your current views to see the change.

This will delete local tags' unnamespaced tags for all your files. If you only want to remove the unnamespaced tags for some files, then you'll have to do it manually (i.e. select the offending files, hit F3, order manage tags tags list by 'lexicographic (a-z) (grouped by namespace)', select all the unnamespaced tags with a shift-click, hit enter, and then go through a whole bunch of add/delete choice dialogs), or you can wait for the 'service-wide operation' dialog to support file selections, which will come in the following weeks as I work on the suggested tag control.

>>2846

I'll be working on manage tags over the next few weeks as part of the suggested tags control, and I'll definitely clear up the uglier workflow. Let me know how it works for you when it is done.

>>2849

>>2850

>>2851

It has been a while since I wrote the danbooru parser, so I can't remember this very well, but I think it used to get the full-size. They might have changed their html, or I might be remembering wrong, but I just had a look, and the way they present the 'original file' links is too complicated for my current parser engine to understand. When I next overhaul it, I or someone else will be able to write a new parser that can get the full-size image from danbooru. I would say stay away from the hydrus danbooru downloader for now.

There is no good solution to duplicates, but I will be writing a fast dupe search algorithm when I am done with the current 'suggested tags control'. That will be the first step in dealing with all the different sorts of 'file siblings' there are in all our collections.

At the moment, the hydrus downloader engine isn't clever enough to fetch md5s from boorus (another thing to update!), but it does check md5 for 4chan/8chan threads, which work on a json api where that info is easier to access. The hydrus client does remember previously visited urls, however, so if you have searched for 'shinji' and downloaded a bunch of files, it will skip over those if it sees them again when you later run a search for 'evangelion'. If you download the same file from gelbooru and sankaku, you are wasting bandwidth at the moment, so don't go nuts.

When the client imports a file, it puts it in a temporary location and calculates its sha256 hash, which is very like md5 but newer and more robust. If the sha256 hash is already in the database, it doesn't overwrite or make any other changes, it just returns 'already in db' status to the importer window, which will move on to the next file. You don't have to worry about this process ever lagging, as the hashing process and db check is very fast, typically a handful of milliseconds, even in a db with a lot of files.

Let me know if I haven't explained any of that well.

>>2853

Yeah, you can set up a one-time import folder. Go file->manage import folders, which is typically for setting up regular background imports, but can be useful for big cases like yours where the gui lags out. Let me know if it gives you any problems.

Note you can also go Ctrl+A to select all thumbs and then right-click->remove selected to clear out the media from view on a big window (although this operation may be laggy itself just from the number of files currently in view). The lag is usually from the cascade of merge and gui-update operations triggered by every new file, which increases in complexity with the number of files in a page.

>>2863

Yeah, I actually only wrote a 'recover from sankaku bombing out at 1,000 files' fix the other week, as the error page is unusual and subs weren't handling it gracefully.

I'd like to add (arbitrary login form/cookie memory) support when I extend the downloader. At the moment, it can basically blindly fetch a few iterative urls and look for a handful of html tags by class or id.

EDIT: The specific login stuff like for pixiv was an attempt on my end to hardcode that stuff. It was taking up too much of my time to add each new site and then keep up with old ones changing, hence the new plan to make a system to let users create their own generalised parsing and login systems.

Is there a hotkey to copying something to your clipboard? If not, would you consider adding something like that? It's a small feature, but it would be helpful for a few things here and there.

>>2882

Sure, what would you like to copy? If you hit Ctrl+C on some selected thumbnails, it'll copy the files to clipboard for quick paste-export, but I don't think I've added it for much else. Were you thinking copying tags, or something else?

>>2884

Yeah, >>2887 is correct. If you just want a simple viewing experience, keep your 'paged' media in folders for now. I expect to add support for cbz-type archive formats in the future that'll make it easier to handle chapters and volumes inside the client.

>>2889

>copy files to the clipboard

Really? For me, all that happens is it copies the file path to the clipboard, rather than the file itself. So if I go into say paint to try and paste the file, there is no image on the clipboard. Paste into a text box, and you get the path file.

What I had in mine is just being able to take images and gifs, put them in the clipboard, and paste them like you would using any normal Windows dialog. (Or I suppose linux and OSX for that matter). Although the ability to assign it different actions for tag work might be pretty cool. I have a few things where the usual methods of uploading files (drag and drop) don't work. Also, wouldn't hurt to be able to put a file on your clipboard and upload it when you need it.

>>2895

Ah, if you want to copy image data, open the image in the media viewer, right-click, and go share->copy->image, which will put a bmp on your clipboard that you can then paste into photoshop or whatever. You can also do this from the preview window in the bottom-left.

'file' on that same menu copies the file(s), which lets you paste to an explorer window or an 'open file' dialog.

'path' on that same menu just copies a bunch of newline-separated paths, for pasting into a script or whatever.

At the moment, ctrl+c does 'file', which is the most flexible and fast clipboard datatype I can generate.

You may also be able to import to a program just with a normal thumbnail drag and drop.

>>2902

Subscriptions can't do this, so you'll probably want to run a one-time download through a normal download page with higher or 'no' file limit. Any files you already have through the subscription won't have to be downloaded again (and will skip over very quickly if you uncheck 'get tags even if file is already in db'). Since this is a big job, you probably want to keep an eye on it. You can also check the import status window (hit the button with the list window icon) to make sure everything is happening how you want.

>>2903

Not yet, but this will be possible after a future overhaul of the downloader engine.

>>2973

I think this is what you're talking about. Add a line to the list of available sort orders with any namespaces you want. Any namespace that is in a sort order line gets added to the list of collection options. Just remember to close and reopen hydrus when you add new lines so they become available.

I get this when trying to open Hydrus now, it always worked before. Google gives no results. I already tried to update/re-install, it does the same thing.

>>3016

Did you recently update from <v186? I think that's the error people were getting in the 32-bit->64-bit change. Please check the release post for v186 here:

>>1589

There are special instructions for that release--you will be doing a 'clean install'.

>>3018

You can experiment with page: tags, like 'page:1' 'page:23' and so on, which can be more easily added with filename regexes as you import, but really hydrus is not very good at doing paged media yet. Unless you are an advanced user, I advise you to keep paged media out of hydrus for now. It should become easier when I add cbr/cbz support.

If you do want to give it a go, I've written some about this here:

http://hydrusnetwork.github.io/hydrus/help/advanced.html

>>3029

That depends on the booru. I just pass what you type on to their query URL, so if they support negation like that, it should go through ok. You may have to bravely experiment.

A future version of the downloader engine will support this hydrus-side, but it'll be a few overhaul generations off yet.

>>2231

Alright, tell me:

What the fuck is the Hydrus Network?

>>3041

I searched a bit more, and this error also came up earlier in a different context. It could be your anti-virus wrongly quarantining one of the dlls. Please check >>2330 and culminating in >>2392 .

Do you have anything like 'SAXReaderNotAvailable: No parsers found' at the bottom of your install_dir/db/client.log?

If you disable your anti-virus and download and extract a fresh v210, does it run?

>>3044

It's an application that helps Anons manage their large image/swf/webm collections. It is basically a booru on your desktop. It can do a lot, and can get as complicated as you like. I've written extensive introduction and how-to guide here:

http://hydrusnetwork.github.io/hydrus/

Let me know if you have any questions about how anything works!

>>3045

I think my ptr is the biggest tag repo. It is what most users are working on, and remains (like everything else hydrus!) a work in progress. It typically gets about 150k new mappings every day.

I have plans to increase file-tag coverage with better duplicate searching and better tagging workflows, but I think trying to get 100% coverage through human action is probably unachievable. I think the years-from-now future will have us using the existing millions of tag mappings to train auto-tagging algorithms to tag our own collections for us.

>>3050

>1-Is there an option to open the images externally with double-click but quick-view them using the built-in viewer by holding down the right-click button (like Xyplorer does)?

Yes, you can also use drag and drop to upload it to a website for example.

>2-Are the thumbnails cached or generated on the fly?

Cached

>3-Can I save multiple sessions?

Yes, on Pages -> Sessions

The new tag dialog doesn't seem to be remembering its size and location, it always comes up on the left side and way too large.

>>3096

Same here… do old versions work on x86?

>>3050

I am glad you like it.

In addition to >>3053 's correct answers:

If you are interested, thumbnails are stored right next to their parent files, under install_dir/db/client_files/prefix/[hash_hex].thumbnail. Depending on the source. it can be either a jpg or a png. .thumbnail is a 200x200 'original' that is generated on import, and .thumbnail.resized is whatever you have your thumbnail dimensions set to in the options, and is generated on first viewing. The first time you view files can be slightly slow, especially if you have just changed the thumbnail size, but things seem to load pretty quick after that. Rendered thumbs are also held in a memory cache as compressed bmps so updates the main view can be nice and quick.

>>3052

There's no 'hide these by default' mode yet, but I have extensive plans to add this in future by partitioning the 'local files' domain into 'nsfw/sfw' or whatever other categories you like.

>>3061

Failed files are usually because of deficiencies in the libraries I use, which are usually not clever enough to deal with slightly broken files.

If you right-click those 'traceback' errors, you should get 'copy notes' or something, which will put the whole error text into your clipboard. Feel free to email me/post here the original file/url and the error, and I'll see if I can fix it on my end.

>>3064

Tag workflow is definitely something I want to make better, with mappable shortcuts and more. The suggested tags control I am currently working on should be another step in this direction.

The database to hydrus is a simple and open, written in SQLite, and I have written an portable db type called a Hydrus Tag Archive to make it even easier to move this stuff around. If I get run over by a truck tomorrow, it would be very simple for anyone on here with simple programming ability to write a script to export anything to any external format. I've had bad experiences with obscure dbs before myself, so I want to make everything as accessible as possible.

>>3065

I expect to add various hydrus 'censorship' controls to halt undesired imports, but it'll be a while off. I can't give firm dates on any feature.

>>3070

Not yet. I've experimented with 'streaming' results, but some edge cases add a lot of complexity, they slinging lots of bits at a live gui tends to be ultimately a lot slower than chucking the whole lot at once. If something is particularly slow for you, please check out how to create and send me a profile, as detailed here:

http://hydrusnetwork.github.io/hydrus/help/reducing_lag.html

>>3076

>>3096

>>3105

>>3107

Unfortunately, x86 is no longer supported for any platform. As the libraries I rely on drop support, I have to as well.

>>3088

Yeah, this is unreliable right now. I think the thumbnail launched one does, but the media viewer one doesn't? Anyway, please hang on for this code to stabilise. I expect to write a whole new listcontrol to manage detailed 'remember' options for all the new dialogs and frames working on the new system. It'll have lots more variables to edit, currently looking like:

name, remember_size, remember_position, last_size, last_position, default_gravity, default_position, maximised, fullscreen

>3108

1) Not at the moment. You can set a minimum under any normal download page's file import options, but not a maximum. Also, this minimum test is applied after the file is downloaded. For now, be careful firing off downloads for 1.1GB webm megathreads if you are short on bandwidth.

2) I tend to default everything to off, so people can set things specifically how they want for their situation. If you would like to change the default, check file->options->default tag import options. You can add different tag preferences for all galleries, or just specific ones.

3) Did you send the files to trash, and then trash cleared them out to your OS's recycle bin, and then you told your OS to undelete them? If so, I presume they were sent back to your client_files directory somewhere. Unfortunately, the client is not clever enough to watch what happens to the recycle bin once it thinks it has deleted a file. It doesn't notice if a file comes back into client_files from an external source. If you would like to gather these files together to reimport them, try going database->maintenance->clear orphans, and choose to send the 'orphan' files to a folder somewhere.

4) File->tag lookup on a booru is unfortunately technically difficult to do in a general way. If you search for the files again with some namespaces selected and 'get tags even if the file is already in db' selected, the client should remember the files it previously saw and not redownload them–it'll only fetch the appropriate html pages so it can parse the tags.

5) Depending on how the gallery works, there should be a 'stop' button somewhere to stop fetching new results to put on the queue. The queue can be edited from the 'file import status' button (right-click and set files to 'skip' if you don't want them), although there is currently no easy way to see results nested by the original query, so if you want to abandon a query, it is best to open a new download window.

6) I'll keep working at it. My immediate plan, as suggested by another user, is to add more tooltips, to keep this stuff lightweight.

7) Is this for a booru? Most of the booru downloaders only grab creator/series/character/no-namespace tags. Can you give me an example link for a file that has medium tags?

>>3110

If you are comfortable with regexes, you should will be able to do this. Drop some of the files onto the hydrus client's gui and then hit 'add tags based on filename'. That path-tagging dialog has simple and advanced tabs–the advanced lets you parse by regexes, which should let you split up by '#'. I'm not great at regexes, but see my image for an example that I think works.

Otherwise, use a proper renaming program (I use Batch Renamer, although this also uses regex for most of its magic) to more easily rename your files and use the simple hydrus tab to parse by foldername or whatever.

>>3112

>>3113

Muh forgotten/typo'd BO trip.

>>3129

From what I understand you're looking for custom filters (select some files -> right click -> filter -> custom filter), you can setup simple shortcuts for pretty much any sort of tag or rating action, and even have different setups depending on what you want to work on at the time.

Sorry for the late replies here. I'm just catching up now.

>>3120

I am not sure why clear orphans didn't find anything. Could your OS have undeleted the files to another location? Could hydrus have re-deleted the files?

I will add tag-type-medium to the Sankaku parser, thank you.

>>3128

>>3108

I messed up the post link. Check >>3113 if you missed it.

>>3129

I'd like to convert the current shortcuts system so you can map as much as possible to as many actions as possible, so you can set '1' to 'rate selection as -good-' or whatever. It is in my todo list, but it'll be a big job, so I can't estimate when I will fit it in. For now, check out custom filters like >>3131 suggests.

>>3130

There's the thread watcher, have you tried that, or does it not do hat you want? Go F9->download->thread watcher, and then put in a 4chan/8chan thread url, and it'll check it however many times you tell it.

>>3137

Double-click usually removes an entry in any list of tags.

>>3141

Wildcards and siblings/parents are complicated behind the scenes, so I think their behaviour is undefined right now. Maybe some siblings get looked up, but I can't guarantee anything.

You can hide the thumbnail overlay/banner at options->gui, and you can define new namespace sorting scemes under the sort/collect options panel, although this is currently a little non-user-friendly. Let me know if it gives you any trouble.

>>3142

That's an interesting thought. I don't think it would actually be too crazy–it might throw an exception as it tried to delete a file that was already open or from the OS trying to copy a file to its source destination, or it might go through ok and just have a record to a missing file that would need to be cleared up with database->maintenance->check file integrity. One of the nice things about hydrus's language, Python, is that it is generally fairly crash-proof. Most problems just throw an exception.

I hope that anyone bright enough to get hydrus going and also poke around client_files probably knows that fucking around with db integrity without a backup is a bad idea, and anyone who doesn't could probably do with the lesson in getting it wrong!

>>3145

Not yet, but this is a big issue that I have a lot of future plans for. I'll be writing some better duplicate search code after I'm done with the current suggested tags control, and when everything eventually shifts over to neural network tagging in a few years, the problem should disappear completely.

>>3148

It should inherit recursively, although now I think of it, I think it'll only be comprehensive if you do A->B before you do B->C. I'll make a note to check this, thanks.

>>3150

Clear orphans checks for unexpected files in the db structure. Over time, some code failed to delete certain files or created little stubs on a failed import and so on, which left some cruft in the db. Clear orphans is mostly a thing I can tell people to run after I fix a problem. If you haven't noticed any problems, you can ignore it.

Vacuum is a full internal database defrag. By default, the client does it automatically every few days, but again you can force it if you want. This again is mostly debug stuff.

Analyze calculates statistical information about the frequency distribution of database indices to help the query planner do searches effeciently. If I ask the db for all rows that have A=1 and B=1000, it may be quicker to search for A first or B, depending on how many unique values there are in the A and B columns. Again, the client does it automatically, but you can force it.

There are a couple of ways to delete tags for files no longer in your db, but for most users, this is more hassle than it is worth. Are you seeing the tags appear somewhere you don't expect?

>>3156

I'm sorry you are having this problem. I think it means that a dll is not loading for you, which has typically meant a false positive from anti-virus, although perhaps your OS has a different problem. If you extract a fresh v209 or v210, whichever last worked for you, does it still boot ok?

Do you have anything unusual in your system, like an odd graphics card? Which OS are you using?

>>3174

Good question! I think it does. The client sends the file to the db to be imported, and if it gets 'new file - successful' or 'already in db', it tries to add the tags. The import folder 'action', be that move or delete or what, happens after that.

>>3186

(>>3156) I tried to extract multiple times a fresh v209, v210 and v213, it always gives the same error. Pic related for my specs. What do you mean by "unusual things" besides hardware problems ?

I've been using Hydrus since February, until then I had no problem, and one day when I launched it and it just refused to boot with this error. I tried extracted v213 on my current installation and booted Hydrus on a virtual machine with fresh W7 installed, DB upgraded fine, everything worked without problems, so it's not hardware, Hydrus bug or a DB problem.

>>3190

It won't get them all. It'll do the first 50, and then any new ones after that.

For this sort of situation, I recommend first doing a single big manual download through a normal F9 download page. Get the 2000 files while you are sitting there and can pause and verify everything is ok, and then set up an (initial file limit = 5 or whatever) subscription to manually keep up with that good search in future.

>>3191

Some people have had dll-related problems with some Nvidia cards recently, so I thought maybe that could be doing it, if you had recently set up an unusual piece of hardware, or a 16-monitor setup or something. Your system looks fine though.

As I understand it, that error is because something basic, like a dll, could not be loaded. Until now, that has typically meant an anti-virus program has blocked it or similar. I can reproduce the same pyi_rth_twisted error dialog if I remove _socket.pyd from the install folder and try to run client.exe–you may have a problem with the same file, or it might be another. My _socket.pyd is 24,576 bytes. Do you have that file in your install folder, and is it the right size? Another user reported that his anti-virus was silently removing a file as he extracted, when he didn't notice–could anything like that have happened? A Windows v214 .zip extract should have 115 files in the base directory totalling 67,316,109 bytes–how about you?

>>3196

Not yet, but it has come up several times. I'll add something to the next overhaul of the downloader engine.

>>3201

Please check the getting started guide here:

http://hydrusnetwork.github.io/hydrus/help/index.html

And let me know if you have any problems.

>>3214

It got moved during some update, what used to be the admin menu is now located under:

services -> administrate services -> <service name>

It'll only show up if you have at least one server setup properly already.

>>3216

Not really. Hydrus supports importing audio files, and you can organize them by tagging them with string metadata but, this involves making a "copy" of all the audio files into hydrus' own database which would render your folder/file structure useless.

It'd be useful if you're looking to categorize all your music in a deeper way than folders/files can support. Hydrus has no support for custom sorting though so playlists may be a challenge.

>>3218

You can use the booru downloader which will let you batch download anything with a specific tag, stock hydrus has no entry for derpibooru in the downloader so, check here >>2724

If you're looking for individual image downloads you should probably use boorutagparser (>>1914) although vanilla boorutagparser has no support for derpibooru as well. Lucky for you I modified btp to support derpibooru, but never sent a pull request. Here's a pastebin (http://pastebin.com/MGn37dwD), anyone can feel free to pull request this themselves if they like.

A very dumb and basic question, but I am at a bit of a loss as to how the autotagging function works. I downloaded all of the tags from the tag repository, but am unsure how to incorporate them into my images.

>>3225

Ha, shit, didn't notice that as I don't watch tags or use any fancy features, makes sense as to why it missed those tags because of the way it grabs generic tags.

As for downloading the short filename you could change in doDownload a[href*="/img/download/"] to a[href*="/img/download"][title="Download this image at full res with a short filename"].

This is only going to work if you use the site in english (I'm assuming derpibooru has localisation) and if they never change the title text, derpibooru doesn't seem very friendly to scrapers so that seems to be the easiest way to select for the short download link.

You could also probably use a selector that selects last of type but then you're relying on element order which may not always be reliable. Or it may be possible to select based on the contained text but that would require a bit of a change to doDownload overall.

What am I doing of wrong here?

>>3263

At the very least you'll need one namespace to collect by, like the title or artist, and a numbered namespace to sort by. Volume/chapter values make it a lot easier to keep large collections in the correct order, though.

Not sure about the booru, but if you're lucky and the filenames contain the necessary information, you can just pull those via regex on import

>>3209

>>3263

>>3210 is correct about page: tags, although I'm not happy with how collections and sequenced media works right now. I suggest you try it (check the gunnerkrigg example here http://hydrusnetwork.github.io/hydrus/help/advanced.html for inspiration) and see if you like it. I find it a bit awkward, but you might like it. Any feedback would be appreciate. I expect to add a multi-page cbr/cbz-style forrmat in the future so you can sling chapters/volumes about as individual objects.

I expect to add support for multiple 'file domains' in the future, at which point it will be possible to have completely separate nsfw/sfw domains and so on on the same client. When I do this, I'll be able to have separate storage locations for them, but at the moment everything is stored randomly across the locations in options->file storage locations. If you want to make sure alll your nsfw files are stored in the external drive, you can either store everything on that drive or run two separate clients for sfw/nsfw until I write the file domain stuff.

>>3227

>>3228

Great!

I don't have any good ideas right now on what this ultimately was, but let me know if it comes back, and we'll attack it again.

>>3231

>>3232 is correct. The PTR covers about 4 million files, but even then that won't cover everything you have. 15-20% coverage is a good number at this stage, although it depends on whether you have more popular or obscure images.

Try importing my attached image--it has a couple of tags on the PTR. Also, try opening a new search page (F9->files->local files) and typing 'series:*'. This will show a whole bunch of tags and should get you started on seeing what kinds of tags you do have matches for and how to navigate and browse using tags.

>>3233

Are you the person who emailed me a couple days ago?

>>3234

Check out file->options->tags. You can set up a list of favourite tags which will then appear in the manage tags dialog in a new column. You can double click them to add them without having to type them in.

>>3236

Unfortunately, permissions are a mess right now. I've just been talking with a guy in another thread about this, here: >>3250 . I'd like to add better permission and petitions for all content type in future, but there's no way to have 'add' petitions for tags for files yet.

>>3241

If it is any help, every file import is a complete transaction which I'm pretty sure requires a full disk sync (i.e. waiting for your hdd platter to physically rotate and write the data and report back 'ok, that's flushed'). If you are on an HDD rather than an SSD, this puts a 4-8ms or whatever average minimum latency on every change to the hydrus db. Maybe that stuff can get bottlenecked at the HDD hardware level, commiting pages from the internal hard drive cache or something?

If you have any files that are 'already in db', those typically don't even need hdd access, as they can get that info right off the os disk cache and don't need to write anything (and if you only just downloaded the files, they may also still be in ram, so the file hash can be generated super fast), so they can come in real quick in a cascade until the client hits a truly new file.

>>3244

It'll try to mirror your existing client* stuff in your db folder (A) to the chosen backup folder (B) with the following rules:

If file in A and not in B, copy it.

If file in B and not in A, delete it.

If file in A and B, overwrite if it has a different size or modification timestamp.

Backing up onto an existing backup is a good idea, and will be a lot quicker than generating a new one every time.

I personally use FreeFileSync, which basically does the same thing but is moree powerful and flexible and can do any group of folders you want.

>>3262

I assume that last line means something like 'connection refused'? Maybe you have a firewall problem? Make sure both client.exe and server.exe have permission. The server initially tries to host on port 45870.

>>3293

>I assume that last line means something like 'connection refused'? Maybe you have a firewall problem? Make sure both client.exe and server.exe have permission. The server initially tries to host on port 45870.

I'm running it on Arch Linux using the AUR package. Tried it in my laptop too and got the same error. My ports are open and no other program is using the 45870 port either.

Also, this isn't the bug thread but I guess I should try to get some help about it too: why can't I save options? I click the "Apply" button and it shows this error message and trackback.

>>3300

If you check out options->media, do any of the media viewer mime handling dropdowns start blank for you? It is possible they are initialising wrong in certain cross-platform cases–I've patched this for next week, but I'd like to know I'm understanding the problem correctly.

>>3323

Sure, you can do an initial backup to the hdd and then copy wherever. Whatever you are most comfortable with. Hydrus is super portable–just keep the db folder in one piece somewhere, and you are good. There are no registry entries or other nonsense you need to keep track of.

The db backup in hydrus is lightweight and will only copy the files it needs to based on the changed modified/size metric. It is generally as or more efficient than making a copy in Explorer. If you want to be gentle with your external drive, I recommend you use hydrus or FreeFileSync to maintain an existing backup, as it will not overwrite any unchanged files.

FreeFileSync only does the moved/renamed file detection if you turn it on–I've never used it, so I don't know how effective it is. I have it set up to do syncs based on the same modified/size metric. A typical backup of my laptop client, which is about 800,000 total files, takes a couple of minutes to scan and compare and maybe five minutes to sync, which typically involves overwriting a thousand files or so. I find it very convenient and use it to make full weekly backups of my network and code folders and everything else I have.

>>3325

Not yet, but I'd like to add 'namespace' siblings in the future to solve exactly this problem.

>>3331

I have zero VM experience, so I can't offer much help, I'm afraid. That bug is almost certainly a dll issue, so I presume some dll or other library load requirement is getting boshed. Maybe some file is getting loaded and cached in the first case, and then when you later try it, a different file is being loaded?

>>3333

Sorry, by check out, I mean just 'look at'. I presumed that your dropdown choices on the media page were initialising blank for a particular reason, and so on an ok button, they weren't able to transfer their 'choice' to the options data object correctly, so I wanted to make sure you saw blank dropdowns on dialog init. By your error, I thought you were able to open the dialog–is that not true?

>>3334

>>3335

Yeah, simultaneous access to a db is unsupported, and I generally would not recommend opening a .db file across a network, as SQLite likes to have some hard drive exclusive/sync stuff that networks generally pretend to provide but don't.

You can run a client database locally and tell it to store client_files over a network connection very well, but the server doesn't support putting server_files in a remote location yet.

>>3340

Here's my options->media dialog.

Is there a way for me to avoid sharing my external IP in order to share my files? Can I set hydrus up on some kind of anonymous hosting privider?

>>3368

Okay it's working now and I can view it in my browser.

But exactly where is the file stored? Is it being served from my harddrive or from the repository?

Exactly what is the repository? Is it the devs own hydrus server?

Can someone else test this and confirm that it works? QmNUumH6QpMSdDsDA6WnQZpZp6QKdeUtKRnn3BFPJg3EwQ

Is there a reason that find similar doesn't work reliably with gif's?

So say I dual boot - can I share the same Hydrus network library between both systems easily without having to constantly import changes manually?

>>3343

In trying to get the Ubuntu version fixed yesterday, I had some luck getting a debug build to report these errors better, which hasn't always been reliable. Nonetheless, here's a debug Win build of v217:

http://www.mediafire.com/download/vjr3v3ce27r5bod/Hydrus_Network_v217_debug_-_Windows_-_Extract_only.zip

My Windows python environment changed this week, so your import problems may have magically fixed, but if they haven't, please extract it and run it from a command window. It should write a bunch of debug stuff and hopefully give a more verbose load error for whatever dll it is missing.

>>3348

Thanks. That's what I thought was happening–is it fixed for you now?

>>3369

Please check here for my thoughts on this:

http://hydrusnetwork.github.io/hydrus/help/faq.html#external_files

I'm sorry for the inconvenience! Please give the hydrus way a try, and if you don't like it, or only like it for a few kinds of file in your collection, no worries.

>>3375

I used to have it work for everything with a thumbnail, but there was a lot of first-frame-is-black screen spam that caused slight problems elsewhere, so the db now only generates the needed perceptual hash for jpgs and pngs.

I'd like to reintroduce it for animations when I have animated thumbnails. I might have some kind of 'find most interesting frame to represent file' filter as well, which will help collapse the deeper search domain.

I'll also be working on a much faster dupe searching algorithm when I'm done with suggested tags.

>>3407

As >>3413 says, there shouldn't be an essential problem with this–the db is platform agnostic. I'll be adding a db-location parameter to the exe in the nearish future to relocate your /db folder, which should make it easier to run two separate builds pointed at the same db..

>>3396

I just updated to from Ubuntu 14.04 to 16.04 for v217–could that be an issue? Apparently my old 14.04 build wasn't working on 16.04, maybe the reverse is true?

That said, the first release I tried wasn't working 100% because of the new build environment. This one seems to work better:

http://www.mediafire.com/download/s7ldwq6d8qvvm3l/Hydrus_Network_217_packaging.version_-_Linux_-_Executable.tar.gz

Does that give you anything new?