/hydrus/ - Hydrus Network

OK, I have a number of trivial suggestions

Home/End keys don't work in certain/most text fields, but they don't seem to be used for anything else in that context. Could we get those working?

I love sharing files through the booru, but I hate the incredibly long obnoxious URL. Might it be possible to make the booru URLs more manageable?

The 'share to booru' feature gives me a 'copy local url' and 'copy external url'. I'd like the ability to specify an IP/hostname for the external url. I have a vLAN/VPN with friends, so I have to modify the URL every time. Others might appreciate being able to use no-ip or something rather than a raw IP.

The 'paste tags' feature doesn't seem to work, at all. Maybe I'm using it wrong. I copy tags from one picture and try to paste them onto another, but the result is garbage (a lot of one letter tags).

Alternative views for 'tag parents' page. I'm thinking a tree view: show each parent tag once, and all the children underneath, indented. This will better illustrate multiple levels of parents (like you have already done in the tagging text field).

On the subject of parent tags, there doesn't seem to be a way to back those up. How about the ability to export tag relationships? (parent, sibling, censorship)

The image viewer's share menu is less complete than the normal share menu. Since the image you last viewed is automatically selected, this isn't a big deal, but why is this?

thanks

Some more flexibility with scheduling subscriptions would be good. Take these examples:

>update every Saturday at 2am

>update on the seventh of each month

>update on the first Tuesday of each month

>>650

Thank you for all these.

I think home/end is eaten up by the taglist beneath an autocomplete tag entry if there are results. When there are tag results, home/end scrolls to the beginning/end of the list of tags. Same with page up/down and cursor up/down. I'm not sure if that is helpful–maybe this behaviour should be optional?

I want to revisit boorus, including adding aliases. I don't have time now, but it is on my to-do list. Adding the better external url options is a great idea.

The way tags were copy/pasted in the tag dialog was an old method. I have updated it to use the same data protocol as clicking 'copy all tags' on a tags list, so this should work in future (although there are some graphical bugs that I'll clean up later). Thank you especially for this.

Better views for tag parents and siblings is something I really want to do (and maybe ways of filtering, like you might on a web browser's shortcut editing page), but I can't find time. It is on the list, but low priority. Same thing for exporting them. I'll get around to it someday!

The share menu on the media viewer is old. I've made a note to update it.

>>657

I think that tag parents can do what you want. Let me know if I've misunderstood, or if you would like help getting into tag parents.

>>666

Good suggestion, Satan. A good time to do this would be when I move subs over to JSON, which should be in the next month or two.

>>685

>I think that tag parents can do what you want.

Not really. The idea would be to give tags to all members of a set at once. Most mangas share one tag for all images in them in many repositories, simply because even if you search for a tag, you typically want to read the entire manga. So if I want to add tags to the entire set, it would be nice to add them quickly instead of making a search page for the set and control-A F3.

Now that I think about it, tag parents could work if I made e.g. "title:this comic" a parent of all the tags the comic is known to have. However I would have to maintain a huge ass list of tag parents, one for every title I have.

>>691

Not yet. I'm all for adding it. How would you like the workflow to go? What series of clicks and button presses would say:

TagA OR TagB OR TagC

Would making shift+enter start/continue an OR chain work?

Would such a predicate be editable when it is active? (when it is above the autocomplete entry) How would it be editable? Where would an incomplete chain be displayed?

Any thoughts you have would be useful!

>>698

You can do this with a custom filter, although the interface is not ideal. I want to make shortcuts custom across the program so that eventually you'll be able to set specific tags or ratings from anywhere. It'll just take time.

>>701

I planned something similar to that when I was storing files without extensions, but I hated the overall idea of having to copy as an interim step. It also means I have to copy all the files in the gui thread (which blocks) before the DnD finishes. If the user accidentally dragged, that means ui delay. What I really want is:

Create a DnD object, throw it at windows.

Windows treats it like a file DnD, modifies the cursor appropriately

If the user releases on a droppable location, the DnD object is informed, including being told that path. Windows doesn't do anything else.

I can then copy the files to that location at my leisure, in a background thread, with complicated renaming happening dynamically.

But I think that would require a shell extension, which is both beyond my experience and non-multiplat. I tried looking through the Filezilla Client code to see how they do it, and I think they drop a file with a weirdass tmp name and then have a ShellEx that locates/tracks that file and immediately deletes it. They then know the dest_path and can stream ftp stuff at it as they like.

Since I now store files with extensions, throwing "{hash}.jpg" at a copy FileDataObject sounds like an ok compromise. I want to add thumbnail dragging to change thumbnail order as well, and also dragging thumbnails between search pages.

>>703

This is an interesting thought. I see that parents don't quite fix your problem.

Collections are proving a lot more complicated than I originally thought, and I think the ultimate solution I'll move towards is to bundle them up into single media objects–.cbrs or whatever. There are very few times you want to deal with page 62 of a comic alone, and many many times you want to deal with a whole chapter/volume. Tagging should probably work the same way.

I can add what you want, but I am not sure how to avoid false positives. What is to stop an over-eager user tagging creator:whatever onto series:whatever? Parents and siblings all go through a janitor process for this exact reason–they are powerful enough that a mistake can be ten thousand files tagged wrong. Perhaps this is something that could only work for title tags?

>>837

Yes, I can do this. I have added it to my list. I will need to create a new control to handle selection of which mimetypes you want–I suppose a list of checkboxes works, right?

Sure, maybe in a tree of "image/animation/audio/video"

While you're in there can you add a way to move imported images somewhere instead of just deleting them?

>>846

I think I'll change it to be something like:

successful files->do this with them

failed files->do this with them

redundant files->do this with them

already deleted files->do this with them

with the options being:

do nothing

move to location x

delete

Or something like that. Hence I'll be somewhat merging the 'synchronise' and 'delete' modes into a generalised and more granular solution. And maybe I'll add a checkbox for 'don't retry paths you already tried once', to optionally preserve the synchronise mode's current efficient behaviour.

>>858

I plan to improve shortcut customisation in the near future.

That said, at the moment hitting Ctrl+C copies the currently selected files to the clipboard, and if you only copy one, Windows should paste that as the path in a normal file selection dialog's path text box. I don't know about other OSes.

This week, I'll change manage->tags to be just 'manage tags' if you don't have any ratings services.

When downloading from a booru/thread/etc., could it made so you can block files from importing that have certain tags?

Would a flow like this

>define tag x as blacklisted tag

>do a booru search for tag y

>find 10 results with tag y

>booru or tag database says 3 of those 10 results also have tag x

>download only the 7 files that don't have tag x

>add the hashes from the 3 barred files to a list of files not to be downloaded again, similar to when deleting a file

be possible to implement?

>>876

Hentai Foundry does something like this–I have to push some search options before parsing gallery pages to get all the super-rude images to display. Does danbooru work similarly? A simple summary of that script would be great–just the POST calls I need to make or GET args I need to add to unfilter the gallery results, or whatever!

>>877

This is slightly tricky, because hydrus can usually only link a file to a url after the file has been downloaded (it needs the file to generate the hash). I can add a censorship taglist fairly easily, but to be comprehensive, I would probably add the filter check to after the import. I am thinking of adding a 'recycle bin' service soon, so I would probably move a censored file to that immediately after import and inform the import page that that had occured, so where it says '3 successfuil, 1 failed', it might also say '2 censored'.

Is this what you are looking for, or were you specifically looking to reduce bandwidth by censoring before the download?

>>878

Do you have any of those errors still, that you can post here/email to me? They were probably written to your client.log, which is in install_dir/logs.

I am not sure what you mean about feeding dA a user account like pixiv. I thought my dA download page already searched by artist name? Is there a way of searching with some numerical user_id?

I can definitely add a blogger downloader. Do you have an example blog of the sort you want to parse?

>>882

This has been a problem in the past because I wanted to do it in a 'beautiful' way, but I think I will have to bite the bullet and do it in an ugly way that works for Windows, Linux and OS X all the same.

I will make a job in my to-do list to prototype this. For now, you can do very similar just by hitting Ctrl+C on your selected thumbnail and then Ctrl+V on your browser's file selection dialog.

>>894

>>895

I have thought about this a bit and decided that I'll add it when I rewrite the downloader engine (which should be in the next month or two). I'll check for tags before I download a file, and if the tags include any censored tags, it'll skip the file.

>>893

Do you want it only to return counts for the exact match for what you put in? Like if you type 'tank', you would only get:

tank (210)

or whatever? If so, you can probably do something this by changing the various thresholds of the autocomplete dropdown searches in file->options->maintenance and memory. The 'autocomplete' options are what you are looking for–hover over the number control, and you'll get a tooltip explaining what each one does. If you can't figure it out, just ask and I can explain more.

>>903

Once again, doing God's work.

>>913

Thank you for this information.

That first error is an unexpected failure of my parser to find a download link for a file. I have improved the error, and it should also print for which url it failed. If you see it again, please let me know the url so I can see what is going on. I expect it is a webm or something that uses a slightly different link-text than 'Save this file' or 'Save this video'.

I didn't even know that DeviantArt allowed adult images! I can add a pseudo-login, but I would rather keep it simple if I can. I just had a poke around their html, and while they are quite careful to hide mature images and thumbnails (and even identifying information from which I could extrapolate a link) from non-logged in users, they acidentally leak the encoded image link in a tumblr share button. I don't think I'll have time today, but I'll put some time into it next week.

Blogger shouldn't be too difficult–thank you for the example. The jump through the 'are you an adult' splash page will be an extra step to code, but that's it. There is probably an API to skip it entirely. I will look into it next week.

That Paheal JSON error is a mistake from the gui session rewrite from a couple versions ago. Booru pages aren't being saved to the database properly, which also happens every few minutes to maintain the 'last session' gui session. It is just some annoying error spam while a booru page is open, but the downloader otherwise works. It is fixed in tomorrow's release.

>>921

I am sorry, it looks like I missed this post. I'm planning to do something like this, but I've since added the thumbnail drag and drop export, which should speed up doing this the traditional way.

>>1003

Ironically, I just removed this support a week or so ago! It was making my other code complicated, so when I went to rewrite it, I didn't bother to update the zip stuff. I can bring it back, but I'll probably create separate dialogs for it.

I'll add support for more formats than just zip, as well, as here:

https://docs.python.org/2/library/archiving.html

Could we get a feature to rename individual tabs in the session? Simple right click menu on the tab with a rename tab option opening a simple dialog with one textbox would be enough.

Also, when you slim down the leftside panel with all the search and tags fields, the fields themselves stay fixed width.

Given that some very long tags can make it scroll sideways (and the sideways scroll bar appears within the thinner panel without being hidden, unlike everything else in it) anyways, can the fields contract with the panel itself?

Thanks for your time.

>>1030

Thank you, this is a great idea. Should be simple, so I'll try to fit it in this week.

I'll have a look at the management panel minimum widths, as well. I noticed this bad resizing when I was working on the thread watcher the other day.

This may or may not be a sound idea: being able to customize how many files Hydrus downloads from a site before it "takes a break" for a time period before retrieving any more. I find most times when a file fails to download it's after Hydrus has already grabbed a hundred or so images, maybe because the site is trying to limit your access or something. Perhaps be able to make the downloader take a break as soon as a file doesn't download?

I don't know, might not necessarily be the best solution.

>>1053

This is a good idea. I will add options to change the inter-file and inter-gallery-page delay times (currently three and five seconds or something), and adding another option for a break period after n files should be doable. I could also do it based on num_requests and num_mb sent to any particular domain (i.e. after 50MB, wait twenty mins before requesting again from there).

>>1054

This is in the works, and I try to do a bit more every week. The server's wx parts aren't necessary, so I plan to replace them all and move to console switch and network POST request control of the server. So far, I have refactored my general file parsing and rendering code away from the wx-entangled client-specific code and cleaned up my import tree so the server doesn't accidentally import any client code. Now I have to rewrite the wx app controller for the server and pull wx out of my pubsub and db maintenance timer code. And then add some switches to control it.

>>1070

I am considering converting to animated thumbnails for video because of this. Something like grabbing ten frames evenly spaced throughout the video and packaging that into a small gif or just a list of jpegs.

I can then add an option for which frame people want to show by default and also do a preview ~10-frame animation when people mouse over the thumbnail, like how some video sites do.

>>1074

Weren't you going to do something similar for ugoira support? Could do the same thing for both.

Also do you still plan to give us a way to save our tag import regexes into profiles? I have to copy and paste the same five regexes into the namespace import box with every bleeping folder I import.

>>1080

Both ugoira and saved tag rexes are on my to-do list, but it is taking a while. This week I have almost finished my big wxless server rewrite job, and then I want to move subscriptions and gallery downloads to the new download system, and then I should be a bit more free to add complicated new things.

Saved regexes should be the simpler of those two jobs though, so I will bump it up a bit.

I want to be able to skip this and send files directly to the trash on pressing delete.

>>1096

I will add an option for that this week, and for archive/return to inbox multiple files. Thank you for the suggestion.

>>1105

Not at the moment. Would you like to import it like a namespaced tag, like 'rating:explicit' or whatever? I can't add import to literal like/numerical ratings until I move gallery downloads and subs to the new system.

Also, is there a standard for this stuff across boorus? Is it only 'safe' and 'explicit', or can it get more complicated?

>>1108

>>1109

Thanks. I will see about adding this to the booru code as a 'rating:' namespace. This might be something that is easier to add after I convert galleries and subs to the new system. I'll think about it a bit, and have a look at the html and css.

>>1125

Sure, I'll add an option.

Do you have an error for the 'delete after import' stuff, or do the files just not disappear? It is working for me on Linux and Windows, even on read only files. Do you know if your Linux distro has unusual rules for what can be sent to the recycle bin?

Error tracebacks should be written to the bottom of your client.log, which is in install_dir/logs.

>>1130

I looked at the code again, and deletion errors in this case were getting silenced. I have now made them pop up and write to the log. Let me know if you get any continued trouble with this, even after you move to deleting rather than recycling.

>>1135

Thank you for the suggestion. I have had trouble getting taskbar icons to work in Linux/OS X, so this might be a Windows-only option to begin with. I'll add a shortcut for it, as well.

>>1148

Yes, adding tags to subs is planned. I'll have it work like for import folders. I can extend it to multiple tags in future, if people want that.

>>1154

No quick way inside the program. You can make sure your client loads things as fast as possible by making sure your db is vacuumed (database->maintenance->vacuum) and that the hard drive hydrus is installed to is fairly fast and defragged.

That said, I would like to work on speeding up media loading times. Some predicates like num_tags work way too slowly on a big db. Do you find the slow period is when the client just says "Loading…" in the status bar, or when it says Loading… x of y?

Since you want something simple and low level, if you like, you can query the db directly by going to install_dir/db and opening client.db in sqlite3 or SQLiteStudio. If you have never done this sort of thing before, I recommend SQLiteStudio, as it has a friendly gui. In that, you should be able to open client.db and a SQL Editor page and run a command like this:

SELECT SUM( size ) FROM files_info, ( mappings, tags USING ( tag_id ) ) USING ( hash_id ) WHERE tag = 'alphonse mucha';

That will select total size for all namespaces and all tag services. It will result in either a number, meaning total bytes, or NULL, meaning nothing was found. I can generate a more complicated query that'll do namespaces or other pseudo-predicates, if you want.

>>1156

I am planning to work only on the downloader and subs rewrite next week, and likely the one after as well. Once that is done, it will be much easier for me to add things to the gallery downloaders, and I will get right on this. Please keep reminding me if it looks like I have forgotten.

>>1157

Great. I will keep working on speeding this up.

>>1172

I definitely want to support audio natively, and I have a rough plan to get there. Once I am happy with this current downloader/subs rewrite, I should be free to work on big new features like this again.

I should be able to support sound for mp4s and mp3s and whatever else as well, which will open up the program to do several new things and will also need some new gui elements to handle mute and volume and so on. The audio is still in your webms, btw, I just don't render it. I use 'open externally' on those webms for now.

>>1178

I like the idea, but this is outside my scope for now. In the future, the network may run user-written app services, which would support this sort of specialised function very nicely, but for now I just don't have the time.

>>1181

I have a plan to move towards complete shortcut customisation for all functions. The situation should get better as I continue my overall YAML->JSON rewrite. (some shortcuts are currently stored in YAML, and some in JSON, and some are hardcoded, and there are then two or three ways the gui catches key/mouse events, and it is generally a bit of a mess all around that needs cleaning up and harmonising).

I've been thinking of extending the current 'zoom to fit canvas' option to support also saying 'unless the image is >90% normally', as pngs and other sharp images get unnecessarily blurry with the tiny zoomin. Adding 'fit to width/height' and other options is a good idea, and maybe having different options per mime as well. I will make a proper job of it and put it in my to-do, thank you.

>>1188

Someone in the v175 release thread mentioned file imports being slow. I am not sure what is causing it. The file importer should attempt to run as quickly as your machine allows, and my dev machine reliably rises to ~25% CPU (using all of one of four cores) and imports about 2-3 files per second, depending on file size, but some people seem to be having some gui-related delays that I haven't pinpointed yet. I just ran a test on my laptop, and I only get 7%, and about 1 file/s, so perhaps having a tag- or file- heavy db is causing the slowdown. I am looking into it.

HDD latency is also a common problem. Is your hydrus hard drive particularly fragged or full, or could there be some other disk-intensive program running while hydrus is trying to import?

If you like, go help->debug->db profile mode while you are importing some files. That will dump a lot of profile text to the bottom of your log file (install_dir/logs/client.log). You can email/pastebin those profile tables to me and I can have a closer look at your situation.

Unfortunately, the language I write hydrus in (Python) won't work on multiple cores simultaneously unless you bend completely over backwards, so I am limited in some CPU areas.

>>1223

Here is a webm of what I mean when entering a generic tag. As you can see, it takes awhile. With my file streaming suggestion above, it'll stop loading at 100, or whatever the user sets, so that I get my results quickly.

(Loading is also slightly slower due to the fact that I was currently importing files + recording)

>>1223

>>1224

Here's what I just emailed you, so people can see what I generally think:

I like the idea of auto-loading. I am going to have a look at my media-generation code (the bit that does "Loading 512 of 2000") in the near future, and if I cannot speed it up sufficiently, I will look at doing something like this. Thank you for the suggestion.

If you haven't seen it yet, for now you can add a "system:limit" predicate, which will limit the results to a random sample up to the given number. I use it all the time, and it does the first half of what you are talking about here. You can also pause a search by clicking the "searching immediately" button, which is helpful if you want to put in several tags without having the CPU hit of the half-completed search going on in the background while you type the subsequent tags in.

There's no way to see the tag list yet. I have thought about creating a tag cloud to display a client's most popular tags, and there is a thought to add this information to the tagging dialog (e.g. anything tagged with 'series:blah' could have a tag cloud that had various common 'character:' tags for that series), but I am not sure how I would really display that data on a normal application gui.

>>1231

Thank you for the suggestion. I have thought about doing something like this before, but it is actually quite tricky, as the way the server and client store some sorts of similar information is different, and so many tables just can't be combined without hacking things into inefficient shapes. The server stores files and tag mappings with timestamp and account info for every row, for instance, while the client considers everything to be timeless and anonymous and is more concerned with keeping everything quick to search. Combining the two datastores would also need new gui and db code to replicate all their existing http communication on an intra-application level. Furthermore, the db engine I use, SQLite, is a complete pain about multi-threaded access, so trying to get simultaneous server and client access would require some locking magic that would almost certainly slow things down, especially when the server wanted to suddenly generate a big update for 45 seconds straight.

Writing insta-access local http requests produces some not dissimilar problems, in that the server is designed to use very little CPU recording new content and occasionally generating update files, not in performing complicated tag queries. I could create a cache of the server's data inside the client and search that, but that's what a file repository already is.

My general long-term plan is to move many of the server's responsibilities to the client, file sharing included, once I have some p2p functionality going. I don't really like how the file repo works, so it might be better for the server to be a p2p node-connection facilitator, rather than a big media server.

If your concern is more in doubling file storage between the client_files and server_files directories, there is probably a good way to neatly merge them, maybe with an auxiliary db file that tracks which hashes to keep and delete between the two programs, but that makes clean backup and restore a new headache. I will think about it a bit more. I expect I am missing some obvious solution. If you have any ideas on how to cleverly do this (and it needs to be multiplatform, so we can't spam symlinks), I would be very interested.

>>1241

Hydrus.exe is generally where I think I am heading. I would eventually like the network to move towards something like gnutella, but working with hydrus's specifics like sha256 hashes and predicates and optimised for small files like images.

I'd like for people to be able to go "show me some x" and get a page of thumbnails for the various media the other nodes have to share. Not necessarily comprehensive over the whole network, but more like "here's some x I found" like Limewire and those other clients used to do. Nodes would share their and others' file and tag caches about, perhaps with differing levels of trust, and offer proxy and signing services as well. Specific always-on supernode services would probably play a part, but only really as certificate authorities and IP caches and that sort of thing.

That's a very vague description of a very complicated system, so my current practical plan has been to get some instant messaging going and start tacking on features like real-time IM file transfers. In time, I can add asynchronous (email) messaging and so on and so on. I was messing with this stuff a long time ago, but I've been distracted by the sheer mechanics of showing media in the client (which is what most people request me to work on). I have a lot of ideas and a few splatters of mostly broken code lying around. Nonetheless, I feel that the 'searching and viewing and tagging/rating' part of the client is working fairly well right now, so I am starting to think about this stuff again.

I am also planning to add support for many regular client actions (import and tag files, that sort of thing) to the client's local server, through POST requests. Then people will be able to externally script mass imports and tags from various complicated locations. Formalising more client commands into proper serialisable objects is a good step towards client-to-client syncs and so on.

I'm new to hydrus and so far I'm really enjoying it.

There's one thing I would really like to see and that's the implementation of sibling posts. I would like to be able to link two or more entries together and preferably have them show up next to each other.

I often have multiple versions of the same image and I don't like to see them all over the place. This seems like a good solution to that problem.

I could obviously just tag it with different volumes and pages but that just looks bad to me.

I don't know if there is already a better way to do it but this seemed like the place to leave my idea.

Thanks for your time and keep on doing what you're doing!

>>1258

I am glad you like it!

One of my next big plans is to scale my 'looks very similar' duplicate detection code up to the entire database, so you'll be able to say "show me all the images that might be slightly different resolution, or have little watermarks, but are otherwise basically the same thing", and then have those results appear in a way that makes sense. I then aim to add some new concepts for the db to understand like "This file is the same as this file, but it has a slightly higher resolution" or "has fewer jpg artifacts" or "isn't cropped" or whatever that people can enter, perhaps through some filtering system. This information could be shared through a new service, and then people could clean their dbs and merge tags for dupes in an efficient way by setting up commands for each relationship type (like "when you spot dupes, I want you to delete the version that has jpg artifacts and port its tags over to the better version.").

It would be easy for me to add "This is an alternate of this. The first image is the 'parent'." to that sort of system. I could add a little button to the media viewer to show the stack of alternates, or include them in normal queries, or whatever. In any case, I am not very happy with how collections work at the moment–I think I could incorporate this sort of rich metadata into better sorting and display of media. But the first step is in gathering that metadata in an efficient way.

This stuff will be a lot of work that will take me a long time to do, so please follow along and let me know what you think of the components as I build them and keep me reminded if it looks like I have forgotten about siblings. Thank you for your interest and suggestion.

>>1270

beat me to it

>>1247

yeah, ipfs

>>>/tech/412592 should be the current thread they're discussing it in

Seems like a perfect fit, since it takes care of the hard part by trying to be a general purpose protocol for data decentralization and distribution, and it bases everything around hashes similar to your method.

It even has python api bindings: https://github.com/ipfs/python-ipfs-api

My very rough idea of how it would actually be used:

Transfer of all Hydrus hashes to use ipfs' multihash format ( https://github.com/jbenet/multihash ) so as to not lose the millions of existing mappings we already have (likely easier said than done, but still) and then pinning those hashes in Hydrus.

For new files we could get the hash instead by simply adding them in ipfs (which automatically pins them as well in the ipfs client, not sure about the python-api) and recording the resulting hash in the database. You could even do the same for file thumbnails as well.

When you want to retrieve the file, a simple "ipfs cat <file_hash>" (or however it would be written in python) would let you retrieve the file from ipfs and view it in Hydrus.

If you actually did this, then it would immediately come with the added bonus of every hash Hydrus knows about can now be searched for by its tags and retrieved even if the user does not have the file locally as long as at least one other person on the entire ipfs network has that file to share and is accessible. Essentially removing the need for file repos altogether.

Of course, the real implementation will be more difficult than just that, and will beg for the need of a few extra features of how the whole thing would work. You would need to be able to handle cases where a hash is requested but not actually able to be found (likely not too difficult) as well as have a proper interface for pinning, and how it works with the inbox/archive/trash paradigm you currently have (my idea would be to add and pin everything, just marking it differently like you do now, and then when you want to actually delete something you just unpin it and let ipfs garbage collecting take care of the rest). Not to mention the many subtleties of Hydrus I don't even know about that would make implementing this more difficult, I'm sure.

But, again, if you actually did go about doing this, the benefits wouldn't even end there. The tag repos themselves could be decentralized despite having to be update-able (and thus changing its hash that ipfs relies upon) because of ipns. Now, ipns especially is still being worked and has its problems in its current state, but in the future should be more reliable and a practical alternative to tag repos by, again, just having a specific ipns hash that every Hydrus client is told "This is the public tag repo, download this over ipfs and use it locally" In fact, it even supports DNS, so you could even instead tell it to go to /ipns/brand-new-hydrus-website.biz and it would look at brand-new-hydrus-website.biz for a custom ipns DNS record with the correct ipns hash to then use, letting you change the hash without worry.

Hell, there may be a way to optionally not use a local database at all and basically just use an updated, decentralized, shared database directly. There is already talk about how to implement and use databases on ipfs (see https://github.com/ipfs/ipfs/issues/82 ).

And then there are ipfs "services" that I admittedly am not too familiar with but I'm sure are very interesting and has a plethora of possible uses with Hydrus. The most relevant to Hydrus I've seen is https://github.com/ipfs/archives/issues/8 which talks about decentralizing searches, where different nodes running this search service have only a fraction of all combined mappings but can be used together to not only decentralize the data it searches but even take part themselves and decentralize the actual process of searching.

Sorry for dumping a fuckton of info and autistic ramblings in front of you, but ipfs has given me a massive boner that simply will not recede and my doctor says that talking about it will help. Hope you were able to garner something useful from it.

One of these might already be possible but im not sure, but i have two feature requests

A.Dual Monitor Support, I would like hydrus to open fullscreen on the monitor I have hydrus on, not always on my main monitor, or even, a "windowed" fullscreen mode, so I can still browse through images and tag with the hotkeys, but have the images show up in a smaller window, this can even be useful to shrink the window a bit and put the tagging window to the side so they won't overlap (the "fullscreen" window would be locked in size and the image would stretch to fit)

My other request, is a tag downloader, instead of downloading all the images off a booru, it would just grab tags. This would be very useful for people who are picky about their collections, but also want the tags that already exist on popular boorus. So instead of downloading several thousand images of a character, they can just set the tag downloader to go and grab the tags off the site without grabbing the image, this might also lower the resource usage on the booru in question (educated guess here)

>>1278

PLEASE do this. You have so much good work put in to this thing already, if it could be used for distributed serving of tagged files it could be something truly phenomenal. People wouldn't even have to upload many images to chans anymore after a while, just link to a file in the system and the copies that are already out there will be streamed. All the people who like it will just grab it through the system and help distribute it and manage the tags.

>>1278

I appreciate all this info, thank you.

Now I have read a little bit about it, IPFS looks quite interesting and something I can definitely plug into. That python library looks great. I haven't used the network yet, so I do not know how stable or laggy it all is, so I will make the first step very simple and optional.

I have created a new job to basically do what >>1280 suggests–I will add an IPFS option page where you can point to your local IPFS daemon, and add an 'add this to IPFS' option to the right click menu for local files. I'll change my download code to allow downloads from unknown locations and let the client search IPFS for any outstanding downloads, essentially like a different kind of file repo.

If you happen to know, what's the ipfs cat/get failure state? If the file doesn't exist on the network or just can't be found in say 30 seconds, does it dump out with an error?

>>1282

A distributed chan is what hydrus was originally intended to be, funnily enough. I wanted to build a localhost service that would be a p2p node on the internet side, a bit like IPFS, that you would connect to in your browser, and would fetch distributed threads and images from via AJAX. All built through a LAMP stack, originally, until I stumbled over python and the pursuit of "Actually, getting images is fairly easy, let's try to burn some heavy clientside CPU searching and organising them" moved it all towards ACDSee for Anon.

>>1279

For multiple monitors, try pressing 'f', or hit the 'fullscreen switch' (tooltip) button in the top hover window of the media viewer, which should convert it to a normal window. The button looks like a couple of windows overlapping. The media viewer should remember its position and maximised/borderless state, but let me know if it doesn't.

I would love to build an efficient tag downloader for boorus, but I would need them to supply an API call for me to do it. Their API would have to accept file hashes, either MD5 or SHA-whatever, and then return tags. I could then say to the booru "Hey, you got any tags for [file]?" and they would say "Yeah, [tags]." Or they would need to support something like muh_booru.net/post/[file_hash] in addition to their existing muh_booru.net/post/internal_booru_post_id systems they use.

Unfortunately, I currently have to download the file if I want any tags, as I need the file hash to associate those tags with, and the file hash can only be generated by having the file itself. There are ways I can be clever, like if you have downloaded the file before, the client will remember that and will be able to reproduce the hash without having to download the file (that's what "get tags even if file is already in db" is about).

Again, if the boorus wanted to provide some xml/json/whatever API that was something like "Hey, booru, can you give me a hash->tags dictionary for this query?" I could do it and cross-reference with the files already in your db, but the API that danbooru and some others offer doesn't give hashes and at the API level doesn't namespace the tags. At the moment, my parser walks through every gallery page, picking up image pages, and then goes through each image page in turn, harvesting tag lists and image links.

4chan and 8chan bundle the MD5 hash in their API and it is great. I can check for redundant and deleted files without having to download, but boorus do not do it.

Sorry, I feel I didn't explain that very well. Let me know if you still have questions. TL;DR is that it is currently technically impossible to just download tags from a booru and insert them into the db. They need to offer more information on their end.

>>1280

Thank you for the note about the deviant art button. I though the click-zoom (which loads an image with the deceptive classname 'dev-content-full') was the same resolution, but it isn't! It looks like there might be some referral or cookie hoop I have to jump to get the button's link to load, but I will see what I can figure out.

I don't know much about g.e-hentai and exhentai. Isn't one like a login for the other, and one hosts loli or something? I am all for writing a plugin, but I want to do per-site bandwidth management in my next network engine iteration, so if there are complicated bandwidth limits then maybe I should put it off until then?

>>1290

Yup, it's actually not complicated at all. In fact, you can go to the json endpoint of this very thread just by replacing the .html in the url with .json: https://8ch.net/hydrus/res/471.json . Just do a Ctrl+F for "md5" and you can see what he's talking about.

>>1288

I was actually interested in the ipfs python api myself and decided to look into it. For testing I used the hash QmPZ9gcCEpqKTo6aq61g2nXGUhM4iCL3ewB6LDXZCtioEB as that is the standard welcome document that comes with every installation of ipfs and is unlikely to change. Don't quote me on that, though, it might not work in the future.

It looks like a call to api.cat() will return the contents of whatever hash you want, and any errors will raise a general ipfsApi.exceptions.ipfsApiError with the message describing what actually went wrong.

An invalid hash, for example, with have the message "invalid ipfs ref path", and a timeout should have the message "context deadline exceeded". I'm not sure what the default timeout for it is, but you can set your own by sending {'timeout': '<time>'} as the "opts" parameter in the call to api.cat(), which is the equivalent to "ipfs cat <hash> –timeout <time>" in the go client. <time> has to be both a number and what unit your using ("s" for seconds, "ms" for microseconds, etc.)

This is what I used to catch different types of errors. There are probably better ways to do it but it works for now and should answer your question:

import ipfsApi
api = ipfsApi.Client('localhost', 5001)

def retrieve_file(hash):
    try:
        return api.cat(hash, opts={'timeout': '30s'})
    except ipfsApi.exceptions.ipfsApiError as e:
        msg = str(e)
        if msg == 'invalid ipfs ref path': return 1
        elif msg == 'context deadline exceeded': return 2
        else: return 3

contents = retrieve_file('QmPZ9gcCEpqKTo6aq61g2nXGUhM4iCL3ewB6LDXZCtioEB')
print(contents)

Hope that helps! Also, thanks for fucking everything man. You really made my day.

>>1291

*milliseconds

Basically the same, right :^)

Oh man oh man if IPFS support gets added to this I'm gonna freak out (in a good way).

https://blog.neocities.org/its-time-for-the-permanent-web.html

>>1290

As >>1291 says, I need an API that'll give me the MD5. I just looked at Danbooru's and they actually do offer that in their json API, like this:

http://danbooru.donmai.us/posts.json?tags=blue_eyes

I guess I forgot they did, because I do remember looking at that before, or that is a new addition. Moebooru boorus offer something similar, but looking at this:

http://konachan.com/post.xml?tags=kantai_collection

I don't see any hash info. EDIT: Wait, it does, I was searching the wrong thing. Nonethless, in both, the tag list is not namespaced, so in order to get better tags, I would have to pursue the image page. That's not bad (although I don't really want to make a habit of hammering them with 20-100KB page requests just to harvest namespaced tags), but since the API request calls and responses are different across boorus (and some are unreliable, like http://gelbooru.com/index.php?page=dapi&s=post&q=index which currently gives me an error about overuse), I have shied from extending the already-messy services->manage boorus to support different APIs. Perhaps that is a long term project I can do, or it can wait for when downloaders are more complicated plug-ins that users can write themselves, which is a long-term goal.

>>1291

Thanks, this info is great. I will have a long recheck period (something like four hours, I guess) and bottleneck the number of files to attempt so hydrus doesn't end up hammering people's IPFS daemons every twenty minutes with 500 spurious hash requests (each of which I presume sends out some request to other nodes across the network), and also to abandon the download if a file can't be found after x requests.

>>1291

by any chance, could a bit of js be written to display md5 hashes next to images as one browses 8ch?

Sadly I don't know js and am completely incapable of writing something like that myself, but it could turn out to be incredibly useful if I had something like that

>>1304

https://greasyfork.org/en/scripts/13395-8chan-md5/code

Have fun! If you don't know what a userscript is just use the help on that site. On some posts it doesn't work because they're so old they don't have a recorded md5 on 8chan, at least it appears that way.

For some reason the md5 is embedded as an attribute on all 8chan img tags, but not exposed as text to the user? So this was super easy, almost a one-liner with jQuery.

Is it possible to have files which are rated, identified by a color? This would let me see what files I have rated without running a system:rating search/sort.

>>1312

Thats awesome! Just added

question though, if I want to make the md5 appear a line down, where would I put \n, OR is there a way it could a little [-]/[+] be made to hide/display it?

Ive already got a save as original filename script lengthening that row of info, so now its REALLY long lol

Great work btw

>>1329

I added a break before the MD5 (html uses <br/> for newlines), hopefully that unclutters things for now.

I'll add a +/- thing in a little bit.

>>1334

Saw the updates go through, looks awesome!

thought it was js, my bad, didn't realize it was html

>>1322

This is difficult because if you type something in, the autocomplete dropdown (the list where all the tags appear) needs to grab the vertical navigation keys (page up/down, home/end, arrow up/down) before they can propagate to the textbox so you can navigate the list of results. Arrow left/right should work, and with ctrl and shift. Does ctrl+arrow left work for you?

I could stop the list from grabbing the key events when there is only one result, which would often happen with a bad spelling, but I am not sure if that would be jarring/confusing. I'll change it for next week, and we'll see if it is helpful.

Hey dev, I always find it incredibly time consuming that I have to exit the tagging dialogue when using it with the full screen viewer, to archive or rate files. Any chance you could include the rating option and an archive button, or enable the shortcut to work when the tag dialogue box is open?

I usually skip ratings all together because of this, and id like to use them more than I do. I usually have to close the tag dialogue and the media viewer then highlight everything ive been tagging to archive, and I get crashes every once in a while, and I usually set my inbox to random order, so I often end up not being able to hit archive on a bunch of files that are now tagged.

The crashes are to be expected on my end, my hydrus library has now passed 100,000 images, i had to move it off my ssd because its too big, it gets a bit buggy now, especially when tag selecting ill have to wait a moment. But really at that size, again, expected

Hey, just started looking into this program, and it seems amazing so far, but I've got two suggestions I'd thought I'd post before moving my collection into it.

First would be a source field.

Keeping the source of an image has always been important to me, and I've found ways of saving the source with pictures via file names and folders since I started my collection. It hasn't been the easiest or cleanest, but it worked.

However, Hydrus copies imported images with different file names to it's own directory, so I'd lose all the source information I saved with the images.

It'd be awesome if you could add a source field into Hydrus somewhere for images, possibly multiple for artists who post their pictures across multiple websites, so I can import and organize my collection in Hydrus without losing the source of images.

Second is more of a personal preference, but I'd suggest adding an option to change how zooming in on a picture looks.

Zooming in on Hydrus makes the image become all fuzzy, which makes pixel art look poor, jpg compression a bit harder to spot, and makes the image look poor.

I could just open images externally to image preview, but I thought I'd suggest the idea anyway.

Sorry if either of these have been suggested before or are already planned, but thanks if you consider looking into them.

This has probably been asked a few times already, and there's probably a reason for this, but

Why doesn't Hydrus have this?

System tag for if image/gif is transparent?

>>1350

The tricky bit is the search algorithm. There are several ways to compare two files, but comparing n files in anything less than n-squared time is tricky. (I can't remember the exact math, but comparing 100,000 files by doing them all in turn would take something like 5-10 billion comparisons!)

So instead, you basically have to store the chunks of data that represents your images (I use a perceptual hash based off of a DCT of the thumbnail) in something like a VP-Tree that allows for quick hamming distance comparisons, and then encode that data structure in a database and then write some clever insert/remove/search functions to maintain and use it.

At the moment, I store the phash and can do "Is this single file like any of these 100,000 files" in 100,000 inefficient db hits, which usually takes a second or more.

Getting this to be n log n or whatever is all doable, but I just need to find the hours to point my brain at the problem.

>>1383

I had a plan to copy everything in the top hover window (the one with archive and prev/next buttons) to the tag dialog, and then it got overtaken by other things. I have put it up to top priority.

ACTUALLY: Maybe I should instead move the tags dialog into the media viewer? If a click of the tags hover on the left were to 'pop-out' a fully-fledged manage tags panel, maybe that would be better. Or keep it launching a window, but make the manage tags dialog be a frame instead, that isn't on top? So you can drag it about but still interact with other things.

I can make it a frame pretty easy, so I think I will experiment with that. I am not happy with the general multi-filtering-workflow, and I think the clunkiness of dialogs is a part of that.

>>1384

For the first thing, as >>1388 suggests, I think tags will generally do what you want. If I add a box for source or any other specific variable, then I have to add ways of adding and removing and searching and efficiently harvesting that info, when from a data standpoint, it is still a bit of text.

If your important information is stored in filename, and if it is stored in common patterns, you can capture that information very quickly and powerfully with regexes. If you haven't heard of them, they are a great but 200% autistic tool that can convert one bit of text to another.

When you are about to import some files, hit add tags based on filename. Hit add on the quick namespaces panel and do something like

source : (?<=\\)[\w\s]*?(?=\..*$)

There are also ways to get directory and parts of a filename and so on and so on. When you hit ok on that sub-dialog, you will see a preview, so you can go back and edit the regex if it doesn't do exactly what you want.

If you already know regexes, then you are fine, but I am also happy to help, and other people on the board know much more than me. I don't really like them, but I don't know of a better tool.

Also, if you download from hentai foundry or whatever inside hydrus, you can set source:hentai foundry as an explicit tag for everything you download in the tag import options panel. Let me know if you would like any help using this.

For the second, I agree. I want to add some simple 'zoom to fit?' options for each mime, but I think I should allow people to choose which resizing algorithms are used as well. There are some bugs in the zoom code too.

>>1389

I haven't written it yet! Several controls in the program (like the thumbnail viewer) hold too many objects for OSes to handle with normal gui elements, so I had to rewrite them all with custom code. As a result, I have to rewrite basic stuff like shift-selection and so on from scratch.

Funnily enough, I don't think anyone else has suggested drag select yet, although now I think of it, I have wanted it myself a couple of times. It shouldn't be too tricky to add. I will put it on my to-do list, thank you.

>>1391

It looks alright for me. Which OS are you using? What colour do your system predicates usually have? Is the text not showing but the predicate still has background a colour when you select it, or have I misunderstood your problem? Can you post a screenshot?

>>1400

I don't know anything about regexes, but I know you touched them briefly on on the help part of your website and linked out to a full guide, guess I'll have to look into them.

Just to clarify what I mean though, I'm not talking about adding "source:deviantart" or "source:paheal", I'm talking about saving the actual source URL of the image.

So what I meant by source field was a separate thing from tags, where a clickable link would be placed, like on paheal or gelbooru.

That'd also mean you probably wouldn't need to worry about adding in ways to search for or by sources.

Although, it sounds like I'm more or less the first person to suggest a source field, so I guess there really isn't any demand for the feature.

If there isn't enough demand to justify the effort of adding it, that's fine. I managed to found a way to put the source URL into file names, I can probably make it work with a tag.

It just would have been nicer to have a proper source field with a clickable link instead.

>>1401

I think >>1391 is suggesting that you add a global system tag for when images have transparency, not complaining that system tags for him are transparent or something.

I actually thought the same thing and had to re-read his post, though, so I blame his wording.

You talked about offsetting thumbnail generation "so every video doesn't just have a black thumbnail".

Why not take it a step further and generate multiple thumbnails per video? See picture.

>>1402

I see, that's an interesting idea. Anything you download from a website inside hydrus actually has its 'page' url saved to the db (so the client can know to skip a download if that url turns up in a subsequent gallery search), so this particular information is already supported somewhat. It would not be too hard to query this cache and expose it to the gui in a 'known urls/I have got this from' list or something from a thumb's right-click menu. This information could also be shared amongst users, and that might be very much a useful thing to do. Let me think about it a bit, and then I'll prototype something.

>>1404

>>1391

Thank you, I read it wrong.

Transparency info is something I don't currently cache in the db, so I can't search it. Once I add efficient dupe searching, I will likely be adding additional 'this image is like' information, like 'redness' and 'lots of curves' and 'sharpness'. This system could support 'has transparency in its palette' very well, so please remind me when we get to that point.

>>1405

This is the plan. I will move to all animations having gif thumbnails with up to say ten frames, and then you'll be able to set a custom offset for the 'display' frame. Also, I'd like to have these thumbs do their rough animation when you mouse over them.

>>1440

Make sure you have version >=180

>>1444

Or here (i don't quite understand what you mean as both options are obvious for me)

Would it be possible for the tumblr downloader to be able to pull from the source link of a post if it's an appropriate mime-type and exists?

I'm not sure if the behavior should to be to download the source instead of the post (because there can be multiple images and only one source) or just download everything, source & post, perhaps an option?

Here's an example blog that uses the source to link to higher res versions of the posted images (as it seems tumblr limits resolution on uploads): http://t-hoodie.tumblr.com/

>>1456

This is a great and simple idea that I can add quickly–thank you. I will add an option for it, and hardcode tab as the shortcut.

>>1461

I really like the idea (I didn't know tumblr had that source thing–I assume it appears if you embed an image from a url or something?), but I am not sure how to harvest that url information. I use the old tumblr api to generate links, like so:

http://t-hoodie.tumblr.com/api/read/json?start=0&num=50

I can't see that 'source' information or the puu.sh links in there. I looked around and found a per-post api, like this:

http://t-hoodie.tumblr.com/post/133375777797/json

But I can't see it there either. I could write a new parser to parse each post for a 'meta-item source-link' class tag and harvest the href from that, but I won't have time to do that for a little while. Do you happen to know of another API I can plug into, or anywhere else I can get that puu.sh link? Or even a way to get an rss or json or whatever of the artist's puu.sh links directly?

>>1475

I love the thought of adding these complicated sort of rules for parents and siblings, but it will have to be something for the future. For the moment, please add those sorts of tags manually.

>>1479

The source link is manually input on each post, I'm not sure how widespread it's use is on tumblr. I had assumed you were scraping tumblr pages like booru pages and could have just scraped the <a> tag, which thinking about that now doesn't seem like the best solution as tumblr blogs are theme-able and the source link class could differ per blog.

Looks like the v2 api exposes the source url.

https://api.tumblr.com/v2/blog/t-hoodie.tumblr.com/posts?api_key=fuiKNFp9vQFvjLNvx4sUwti4Yb5yGutBN4Xh10LXZhhRKjWlV4

(I don't know who's api key that is. It shows up in the api docs https://www.tumblr.com/docs/en/api/v2#posts)

Also, there's apparently an official client library for their api

https://github.com/tumblr/pytumblr

not sure how useful that is to you, also not sure how viable it is using the v2 api, as hydrus would need a consumer key for tumblr, perhaps have the user request the key and input it to hydrus?

>>1484

Thank you for this information! I think this is ultimately something I will leave to users to create once my downloader engine allows people to create their own gallery parsers. I don't want to mess around with non-open APIs if I can help it.

For now, I suggest you make a job to check that tumblr every x days and manage the download and import cycle yourself. I do this for several webcomics and artists I follow that I would never have time to write individual parsers for, batching up a month's worth or so and then importing and tagging them in an efficient group.

>>1489

Alright, thanks for the advice. I'll probably end up writing that parser when hydrus does come to support that.

>>1505

Thank you for your thoughts. I think I want a button somewhere that says 'retry all failed subs files' or something, and for per-sub as well, and maybe a way of reviewing this info en masse. Maybe I could stick a generic 'my subs' service on the review services window, under local? Or I could put some extra info in the list of subs in the manage subs dialog? Maybe colour 'inactive' and failing subs' names a different colour? Or have a checkbox to filter them?

I can easily add some text to the manage subs dialog's subs panels, like '3 failed files', rather than hiding it all behind the import status button, so I'll start with that.

>>1521

Sure, I will create a new control to allow hours as well as days.

>>1539

This is a great idea, thank you. I have written some intelligent menu items for selection tags and the active predicates boxes for next week–depending on whether the tag or -tag (for any of the currently selected tags) exists in the current search, the menu will provide up to four options for remove/require/permit/exclude. It seems to work well! Let me know if you have any problems with it in v185

>>1541

These are both great ideas. There is a thought to add a tag cloud or something in the future to quickly display and search popular tags–it would be easy to add favourites to that.

For jumping between browsing and searching, for now you might try middle-clicking the the 'creator:blah' tag, which should open up a search page for that tag no matter where you are. You can then alt tab back to your media viewer window when you want. You can also right-click on a tag for a menu to quickly copy it to the clipboard, which might help quickly downloading more from a gallery.

It'd be really cool if there was a way to mass add subscriptions.

I bookmark all the artists I'd like to collect the art of and just recently wrote a script that turns the bookmarks into a list of tags categorised by service. It's A LOT of stuff to enter by hand and I'd much rather have a way of importing them automatically, whether it be through some custom import format or a direct python api.

>>1573

Thank you for this list.

You can sort by volume/chapter/page namespace (not very well, but it generally works) under the sort dropdown on the top left of most pages.

Also, you might be able to tag by import order like you want if you click 'add tags based on filename' instead of 'import now' after you have some pending imports listed. The little bit labelled '#', middle bottom, tags based on the # column in that dialog's list. You can set an offset and a step and a namespace, which should let you do quick page tags if that information isn't easily parsable from the filename.

I force flash files to be slightly smaller than their media viewer frame, so you should have a slim gap on every edge to quit or bring the hover windows to the front if you are in fullscreen. Let me know if that doesn't work for you.

The other things I have added to my to-do list. Some I already hope to get to as part of larger features.

>>1574

Better mass-handling of subs is a planned feature that'll be on the soon-to-come voting list. Doing mass-adding is a great idea to add to it, thank you.

>>1598

Thank you, that's a great idea. Icons would be good, but I might be able to add this as part of another system I am planning, like the custom thumbnail border thing (i.e. you could set: "if file is less than an hour old, give it orange border"). I will think about it a bit.

Until I can get something done on this, you could try sorting by oldest/newest first, which will put all the new files at the bottom or top.

We already have the '-' sign to exclude a tag from search results, how about adding the '+' sign to include a tag's results even if it doesn't match what's already there? Basically an "or" search.

>>1607

Now I think of it, adding OR searching logic was something on the big to-do list. I will put it on the poll for next week.

Using + is a good idea, but I am not sure how you would differentiate which parts of the query you wanted to OR with and which you wanted to AND. I think my original idea for that was to say that shift+enter would start a predicate OR chain and normal enter would end one, so you could create complicated queries like:

system:size>1MB

character:shinji

character:asuka OR character:rei OR character:mana

Which would give you large images with shinji and one or more of the girls.

Funnily enough, the db search code for this is fairly easy–it is getting the gui workflow and feedback right that seems to be the impediment.

I'd love if we could add a permanent tag to different sources.

For example; Idol Complex, I'd love that every file downloaded from there would have a tag "3dpg"

Alternatively, we could do save tags from what booru files are from when downloading (#130) and then it would be up to the user to parent those tags to other tags, like 3dpg.

>>1678

I am not sure I can do your first suggestion easily–I use a wx.Notebook for the main page display, and I don't think it supports hiding the tab area. I would probably have to write my own custom notebook control to do that. But I may have to end up doing that anyway when I eventually go for tab dragging/reordering, so perhaps I can add tab hiding once I have that done.

Your second suggestion is planned. I ran into the exact same annoyance myself yesterday. I will have separate regex favourites for the namespaced and the unnamespaced controls, and make it so if you select a favourite for the namespaced area, it pre-fils the text fields rather than copying to clipboard. Perhaps I will do it with dropdown choices. Anyway, thank you, I have it planned.

>>1689

>>1690

You should be able to do this with file-options->default tag import options and then setting an explicit tag for a particular site. e.g. you could add 'site:hentai foundry' to the hentai foundry default import tag, and then anything you subsequently set to download from hf, or any new subscriptions you created for it, would have that explicit tag pre-set.

I think the default tag import options extends to individual boorus, so you should be able to set '3dpg' for Idol Complex.

>>1691

I haven't set one up, so I don't think so. I'm afraid I don't really do chatrooms/IM.

>>1709

It does, thank you! I had never heard of it before. I love the aesthetics and animations of the ui, and I particularly like the five colours it puts beside the image in the detailed view.

Hydrus might have looked a bit like this once, but I just found that browsers didn't give me the kind of large-data power I wanted.

>>1710

You can change the collect namespaces, but be warned that it is very ugly. Go file->options->sort/collect. The collect by namespaces are derived from the sort by schemes, which default to 'series-creator-title-volume-chapter-page' and 'creator-series-title-volume-chapter-page'. To add a new sort entry, enter the namespaces in the text box at the bottom, ordered by sort precedence and separated by hyphen, and then press enter. Double-click an existing scheme to delete it. You might want to just add 'artist' to begin with. Then any new pages should show that new scheme in its sort by dropdown and any new namespaces in the collect by dropdown.

I apologise that this ui is so ugly and unexplained–I threw it together after I first got collect working, and never got around to upgrading it from its current 'debug' level.

Yes, please feel free to add artist->creator siblings to my PTR. I like the species namespace for pokemon and digimon and so on as well. Don't bother to give a long reason–I'll approve anything obvious.

I do plan to support local namespace siblings, probably through an expansion of the current tag censorship system. I might also add that meta-sibling relationship to be sharable through repositories. I appreciate that some people would always want to see 'artist' instead of 'creator' or 'copyright' instead of 'series', or would want to unify semantically similar namespaces from different tag repos that use different standards, or even to translate these terms to different languages.

>>1720

>>1722

Try file->options->media->zoom smaller media to fit media canvas. I'd like to make this more complicated in future, as well, like for saying 'if a png is > 100% and less than 120% zoom at fitted size, just show it at 100% so its pixels stay sharp'.

The default shortcut for making the media viewer fullscreen borderless is 'f'.

Yo!

we have ffmpeg a part of the package

Can we get a method to convert a video into another format (mp4->webm) on/before import for storage

>>1844

That's a good idea, thank you. I like the idea of conversion on import and also from the right-click menu.

I would probably add a warning for any conversion to webm though, as the codec introduces some random data that means every generated webm has slightly different content (and thus file hash), which makes it an extreme pain in the ass to match up differently generated webms across clients.

For instance, if I go:

ffmpeg -i a.gif a.mp4

ffmpeg -i a.gif b.mp4

The two mp4 files will be exactly the same byte-for-byte, but:

ffmpeg -i a.gif a.webm

ffmpeg -i a.gif b.webm

Creates two slightly different files. Importing a.mp4 and b.mp4 would give '1 successful, 1 already in db' and one thumbanail, but the webms would give '2 successful' and two thumbnails. This is fine right now, where webm generation remains rare and the same original webm file is usually shared across imageboards so everyone imports and tags the same file, but if I automate mass webm production via conversion, we'll just be massively diluting our hashes.

Out of interest, why are you thinking of collapsing everything to webm specifically? Or did you just pick mp4->webm as a random example?

>>1848

webm is made with web content in mind, it's much smaller than gif/mp4 files when done right (unless it's a tiny gif or something)

Imageboard is a plus, as webm has been adopted into it's culture, but more importantly it's easier for me to share/upload due to the compression(??) in the format

I would suspect this 'random' data comes from it's unique ID, something I didn't know about five minutes ago

>>1869

Pictures of what foobar2000's export/conversion interface looks like. One is a dialog of different profiles, another is the dialog to edit a profile. Hydrus could have something similar under the existing export interface.

>>1848

>we'll just be massively diluting our hashes

I really think conversion on import is a _bad bad bad bad bad bad bad_ idea because of this. Sure, give people the option to convert on export, but if a user wants to ruin the integrity of what goes into their database, they should have to work hard to shoot themselves in the foot like that.

My rationale:

-Converting on export is a per-image, one-off event that you do when the content you want to share is incompatible with the destination.

-Converting on import is, if available, going to be (ab)used for everything. Like I said, if the user really wants to do that, they can do it themselves, don't help them.

>>1856

>webm is made with web content in mind, it's much smaller than gif/mp4 files when done right

Mp4 can be just as small. webm/mp4 are just containers, and containers don't say anything about what is actually inside. While webm is typically pretty locked down, an mp4 can range from lossless video to webm equivalent. Twitter actually uses mp4 instead of webm.

Groups for services(like/ratings)/subscriptions.

This is more of an organization suggestion. I'd like to keep my subs/ratings contained in a group. For example, pixiv subs will belong in the [Pixiv] group. Since it's a list, I figure it'd be easy to make it a tree node.(Pic-related)

It also helps me achieve more narrow ratings, such as if I wanted to do 1~5 ratings based on images from a specific character.(I have global ratings and ratings based on other factors)

>>1871

I don't really 'get' github, I don't know why. Maybe it is their emphasis on collaboration, and perhaps diligence, which has never been my thing. I focused on clearing out the github bug reports a little while ago, and that was working fairly well, but then I lapsed back into my normal routine of checking this board and my email and then going straight to work on my existing to-do.

When I am back to normal schedule, I am going to try again, maybe checking one github issue every time I sit down to code. I hate leaving people hanging, and I keep telling myself to sort it out, but I still can't seem to connect the two in this case. I can only apologise for being a sperg, really.

If you delete the ffmpeg executable in the install_dir/bin directory, the client will try to just use 'ffmpeg', so if that is in your path, it should all work.

>>1883

One of my big planned jobs is an overhaul of the subs dialog. I like the idea of grouping by tree, thank you.

>>1896

That's an interesting idea, thank you. I will add it to my to-do list.

>>1980

Thank you for this suggestion. I have set it to throw up a yes/no dialog if it detects more than one pending potential sibling choice. If you hit yes, it'll preference siblings in every subsequent choice. Although this works when I test it, I am not sure if it produces a good workflow in the real world, so let me know if it causes you any problems.

>>1991

I have added a button for this to services->review services->local file service. Again, let me know if it doesn't work right.

>>2007

Thank you for this–that's an interesting way of doing it. I currently use a 64-bit DCT-based perceptual hash that describes the general colourless shape of the image. If you are interested, I do it in GeneratePerceptualHash in include/ClientImageHandling.py.

My current challenge is searching hamming distance of two such hashes quickly, but perhaps I am being too complicated. Do you know how well your hashes stand up to similar images, like when one has from 9gag plastered on it? I suppose I could run a hamming search like I do for my current phashes.

Once I am done with IPFS and a new tag suggestion control, I will be working on this. If I can't figure out an easy solution, I would appreciate your further suggestions.

>>2042

>>2046

Thank you for these suggestions.

Ordering namespaces is something I have also wanted, I just never got round to doing it. It should be fairly easy, as I just need to add an option for it and tweak the sort code. I'll bump it up my list.

Grouping tags is doable but more complicated, as the tag lists are all custom controls I wrote, so I would have to overhaul the internal data representation and external layout stuff to support trees. I will put it on my to-do list and save it for the next time I do a big rewrite of that stuff.

Partial tag searching should work already, much like you suggest. Putting a '*' in the search box should insert that exact search string in the autocomplete list below, and selecting that predicate should return all files that have tags that match the wildcard. For instance, putting in 'star*' should give you all the files matching 'star trek' and 'star wars'.

>>2042

>>2046

I second this, because I have stuff like this

Tag groups is also majorly wanted.

Now, for my own suggestion, I'd want a way to easily identify what file belongs to what url in the detailed file import status window.

Something like a query tree instead of displaying everything in one go.

>>2164

When you say:

>I'd want a way to easily identify what file belongs to what url

Can you explain a bit more what workflow you are looking for? Would something like right-click->open selected in a new search window do what you want? Or are you thinking of something like 'as you move the mouse over the lines in the list, the thumbanil is highlighted' or similar?

>>2196

Almost any website is possible, but I have generally stopped working on new website plugins so I can instead rewrite my downloader engine to support user-created plugins.

>>2197

Yeah, that's my general plan. The next iterations of the downloader engine should support more detailed html/json parsing rules, login FORM and cookie info, and also a migration of all that custom info to external files that anyone will be able to create and import/export.

Regarding tag archives: Would it be possible to create a more streamlined/efficient method of updating a particular tag database, assuming that we as a community can get one or more of them to the point that we have regular updates releasing?

I tested out my tentative e6 DB on a blank DB containing only ~7000 images, and it matched 1800 of those but still took 30-60 minutes to apply the tags from the DB. Removing those tags (so I could apply a new version) was a little faster, but still, it is a not insignificant operation, and I'm leery of how bad it will be on my live Hydrus install (which has probably 30-40,000 matches to this one tagrip, never mind the others I'll be working on).

I ask because this time around, I'm proceeding with the goal of making the spidering, collation, generation, and posting of the rips I'm concerned with as fully automated as possible, so that - for example - if e621 takes ~8 hours to rip in its entirety, and that process requires zero input from me, I can set up a monthly cron job that releases updates automatically, so those of us concerned with e6-sourced tags have the most up to date metadata possible. Other DBs such as Furaffinity may not be feasible to do monthly - but then again, e6 is a booru, so tags on old files can be changed, and therefore since I can't query by mod date the whole site must be rescanned every update, whereas FA is generally past-immutable and thus incremental update tag archives could be released on a shorter schedule.

So, I guess my question is, is it possible for you to code in a streamlined "update tags from archive with new version" operation in, or is that not how the system works?

If that isn't possible, what can I as a tag archive producer do to reduce the length of the update operation for myself and other users? I could theoretically calculate a "diff" - where I release two archives every time there's an update, one with added mappings and one with removed mappings - but since I don't know the rate of mapping changes on a large booru, that might not turn out to save that much time when applying them.

Is there anything else you can think of? Should this sort of thing be a live remote tag service, instead? I'd actually love that, from a user perspective, except that I'm not sure how to make the current Hydrus server interface work on a headless remote Linux server, which is where I do all this shit, and I'm also not sure how that could be made resilient - if I vanish, I don't want the data I've compiled to become unavailable to the community. Tag archives on Mega.nz will always be there, and can be shared around as files, but if I'm running some custom server, that relies on me providing that resource *forever*. I'm not down for that, really.

>>2226

That's an interesting thought, and I am not sure the best answer. The more generalised and powerful I make tag archives, the more they look like hydrus services. I am strongly considering moving all tag archive functions to the new extracted service database I am planning. When I pull client.db apart, all service-specific data will be extracted to its own folder. It'll be a portable container for most sorts of hydrus-compatible content, so it'll probably be able to do everything tag archives currently do, and with siblings and stuff as well.

So, in future, I'd like the user to be able to go something like file->import service->pick file/directory/whatever->would you like to import it as a new service, or merge it into an existing service?. Maybe with a single wizard that covers a lot of potential bases, rather than having different functions hidden all over the client.

For your plan for now, it might be reasonable to make three-monthly update archives subsequent to your original 'e621 up to 2015-09' or whatever, and then people can add those smaller (~50MB? something like that?) files a few times a year rather than having to deal with larger ones all the time. Having two archives of 1970-01->2015-09 + 2015-09->2015-12 works the same as 1970-01->2015-12 from hydrus's perspective, it'll just take two db queries. Having your users manage multiple files could be awkward, but it is easy.

The only difficulty there is figuring out what is the new content. If there is a way to grab that from the site (like if you know the guy who runs it, who has direct db access), then that's easy, but I presume you are expecting to have to do a full sync every time you update your tag archive, in which case you can go:

A = Tag archive 2015-09

B = Tag archive 2015-12

C = New tag archive

For every mapping in B:

If it is not in A:

Add it to C.

I am not sure if tag archives currently support membership queries, but if you figure out any new calls you would like, just let me know and I can add them no problem. I can even add a 'create_diff_tag_archive( A, B )' or something. Or you can query them directly, if you are comfortable with the sql.

I can easily add 'merge these two tag archives' code as well, but I don't want to get tied up writing client gui stuff for it when it might just be generalised to the new service db fairly soon anyway.

>>2227

Exactly similar mappings are merged on the same service, so if a tag is 'current' for a file on a tag repo, trying to 'pend' that same tag to the file is a no-op. The duplicate tags issue comes in gui-side, with siblings and when the client has multiple tag services (e.g. 'local tags' has a lot of tags and you have multiple tag repositories synced that have file and tag overlap) but the viewed domain is 'all known tags'. It is just a bit more difficult to quickly and accurately union and count siblings and merged services on-the-fly, so in some places, those counts are inaccurate.

Otherwise, applying multiple tag archives that have very similar content to a single repository isn't a big deal. Anything duplicated will be (quickly) ignored.

>>2245

At the moment, I think you can do two things with tag archives:

1. Permanently sync to them, by putting them in client_archives and then booting the client and going services->manage services->tag service->archive sync.

2. Doing a one-time import via services->review services->tag repo->perform a service-wide operation.

I don't like the workflow for either, really, nor the dichotomy of those two dialogs. The manage/review difference represents what is going on behind the scenes, but not what the user necessarily wants to do. I might end up merging them one day, or removing all commands from review services, or something.

>>2244

- Rename the window

Great idea, thank you. I would use this myself!

- Pull whitelisted tags from PTR to Local Tags

I hope to greatly expand tag display filtering and service-to-service copies and moves once the client.db split-up is done. I hope to pull all service data to separate directories, and I expect it to be easier and simpler to do service operations at this point. When I start work on that, please give me your feedback on the gui I create for it and make sure I include the features you are interested in.

- click-drag selection

This has been on the wishlist a long time (so long that I think I put it there!), and I have just never found time for it. I think it was on the last 'vote on what to work on next' poll, but I cannot remember how well it did.

- a tag option that gives each file tags based on the names of the folders it's in

You can do this now, but it is technical. When you import some files, click 'add tags based on filename', which will let you automatically parse tags from filenames using regexes. These are powerful and can get quite complicated. If you are a programmer, you may be familiar with them and can figure it out yourself, but if you are not, let me or the board in general know, and someone will help you. My attached image shows an example regex that parses simple folder names. It is:

(?<=\\).+?(?=\\)

I am not very good at regexes myself, so there may be better ways of doing it.

>>2259

Ah, I see. Yeah, tag archives have no deleted knowledge yet. The service db will, so again that's probably the direction we'll head.

Initial sync takes ages, although it no longer locks up the gui. It now runs off a pausable and cancellable popup window that does 50 files at a time or something, so the program runs a little loggier, but is still usable. The total time it will take will be roughly proportional to the number of files that are cross-referenced. If only 5,000 files are matches, it will be about ten times faster than if 50,000 were. (The exception to this is if the tag archive is sha256, in which case 100% of the files are matched, so it'll take ages every time.)

I was heartened to see tag processing generally working faster with the recent autocomplete cache change, so we'll see how that generally holds up as I continue to alter the cache. I expect some further speed-up and also some slow-down, but I am not sure how much they will balance out.

>>2328

Sure, I will add this to the list.

>>2357

That's a great idea, thank you. I'll set it to advance the slideshow if autoplaying durational media has played through at least once, and we'll see if further options are needed.

>>2362

I have big plans to handle near-duplicate files' metadata in the future of the program, including sharing that between clients like tags currently can be. I'd love to be able to automatically merge tags across extremely similar images (e.g. when one has a watermark, or is just a resize), and also to group your sorts of 'small alterations' images together.

After I am done with this first version of IPFS and then a tag suggestion control, I am due to write a new 'find similar images' algorithm that should work much faster, which will be the first step in being able to search for and display these sorts of 'these two images look the same–is one a better quality version?' types of filtering workflows.

>>2378

Thank you, this is a great idea. After I am done with IPFS, I will be adding a 'suggested tags' control to the manage tags dialog to pull suggested tags from external sources, and iqdb would be a nice plugin to it.

>>2384

I have stopped manually adding complicated new site downloaders for now. My new plan is instead to ultimately expand my downloader engine to better allow users to create and share those site profiles themselves. The next time a 'what big thing to work on next' poll happens, please vote for the downloader engine expansion.

>>2402

Thank you for this suggestion. I have improved some of the status bar grammar, and when one file is selected, I have appended the same info text you see at the top of a single thumbnail's right-click menu. For instance, you might now see:

480 files - 1 image selected, 7.6KB image/png (200x200)

Let me know how it works for you.

In future, I would like to make this customisable, like how some mp3 players allow you to configure the in-program label of a song like:

[artist] - [album] [album_num] - [title]

But for hydrus like:

if multiple selected: [selection_num] totalling [selection_size]

if single selected: [focussed_file_mime] - [focussed_file_size] - [focussed_file_resolution]

It'll be a larger job to allow this, but it is on my to-do list. If you are interested in this and I don't get around to it by the time I have the next 'what big thing to work on next' poll, please remind me then, and I'll put it on that.

So I am not sure who else would be interested in this, but I wanted to toss it up and maybe get some other ideas.

So, since hydrus uses its own naming scheme, file names are messy as hell, and quite frankly, ugly. One step further, I have to admit, filename threads really are my bread and butter.

I think the ability to export files based on specific parameters is certainly a worthwhile feature. One step further though - I think there is some good potential here for some powerful features that only something like hydrus could provide. Tags are awesome.

When I say export, I am referring to drag and drop interfaces FYI, since that is the easiest way to share files on 4/8chan. So, I think an interface that allows some detailed information when exporting would be awesome.

For instance - the ability to set custom titles based on namespace. For example - exporting porn? List the title and artist. Exporting a dank meme? Export based on a file name name parameter. Dumping a series? Add page number parameters.

One optional step would be able to optionally name them all by "author:'paramater' - title:'paramater'" I don't know how many people would find it useful, but could certainly be a huge step for those who like to dump.

Just tossing some stuff into the think tank. R8 and suggest stuff, because I'm sure there is some stuff I'm not thinking of.

>>2449

I would very much like to do something like this. If you haven't seen the export dialog, check out the regular thumbnails' right-click share->export->files, which lets you generate 'creator - series - chapter - page.jpg' type filenames based on existing tags. I would like to add a similar pattern-match to quick drag-and-drop exports as well.

>>2469

I like all of these thoughts. I've wanted to do a couple of them for a while as well, but they are large jobs that will take me several weeks to do. Please bring these up again when I next have a 'what big things to work on next' poll (if they are not already on there), and I will add them to the list.

>>2474

Done, thank you for the suggestion! It should appear dialog off the file->options->shortcuts page. Let me know if it doesn't work for you.

>>2589

I've had a think about this, and having recently played around with hydrus on a 4k screen, I can better see why people might want more.

I'm open to adding larger thumbnails, but they'll appear a little bit slower the first time they are loaded and anything from a file repo will have to be upscaled from the network-delivered 200x200. It's now on the to-do list.

>>2590

I would love to add all sorts of IPFS stuff like this in future. Ideally, I'll make a mini-API so you'll be able to deliver all sorts of shareable content on any sort of p2p network.

>>2688

I would like to do a lot with siblings and parents, mostly filtering the giant list and displaying and editing the relationships in a more visual way, but have yet to find the time.

I like the idea of combining the two workflows into one ok button. Maybe if I turn it into a notebook with sibling/parent tabs, like review services has local/remote? Or were you thinking of displaying both sibling and parent information at the same time?

>>2715

Sure, I'll add it to the list.

>>2756

I think I understand what you are getting at, and I think its interesting. Tag clutter is something that's been on my mind as well, and I think collapsible trees may be part of the answer. I'll also be adding 'gui censorship', so you'll be able to filter page: tags or whatever from spamming the selection tags box without removing them from sorting rules and so on.

Before we can do any tag grouping, though, I need to write tree support into the taglist control. The data behind it is just a flat list at the moment, so I'll just have to rewrite it to be more clever and to do collapse/expand events. I'd like, in lexicographic views, to be able to quickly collapse namespace, and I'm sure final fantasy could be reasonably hidden behind final fantasy vi for many purposes.

>>2788

That's an interesting idea–do you mean as a backup, in case your uploaded tags get deleted serverside by a janitor?

>>2792

Sure, I'll add a button to review services. Be warned that it will still likely be a very laggy operation, as transferring any single file between file domans (like local->trash or trash->None) needs to update a bunch of caches (like the autocomplete tags caches), which can add up if you are doing thousands at a time.

If you haven't already, check out file->options->files and trash for some options about how files are automatically removed from the trash.

>>2796