Version 425
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v425/Hydrus.Network.425.-.macOS.-.App.dmg
linux
I had a good week. I optimised and fixed several core systems.
faster
I messed up last week with one autocomplete query, and as a result, when searching the PTR in 'all known files', which typically happens in the 'manage tags' dialog, all queries had 2-6 seconds lag! I figured out what went wrong, and now autocomplete should be working fast everywhere. My test situation went from 2.5 seconds to 58ms! Sorry for the trouble here, this was driving me nuts as well.
I also worked on tag processing. Thank you to the users who have sent in profiles and other info since the display cache came in. A great deal of overhead and inefficient is reduced, so tag processing should be faster for almost all situations.
The 'system:number of tags' query now has much better cancelability. It still wasn't great last week, so I gave it another go. If you do a bare 'system:num tags > 4' or something and it is taking ages, stopping or changing the search should now just take a couple seconds. It also won't blat your memory as much, if you go really big.
And lastly, the 'session' and 'bandwidth' objects in the network engine, formerly monolithic and sometimes laggy objects, are now broken into smaller pieces. When you get new cookies or some bandwidth is used, only the small piece that is changed now needs to be synced to the database. This is basically the same as the subscription breakup last year, but behind the scenes. It reduces some db activity and UI lag on older and network-heavy clients.
better
I have fixed more instances of 'ghost' tags, where committing certain pending tags, usually in combination with others that shared a sibling/parent implication, could still leave a 'pending' tag behind. This reasons behind it were quite complicated, but I managed to replicate the bug and fixed every instance I could find. Please let me know if you find any more instances of this behaviour.
While the display cache is working ok now, and with decent speed, some larger and more active clients will still have some ghost tags and inaccurate autocomplete counts hanging around. You won't notice or care about a count of 1,234,567 vs 1,234,588, but in some cases these will be very annoying. The only simple fixes available at the moment are the nuclear 'regen' jobs under the 'database' menu, which isn't good enough. I have planned maintenance routines for regenerating just for particular files and tags, and I want these to be easy to fire off, just from right-click menu, so if you have something wrong staring at you on some favourite files or tags, please hang in there, fixes will come.
full list
- optimisations:
- I fixed the new tag cache's slow tag autocomplete when in 'all known files' domain (which is usually in the manage tags dialog). what was taking about 2.5 seconds in 424 should now take about 58ms!!! for technical details, I was foolishly performing the pre-search exact match lookup (where exactly what you type appears before the full results fetch) on the new quick-text search tables, but it turns out this is unoptimised and was wasting a ton of CPU once the table got big. sorry for the trouble here–this was driving me nuts IRL. I have now fleshed out my dev machine's test client with many more millions of tag mappings so I can test these scales better in future before they go live
- internal autocomplete count fetches for single tags now have less overhead, which should add up for various rapid small checks across the program, mostly for tag processing, where the client frequently consults current counts on single tags for pre-processing analysis
- autocomplete count fetch requests for zero tags (lol) are also dealt with more efficiently
- thanks to the new tag definition cache, the 'num tags' service info cache is now updated and regenerated more efficiently. this speeds up all tag processing a couple percent
- tag update now quickly filters out redundant data before the main processing job. it is now significantly faster to process tag mappings that already exist–e.g. when a downloaded file pends tags that already exist, or repo processing gives you tags you already have, or you are filling in content gaps in reprocessing
- tag processing is now more efficient when checking against membership in the display cache, which greatly speeds up processing on services with many siblings and parents. thank you to the users who have contributed profiles and other feedback regarding slower processing speeds since the display cache was added
- various tag filtering and display membership tests are now shunted to the top of the mappings update routine, reducing much other overhead, especially when the mappings being added are redundant
- .
- tag logic fixes:
- I explored the 'ghost tag' issue, where sometimes committing a pending tag still leaves a pending record. this has been happening in the new display system when two pending tags that imply the same tag through siblings or parents are committed at the same time. I fixed a previous instance of this, but more remained. I replicated the problem through a unit test, rewrote several update loops to remain in sync when needed, and have fixed potential ghost tag instances in the specific and 'all known files' domains, for 'add', 'pend', 'delete', and 'rescind pend' actions
- also tested and fixed are possible instances where both a tag and its implication tag are pend-committed at the same time, not just two that imply a shared other
- furthermore, in a complex counting issue, storage autocomplete count updates are no longer deferred when updating mappings–they are 'interleaved' into mappings updates so counts are always synchronised to tables. this unfortunately adds some processing overhead back in, but as a number of newer cache calculations rely on autocomplete numbers, this change improves counting and pre-processing logic
- fixed a 'commit pending to current' counting bug in the new autocomplete update routine for 'all known files' domain
- while display tag logic is working increasingly ok and fast, most clients will have some miscounts and ghost tags here and there. I have yet to write efficient correction maintenance routines for particular files or tags, but this is planned and will come. at the moment, you just have the nuclear 'regen' maintenance calls, which are no good for little problems
- .
- network object breakup:
- the network session and bandwidth managers, which store your cookies and bandwidth history for all the different network contexts, are no longer monolithic objects. on updates to individual network contexts (which happens all the time during network activity), only the particular updated session or bandwidth tracker now needs to be saved to the database. this reduces CPU and UI lag on heavy clients. basically the same thing as the subscriptions breakup last year, but all behind the scenes
- your existing managers will be converted on update. all existing login and bandwidth log data should be preserved
- sessions will now keep delayed cookie changes that occured in the final network request before client exit
- we won't go too crazy yet, but session and bandwidth data is now synced to the database every 5 minutes, instead of 10, so if the client crashes, you only lose 5 mins of login/bandwidth data
- some session clearing logic is improved
- the bandwidth manager no longer considers future bandwidth in tests. if your computer clock goes haywire and your client records bandwidth in the future, it shouldn't bosh you _so much_ now
- .
- the rest:
- the 'system:number of tags' query now has greatly improved cancelability, even on gigantic result domains
- fixed a bad example in the client api help that mislabeled 'request_new_permissions' as 'request_access_permissions' (issue #780)
- the 'check and repair db' boot routine now runs _after_ version checks, so if you accidentally install a version behind, you now get the 'weird version m8' warning before the db goes bananas about missing tables or similar
- added some methods and optimised some access in Hydrus Tag Archives
- if you delete all the rules from a default bandwidth ruleset, it no longer disappears momentarily in the edit UI
- updated the python mpv bindings to 0.5.2 on windows, although the underlying dll is the same. this seems to fix at least one set of dll load problems. also updated is macOS, but not Linux (yet), because it broke there, hooray
- updated cloudscraper to 1.2.52 for all platforms
next week
Even if this week had good work, I got thick into logic and efficiency and couldn't find the time to do anything else. I'll catch up on regular work and finally get into my planned network updates.
It looks like 8kun is going off clearnet, or at least expecting to. The site has been dying for a long time now, and it is well past time I moved. I simply procrastinated. I also regret being behind the 8kun bug report and question threads for several weeks now. In the coming week I will clear out appropriate responses and lock the board in prep for deletion. I will update 426's hydrus links so Endchan is our primary board for the time being.
This board will be deleted next Wednesday
8kun says they are expecting to lose clearnet support and be TOR only soon. I have been meaning to move away from here for some time, but I put it off due to procrastination. Now seems the time to do it, so on 426, Endchan will be our primary /hydrus/ going forward.
I am catching up with the Q&A and Bug report threads here and archiving them and posting those links to Endchan. On Wednesday 20th, I will delete this board. Please archive anything important to you and migrate over to Endchan if you are comfortable.
All hydrus links are maintained here: https://hydrusnetwork.github.io/hydrus/help/contact.html
Thank you!
The plan to move to Endchan only has changed!
Codexx on 8chan.moe has kindly offered to host me on /t/ over there. As I have never been a comfortable BO, this is a much better solution for me as the primary place to talk Anon about hydrus.
I will be maintaining a Hydrus Network General, the first instance is here: https://8chan.moe/t/res/2219.html
This board has been completely archived at https://8chan.moe/hydrus/
The links in the release tomorrow will reflect this. Endchan will remain as a bunker.
Remote backups with Hydrus
Hello everybody, I've been wondering how some of other Hydrus Network uses handle backups for their files.
My current setup is like thus: I make weekly local backups of my database, which are then sent to a NAS through UrBackup, From there, I've been using Duplicati with cheap S3-compatible remote object storage.
Now, supposedly Duplicati performs deduplication before uploading, however it still seems like it takes its sweet time on the SQLite database files themselves, so I'm wondering if their format just doesn't lend itself to be deduplicated adequately or if there is some other issue. I don't use password protection (although from what I've seen, it doesn't actually encrypt anything by itself so it shouldn't make much of a difference either way)
So, what other methods do you use to backup your Hydrus files remotely? Ideally it should be with deduplication because although space is not at a premium for me, my upload speeds are very limited.
Parsing scripts
How about a thread for discussing/creating/sharing parsing scripts?
I made one for md5 lookup on e621.net (actually I just modified Hydrus_dev's danbooru script). Let me know if I did anything wrong with it, I'm pretty clueless… but it seems to work fine.
[32, "e621 md5", 1, ["http://e621.net/post/show", 0, 1, 1, "md5", {}, [[30, 1, ["we got sent back to main gallery page -- title test", 8, [27, 1, [[["head", {}, 0], ["title", {}, 0]], null]], [true, true, "Image List"]]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-general"}, null], ["a", {}, 1]], null]], ""]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-copyright"}, null], ["a", {}, 1]], null]], "series"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-artist"}, null], ["a", {}, 1]], null]], "creator"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-character"}, null], ["a", {}, 1]], null]], "character"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-species"}, null], ["a", {}, 1]], null]], "species"]], [30, 1, ["we got sent back to main gallery page -- page links exist", 8, [27, 1, [[["div", {}, null]], "class"]], [true, true, "pagination"]]]]]]
Yiff.party is Dead... Any plans to help rip patreon or SubscribeStar?
Kemono.party is not working right now and seems to be very unstable. A good idea would be to get patreon/subscribestar downloaders working. Currently hydrus will try to look at patreon pages then get booted out, a possible code reference for how to rip it might be here https://github.com/OpenYiff/Kemono but this is for running a kemono.party service. Any ideas? I got used to using yiff.party as a backup/3rd party ripper to get HD content to hydrus, then deduplicate the shit booru quality versions.
Version 424
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v424/Hydrus.Network.424.-.macOS.-.App.dmg
linux
I had a good week. There are some quality of life improvements and faster tag search across the board.
The update will take some time this week to update a cache. If you do not sync to the PTR, it will be just a few seconds. If you sync to the PTR, expect about 5-15 minutes.
faster tag search
In the second half of 2020, I tried several times to tune the database for different sorts of wildcard tag search, which is used in all autocomplete lookups and many file searches. I was sometimes able to get small clients always running well, or complicated large systems running, well, but I failed to get it good for all situations with code alone–the structure of the database tag lookup cache made the tuning difficult.
So, I have updated how that cache works. Rather than always searching one big master table, the client can now 'zoom' in on the appropriate search context based on the type of search page or manage tags dialog or whatever.
Pretty much anything related to autocomplete and tag-based file searches is faster. Most importantly, the worst-case time for these searches is greatly improved. Complicated searches, like a 'namespace:*anything*' file search, should no longer have sudden gigantic lag spikes. These searches may still take ten seconds or more when searching millions of tags and files, but they won't accidentally lag out for two minutes on some tiny 'my tags' search with only 60 results.
The only exception in my testing is 'number of tags' searches still have bad cancelability. It is better, but not great. I'll keep working here.
The cache replaces an existing one. It will take some time to build it on update. If you do not sync with the PTR, it should just be a few seconds. If you sync with the PTR on an SSD, it should be 5-15 minutes (on my heavy client with a nice SSD, it was 7 minutes). If you sync with the PTR on an HDD, it will take significantly longer, so please plan for it. If you sync with the PTR, you will see some numbers count up as it builds the different parts of the cache. There will be some deletion work to start, then counting up to perhaps a million, and then up to 16 million or so, at about 30,000 a second.
I have more plans here, and more work to do to optimise the tag display system, but I will let this new cache breathe for a bit before going back in here with a machete.
full list
- new tag caches:
- as 2020 ended, I attempted but failed to tune fast search for all kinds of clients, big and small and simple and complex. unable to guarantee decent speeds with just code, I have redesigned the tag text search cache. rather than checking the gigantic master table for all namespace and subtag lookups, the client can now zoom in on a small fast cache limited to the current search context, so doing a clever lookup on 'my tags' will no longer be hampered by having PTR beside it, and doing a solid lookup on the PTR or 'all known tags' will no longer be accidentally hampered by an optimisation for another situation
- the 424 update will take some time to generate the new caches for your existing data. if you don't sync with the PTR, it should be a few seconds. if you do sync, it will be about ten minutes on an SSD (seems about 30,000 definitions a second), and somewhat longer on an HDD. it will count up the tags as it goes, and on the PTR there will be a bit of deletion work, then one or two counts up to perhaps a million, and then one big count up to about 16 million.
- in my initial tests, this cache adds about 1-2% additional processing time to mass tag changes, but a wide variety of tag lookups and file searches are now significantly faster, have much nicer worst-case lag spikes, and should cancel quicker. these are best in any specific tag domain, although 'all known tags' should still be much better. a future expansion of the tag cache is planned to finally address clean and accurate 'all known tags' searches
- summary; all these should be faster and cancel faster:
- autocomplete searches for 'subtag*' (most normal searches) are optimised
- autocomplete searches for 'namespace:*' are optimised, including when the namespace itself is a wildcard
- autocomplete searches for wildcards with an asterisk in the middle of the subtag are optimised
- autocomplete searches for wildcards with an asterisk at the beginning of the subtag are optimised (but this is still generally the slowest query)
- autocomplete searches for namespace and subtag wildcard combinations are optimised, with either or both as a wildcard of any type
- autocomplete searches for '*' are optimised
- tag file searches without a namespace (i.e. in file search, with any namespace) are optimised
- namespace file searches are optimised, including when the namespace is a wildcard
- wildcard file searches are optimised, for all the classes of wildcard above
- 'tag as number' file searches are optimised
- 'has ><= x namespace tags' file searches are optimised for speed, including when the namespace is a wildcard, but still have bad cancelability on large domains. I'll work on this more
- .
- other tag cache info:
- the 'tag text search cache' regeneration routine under the _database->regenerate_ menu is replaced with a service specific routine for the new cache
- on boot, if the client sees any of the new cache tables are missing, it notifies you and regenerates the affected subsection of the cache
- an old method of performing complex wildcard searches was using surplus data and has been eliminated. these searches are now also computationally cheaper beyond the other domain-based optimisations this week
- I have identified the next bottleneck in the tag search pipeline and have a plan to speed all the above up even further, which can all be done in code
- thanks to user feedback, I have also identified other wasteful overhead in tag processing. I'll keep working!
- while the planned 'all known tags' cache will be useful since most file searches are in this domain, it will be a bit of work, so I will first let this new lookup cache breathe for a bit. 'all known tags' will not be nearly as big as the 'all known files/combined file' caches that have hit us with so much CPU recently. I expect it to increase the client.caches.db size by about 5%
- unified all increments or decrements to autocomplete count caches, no matter the service domain, to one location
- unified how autocomplete counts are fetched across different service domains
- optimised specific and combined autocomplete count cache update overhead for new, existing, and deleted tags
- optimised display autocomplete count cache updates for tags with multiple siblings or parents
- optimised the 'local tags cache', which does fast tag text fetching for local files, when new tags or files are added/removed from the 'all local files' domain. this now occurs in the same unified autocomplete count update process. it now also caches pending tags that have no current count
- merged 'exact match' autocomplete tag searching code into generalised wildcard search
- misc autocomplete and other tag code cleanup and harmonisation
- ditched some old mass UNION queries that were not cancelling well
- .
- the rest:
- when you paste queries into a sub, the summary 'these were/were not added' dialog now always appears, and if you paste empty whitespace, it now says so
- the manage siblings/parents dialogs now specify which services apply which siblings, whether they are fully synced, the current display tag sync maintenance settings, and ultimately whether you can expect changes to apply quickly after dialog ok
- when a text entry dialog comes with suggestion buttons, it now focuses the text box by default. sorry for the trouble here! (issue #765)
- updated a couple petition reason suggestions in manage tags and parents
- added a shortcut to 'main window' to refresh _manage tags'_ related tags suggestions with 'thorough' duration. in future, these dialog-specific actions will be moved out of 'main window', these have just been a 'temporary' patch
- updated the 'running from source' and 'install' help with some new numbers and info about mpv, and updated the 'server' help with a document helpfully provided by a user explaining that the server does not do what many new users think
- sped up 'has tags' file searches in certain situations, mostly when there are few if any other search predicates
- the default e621 parser now pulls meta tags, thank you to a user for providing this
- the default nitter timeline url classes are updated, thank you to a user for providing this
- the new little hook that takes 'file:///' off of paths pasted into the filename tagging path text now also normalises the path, so if you are on Windows, the URI's slashes will be Windows-corrected to backlashes. it also now removes wrapping quotes
- the hydrus logger again correctly restores stdout and stderr after it is closed on program exit (this was disabled for some reason, but fingers crossed it seems fine now!)
- an issue where automatically started duplicate potentials file search could not cancel when shutdown 'stop work' button was clicked or where idle maintenance mode turned off should be fixed
- the shutdown maintenance work for the first client shutdown now has a little text saying it is just some quick initialisation work
- for hopefully the last and completely final time, I think I fixed the invalid tag repair function for certain sorts of tags applied to currently local files
- improved the way a job thread was pulling new jobs (issue #750)
next week
The poll is done! Here's the link again: https://www.survey-maker.com/results3310902xA574481e-102#tab-2
Multiple local file services has won. It looks like better URL sharing and file alternates will be soon after, as well. Thank you for voting–seeing what isn't popular is as useful as seeing what is.
Unfortunately, I cannot start that immediately. I have a fire to put out next week related to the network objects lagging too much when saving their updates. I will spend the rest of Q1 doing the delayed network improvements. So, with luck, I will get going on local file services in Q2.
I also have a ton of messages to catch up on!
/big data general/
or, "i've got xxxxxxx thousand images in my inbox, what the fuck do i do?"
i'm currently having this problem myself, so i thought it would be nice to have a thread to collect information about various ways to automate sorting/tagging/etc, such as:
* image deduplication scripts:
>https://github.com/knjcode/imgdupes
i'm presently also writing my own as well which should work somewhat better for large datasets, i'll post it here when it's usable
* datasets, useful for running against aforementioned dedupe scripts
>danbooru2019, contains all danbooru pictures + metadata up to early 2019: >https://www.gwern.net/Danbooru2019
i remember there being large dumps of other booru metadata on here years ago, but i can't find them anymore
* AI/neural network software for automated tagging, classifiers and etc:
https://github.com/KichangKim/DeepDanbooru
https://github.com/imamar94/ramrem-classifier
things i couldn't find but would find extremely useful:
>AI anime/photograph classifier
this would be very helpful, especially with deduplicating from danbooru2019
>subject classifier
if there was a NN that could tell me if the subject of a picture was a person or something else, this would be insanely useful, especially for the next item
>SFW/NSFW classifier
about half of my collection is porn. i'm planning on deleting basically all of it, so just being able to get that out of the way quickly would cut my workload in half
Suggestions
Drag and drop windows with tag rules. Show two windows side by side and one window can be programmed with the rule "ADD tag foo" and the other one has the rule "REMOVE tag foo, ADD tag bar" and you can drag and drop files to them.
Deriving tags from regex of other tags/namespace tags. A file has the tag "filename:big_ugly_name" and we could regex that namespace for another tag.
Tag sets with hotkeys: save a set of tags under a hotkey so it's quick to add them to a file while filtering
Opaque window behind tag list in the corner so it doesn't get hidden by picture background
Option to default certain mime types to be excluded from slideshow and only open externally, will help with videos with odd codecs that don't preview in the slideshow correctly
Option to specify hamming distance in "find similar images", you can't change the option once it's in the filter window and you have to enter the hash manually in the "system:similar to" option
Organizing 2D and 3D?
Do you only save 2D to Hydrus, or both 2D and 3D?
How do you go about organizing different file types? For example, 2D Photos, 3D Photos, Books/Doujins, etc.
I've stayed away from archiving any types of Books/Doujins into Hydrus, and keeping that into Calibre.
As for videos, anything full length (stuff you wouldn't find on Boorus) I just run it through Jellyfin.
So I guess personally my biggest thing is 2D and 3D photos/short videos/animations.
Right now, I just have them seperated with a tag. Filetype:2D, Filetype:3D. I have my import folders seperate (3D folder and 2D folder) and they import with that tag automatically if stuff goes in there.
Is there a more organized way?
I can't say I'm entirely a fan of having both 3D and 2D in the same database. 99.9% of my 3D stuff I don't even tag, it's only there so I can view it. So it feels useless to be in hydrus. Opinions?
papernow help
The quickest professional writing service https://papernow.org/descriptive-essay-help saved my time and nerves. They`re all super smart and super professional essay writers! I received a fabulous paper and got a discount, spent a good time. Fantastic assistance!
Version 423
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v423/Hydrus.Network.423.-.macOS.-.App.dmg
linux
𝕸𝖊𝖗𝖗𝖞 𝕮𝖍𝖗𝖎𝖘𝖙𝖒𝖆𝖘!
I had a good week making some small fixes and improvements to finish up the year. This is the last release of the year. There is a large poll on what 'big thing' to work on next:
poll
Here is the poll on what large work to go for next:
https://www.survey-maker.com/poll3310902xA574481e-102
You can vote on multiple items. Please don't worry about the seriousness of it too muchI have a good idea of what is likely to win already, and if there are obviously jank votes, I'll reserve the right to discount a resultbut I am particularly interested to know what is and is not popular further down the list. I'll take the results on the 6th of January.
The end of 2020 has come quick for me. I still have the network updates 'big job' to do, so that's first for 2021 Q1, but after that, I will plan out and hack away at the top item(s) on the poll.
If you were not aware, a team of users is doing great work managing the Github issue tracker for hydrus. There is also a process there for bumping issues with reaction votes, viewable for big jobs like so:
I regret that am doing pretty terribly at consuming my Github work queue, but we'll see how 2021 goes. I'd like to start relying on that more to better prioritise my work.
work is all misc this week
The settings for autocomplete auto-search and character threshold have been moved from options->speed and memory to tags->manage tag display and search. They are also now service-specific! So, you can now set 'all known tags' or 'PTR' to autocomplete after, say, 5 characters but your 'my tags' to always get all results.
I worked on the recently re-activated '*' and 'namespace:*' advanced searches, which were running slow in the new pipeline on larger clients. I have improved some situations, and I think I have reduced the worst case scenario, but some large clients will still have trouble. I am not happy here, nor with other namespace- and subtag- lookup speed across the program, so I have made a plan to make some clever new indices here. As I have simplified and cleaned much of the tag logic over the past year, I now see the next holes that I should fill.
The new siblings and parents tag right-click menu proved a little tall after all, so I have reworked it to group items by all the services that share them. It is denser to look at, but I think it'll highlight unusual exceptions when you are trying to fix application. The 'this is the ideal' line is also removed.
You can now sort a row of pages by name from their right-click menu.
Advanced users: The parser edit UI's test panel now shows more data, and it formats JSON prettily. It isn't anything close to what a browser's developer mode will give you of course, but it is nicer. That panel also detects if you hit a jpeg or something and says so, rather than dumping garbage to the test panel or just throwing an error.
full list
- tag autocomplete searches:
- the 'fetch results as you type' and 'do-not-autocomplete character threshold' options are moved from _options->speed and memory_ to _tags->manage tag display and search_. they are now service specific!
- getting the raw '*' autocomplete is now hugely faster when both file and tag domains are specific (i.e. not 'all known xxx')
- getting the raw '*' autocomplete is now hugely faster in 'all known tags' domain. this is likely still bonkers on any decent sized client that syncs with the PTR, but if you have a small client that once synced with the PTR, this is now snappy
- the cancelability of 'namespace:*' and 'match namespaces from normal search' searches should be improved
- 'namespace:*' queries are now much faster in some situations, particularly when searching in a specific tag domain (typically this happens in manage tags dialog) or a small-file client, but is still pretty slow for clients with many files, and I think some scenarios are still bananas. I am not happy here and have started a plan to improve my service domain caches to deal with several ongoing problems with slow namespace and subtag lookup in different situations
- fixed an issue with advanced autocomplete result matching where a previously cached 'character:sam' result could match 'char:sam' search text
- some misc code cleanup and UI label improvements in autocomplete
- .
- the rest:
- the siblings & parents tag menu, which proved a little tall after all, is now compressed to group siblings, parents, and children by the shared services that hold them. it takes less space, and odd exceptions should be easy to spot
- this menu also no longer has the 'this is the ideal tag' line
- added 'sort pages by name a-z/z-a' to page right-click menu and tucked the sorts into a submenu
- the parsing test panel now shows up to 64KB of what you pulled (previously 1KB)
- the parsing test panel now shows json in prettier indented form
- when the parsing test panel is told to fetch a URL that is neither HTML or JSON, this is now caught more safely and tested against permitted file types. if it was really a jpeg, it will now say 'looks like a jpeg' and disable parse testing. if the data type could not be figured out, it tries to throw the mess into view and permits parse testing, in case this is some weird javascript or something that you'll want to pre-parse convert
- the dreaded null-character is now eliminated in all cases when text is decoded from a site, even if the site has invalid unicode or no encoding can be found (e.g. if it is truly a jpeg or something and we just end up wanting to throw a preview of that mess into UI)
- the 'enter example path here' input on import folders' filename tagging options edit panel now uses placeholder text and auto-removes 'file:///' URL prefixes (e.g. if your paste happens to add them)
- the 'fix invalid tags' routine now updates the tag row in the local tags cache, so users who found some broken tags were not updating should now be sorted
- added –db_cache_size launch parameter, and added some text to the launch_parameters help about it. by default, hydrus permits 200MB per file, which means a megaclient under persistent heavy load might want 800MB. users with megamemory but slow drives might want to play with this, let me know what you find
- updated to cloudscraper 1.2.50
next week
I am now on vacation. I have some family Christmas things going on, and otherwise I am looking forward to grinding away at lategame X3 TC with some Wagner. I hope you can have a good break as well. I'll be back to normal in the new year, and 424 should be out on the 6th of January.
Booru.org is broken thanks to Cloud flare...
Furry.booru.org no longer works
when downloading I get this fun error.
Looks like an unsolvable CloudFlare issue: Detected a Cloudflare version 2 challenge, This feature is not available in the opensource (free) version.… (Copy note to see full error)
Traceback (most recent call last):
cloudscraper.exceptions.CloudflareChallengeError: Detected a Cloudflare version 2 challenge, This feature is not available in the opensource (free) version.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
hydrus.core.HydrusExceptions.CloudFlareException: Looks like an unsolvable CloudFlare issue: Detected a Cloudflare version 2 challenge, This feature is not available in the opensource (free) version.
Version 422
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.macOS.-.App.dmg
linux
🎉🎉 It was hydrus's birthday this week! 🎉🎉
I had a great week. I mostly fixed bugs and improved quality of life.
tags
It looks like when I optimised tag autocomplete around v419, I accidentally broke the advanced 'character:*'-style lookups (which you can enable under tags->manage tag display and search. I regret this is not the first time these clever queries have been broken by accident. I have fixed them this week and added several sets of unit tests to ensure I do not repeat this mistake.
These expansive searches should also work faster, cancel faster, and there are a few new neat cache optimisations to check when an expensive search's results for 'char' or 'character:' can quickly provide results for a later 'character:samus'. Overall, these queries should be a bit better all around. Let me know if you have any more trouble.
The single-tag right-click menu now always shows sibling and parent data, and for all services. Each service stacks siblings/parents into tall submenus, but the tall menu feels better to me than nested, so we'll see how that works out IRL. You can click any sibling or parent to copy to clipboard, so I have retired the 'copy' menu's older and simpler 'siblings' submenu.
misc
Some websites have a 'redirect' optimisation where if a gallery page has only one file, it moves you straight to the post page for that file. This has been a problem for hydrus for some time, and particularly affected users who were doing md5: queries on certain sites, but I believe the downloader engine can now handle it correctly, forwarding the redirect URL to the file queue. This is working on some slightly shakey tech that I want to improve more in future, but let me know how you get on with it.
The UPnPc executables (miniupnp, here https://miniupnp.tuxfamily.org/) are no longer bundled in the 'bin' directory. These files were a common cause of anti-virus false positives every few months, and are only used by a few advanced users to set up servers and hit network->data->manage upnp, so I have decided that new users will have to install it themselves going forward. Trying to perform a UPnP operation when the exe cannot be found now gives a popup message talking about the situation and pointing to the new readme in the bin directory.
After working with a user, it seems that some clients may not have certain indices that speed up sibling and parent lookups. I am not totally sure if this was due to hard drive damage or broken update logic, but the database now looks for and heals this problem on every boot.
parsing (advanced)
String converters can now encode or decode by 'unicode escape characters' ('\u0394'-to-'Δ') and 'html entities' ('&'-to-'&'). Also, when you tell a json formula to fetch 'json' rather than 'string', it no longer escapes unicode.
The hydrus downloader system no longer needs the borked 'bytes' decode for a 'file hash' content parser! These content parsers now have a 'hex'/'base64' dropdown in their UI, and you just deliver that string. This ugly situation was a legacy artifact of python2, now finally cleared up. Existing string converters now treat 'hex' or 'base64' decode steps as a no-op, and existing 'file hash' content parsers should update correctly to 'hex' or 'base64' based on what their string converters were doing previously. The help is updated to reflect this. hex/base64 encodes are still in as they are used for file lookup script hash initialisation, but they will likely get similar treatment in future.
birthday
🎉🎉🎉🎉🎉
On December 14th, 2011, the first non-experimental beta of hydrus was released. This week marks nine years. It has been a lot of work and a lot of fun.
Looking back on 2020, we converted a regularly buggy and crashy new Qt build to something much faster and nicer than we ever had with wx. Along with that came mpv and smooth video and finally audio playing out of the client. The PTR grew to a billion mappings(!), and with that came many rounds of database optimisation, speeding up many complicated tag and file searches. You can now save and load those searches, and most recently, search predicates are now editable in-place. Siblings and parents were updated to completely undoable virtual systems, resulting in much faster boot time and thumbnail load and greatly improved tag relationship logic. Subscriptions were broken into smaller objects, meaning they load and edit much faster, and several CPU-heavy routines no longer interrupt or judder browsing. And the Client API expanded to allow browsing applications and easier login solutions for difficult sites.
There are still a couple thousand things I would like to do, so I hope to keep going into 2021. I deeply appreciate the feedback, help, and support over the years. Thank you!
If you would like to further support my work and are in a position to do so, my simple no-reward Patreon is here: https://www.patreon.com/hydrus_dev
full list
- advanced tags:
- fixed the search code for various 'total' autocomplete searches like '*' and 'namespace:*', which were broken around v419's optimised regular tag lookups. these search types also have a round of their own search optimisations and improved cancel latency. I am sorry for the trouble here
- expanded the database autocomplete fetch unit tests to handle these total lookups so I do not accidentally kill them due to typo/ignorance again
- updated the autocomplete result cache object to consult a search's advanced search options (as under _tags->manage tag display and search_) to test whether a search cache for 'char' or 'character:' is able to serve results for a later 'character:samus' input
- optimised file and tag search code for cases where someone might somehow sneak an unoptimised raw '*:subtag' or 'namespace:*' search text in
- updated and expanded the autocomplete result cache unit tests to handle the new tested options and the various 'total' tests, so they aren't disabled by accident again
- cancelling a autocomplete query with a gigantic number of results should now cancel much quicker when you have a lot of siblings
- the single-tag right-click menu now shows siblings and parents info for every service, and will work on taglists in the 'all known tags' domain. clicking on any item will copy it to clipboard. this might result in megatall submenus, but we'll see. tall seems easier to use than nested per-service for now
- the more primitive 'siblings' submenu on the taglist 'copy' right-click menu is now removed
- right-click should no longer raise an error on esoteric taglists (such as tag filters and namespace colours). you might get some funky copy strings, which is sort of fun too
- the copy string for the special namespace predicate ('namespace:*anything*') is now 'namespace:*', making it easier to copy/paste this across pages
- .
- misc:
- the thumbnail right-click 'copy/open known urls by url class' commands now exclude those urls that match a more specific url class (e.g. /post/123456 vs /post/123456/image.jpg)
- miniupnpc is no longer bundled in the official builds. this executable is only used by a few advanced users and was a regular cause of anti-virus false positives, so I have decided new users will have to install it manually going forward.
- the client now looks for miniupnpc in more places, including the system path. when missing, its error popups have better explanation, pointing users to a new readme in the bin directory
- UPnP errors now have more explanation for 'No IGD UPnP Device' errortext
- the database's boot-repair function now ensures indices are created for: non-sha256 hashes, sibling and parent lookups, storage tag cache, and display tag cache. some users may be missing indices here for unknown update logic or hard drive damage reasons, and this should speed them right back up. the boot-repair function now broadcasts 'checking database for faults' to the splash, which you will see if it needs some time to work
- the duplicates page once again correctly updates the potential pairs count in the 'filter' tab when potential search finishes or filtering finishes
- added the –boot_debug launch switch, which for now prints additional splash screen texts to the log
- the global pixmaps object is no longer initialised in client model boot, but now on first request
- fixed type of –db_synchronous_override launch parameter, which was throwing type errors
- updated the client file readwrite lock logic and brushed up its unit tests
- improved the error when the client database is asked for the id of an invalid tag that collapses to zero characters
- the qss stylesheet directory is now mapped to the static dir in a way that will follow static directory redirects
- .
- downloaders and parsing (advanced):
- started on better network redirection tech. if a post or gallery URL is 3XX redirected, hydrus now recognises this, and if the redirected url is the same type and parseable, the new url and parser are swapped in. if a gallery url is redirected to a non-gallery url, it will create a new file import object for that URL and say so in its gallery log note. this tentatively solves the 'booru redirects one-file gallery pages to post url' problem, but the whole thing is held together by prayer. I now have a plan to rejigger my pipelines to deal with this situation better, ultimately I will likely expose and log all redirects so we can always see better what is going on behind the scenes
- added 'unicode escape characters' and 'html entities' string converter encode/decode types. the former does '\u0394'-to-'Δ', and the latter does '&'-to-'&'
- improved my string converter unit tests and added the above to them
- in the parsing system, decoding from 'hex' or 'base64' is no longer needed for a 'file hash' content type. these string conversions are now no-ops and can be deleted. they converted to a non-string type, an artifact of the old way python 2 used to handle unicode, and were a sore thumb for a long time in the python 3 parsing system. 'file hash' content types now have a 'hex'/'base64' dropdown, and do decoding to raw bytes at a layer above string parsing. on update, existing file hash content parsers will default to hex and attempt to figure out if they were a base64 (however if the hex fails, base64 will be attempted as well anyway, so it is not critically important here if this update detection is imperfect). the 'hex' and 'base64' _encode_ types remain as they are still used in file lookup script hash initialisation, but they will likely be replaced similarly in future. hex or base64 conversion will return in a purely string-based form as technically needed in future
- updated the make-a-downloader help and some screenshots regarding the new hash decoding
- when the json parsing formula is told to get the 'json' of a parsed node, this no longer encodes unicode with escape characters (\u0394 etc…)
- duplicating or importing nested gallery url generators now refreshes all internal reference ids, which should reduce the liklihood of accidentally linking with related but differently named existing GUGs
- importing GUGs or NGUGs through Lain easy import does the same, ensuring the new objects 'seem' fresh to a client and should not incorrectly link up with renamed versions of related NGUGs or GUGs
- added unit tests for hex and base64 string converter encoding
next week
Last week of the year. I could not find time to do the network updates I wanted to this week, so that would be nice. Otherwise I will try and clean and fix little things before my week off over Christmas. The 'big thing to work on next' poll will go up next week with the 423 release posts.
Version 419
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v419/Hydrus.Network.419.-.macOS.-.App.dmg
linux
I had a great week finishing editable system predicates and making some long-running search situations faster and snappier.
editable predicates
As a reminder, if you shift+double-click on the search terms (predicates) in your current search, you can now edit them! I also added an right-click 'edit' menu entry, if you do not like the shift+double-click.
OR and system:rating predicates are now editable. Also, 'invertible' predicates like inbox/archive or tag/-tag will now be stacked in the editable panel as buttons you can click to flip.
You can now set the default for any editable system predicate type! Little 'star' buttons beside the panels now let you set the current value as the new default.
The editable system predicate panels also remember the five most recently entered predicates for each type, and these are stacked up top as quick buttons. The UI here gets a little tall and jank, but see how you like it!
You can now shift+double-click tags in a place other than your current search predicates (e.g. the 'selection' tags in center-left) to add the selected tags as a bundled OR tag. Shift+middle-click does the same for a new search page. There are also right-click menu entries for these.
All tag lists now support drag selection, so you can click a tag, drag down three rows, and all will be selected without having to click more.
faster and snappier searches
First off, any 'wildcard' search for autocomplete results or files that is A) 'complex', which means it has an asterisk in a non-rightmost character position, like 'bl*ah' or 'bl*ah*', and B) does not start with an asterisk, should now be much faster than before to find results. Tag/file wildcards that start with an asterisk are still basically hellmode in all situations.
Secondly, pretty much every component of tag-based search, from autocomplete lookup to tag, namespace, wildcard, or system:num_tag searches, now reacts faster to 'cancel' signals. This means that if you enter system:num_tags>4, and it is taking ages to finish, if you stop the search or update it with a new search predicate, the old search should stop its work more quickly. Also, when you type in autocomplete, old unfinished searches are often being quickly cancelled in favour of new ones as you type. This process is now much faster, particularly when wildcards are involved.
client api virtual tags
The Client API now provides 'display' tags, as calculated by the new virtual siblings and parents systems, if you want to show 'pretty' tags to your users. Check the help for /get_files/file_metadata for the new service_names_to_statuses_to_displayed_tags structure.
a note on macOS Big Sur
Hydrus currently does not run on Big Sur. It is a library compatibility issue, as my 10.12 macOS laptop is getting a little old. I am working with some users to figure out a better build solution, so please hang in there.
full list
- tag lists and editing predicates:
- you can now set the default value for any editable system predicate. a star button beside each panel lets you set or clear the custom default
- all editable system predicate panels now put 'recent' predicate buttons up top, for the five most entered predicates of the respective types. this is a little jank and grows pretty tall with multi-pred-type panels, but let me know what you think
- all tag lists now support drag-selection!
- taglists now have 'open a new OR page' menu entry when more than one tag is selected
- when taglists can change the current search, they now have an 'add an OR to current search' menu entry when more than one tag is selected
- OR Predicates are now editable! they launch their own little autocomplete input that is a little jank because you can technically make nested ORs, but it works!
- system:rating is now editable! it launches the whole stack every time. the stack alignment is messed up though :/
- invertible predicates (inbox/archive, tag/-tag, etc…) now flip on double-click only if you have one selected. if you have more than one selected, they appear as invertible buttons along with the rest of the edit UI
- the active search predicates taglist now has an 'edit search terms' menu entry, if you find shift+double-click a pain
- when you shift+double-click on more than one tag to add them to the current search, this is now added as an OR
- similarly if you shift+middle-click on more than one tag, the new page is now an OR
- when editing predicates, edited predicates now stay selected
- shift+clicking on an already selected tag no longer adds any new selections (i.e. shift+click filling-in). this should make it nicer to do shift+double-click on selections. furthermore, the 'last clicked' focus ghost (from which a shift+click selection cascade starts) on tag lists is now cleared on edits or removes, which should reduce some other crazy/annoying select behaviour here
- the list of active search predicates now correctly initialises sorted
- entering hex hashes into system:hash or :similar_to now has unified hash parsing, auto-removes 'md5:'-style prefixes, and presents detailed error information when a hash is too long or short
- .
- faster and snappier file and tag searching:
- searching for files by complicated wildcard (i.e. a search phrase that includes an asterisk in a non-rightmost character position) is now greatly optimised when the tag does not start with an asterisk (e.g. 'sm*l' is now much faster, '*all' is still hellmode), and now cancels (due to hitting the stop button or changing the query before results come in) much faster thanks to a new unified results fetching and cancel-checking routine
- rewrote my autocomplete tag search to use the new namespace and subtag lookup code from the virtual siblings and parents system, unifying lookup logic and benefitting from the same new complicated wildcard optimisation and fast-cancel tech
- autocomplete tag count aggregation (a later step, after the initial lookup) benefits from a little faster cancel tech
- all file queries based on tag, wildcard, namespace, tag count, and tag existence now use the new fast-cancel tech. if you put in a 'has >4 tags' query and it is taking ages, changing the query or just hitting the 'stop' button should now free up the db pretty fast
- related tags suggestions also gets the cancel tech and is now more timing precise for tags with either huge or tiny count
- .
- client api:
- the /get_files/file_metadata call now returns a service_names_to_statuses_to_displayed_tags structure, which reflects the sibling-collapsed and parent-added tags, as displayed to the user in UI. the help is updated to reflect this
- the client api version is now 15
- .
- the rest:
- fixed an issue where regenerating the tag definition search cache would not tidy up the 'I am busy' modal dialog once it was done, resulting in a soft lock
- fixed another upnp error handling bug, this time in the upnp daemon
- updated Qt to 5.15.2 on Windows and Linux builds. this should fix the unusual button clicking area problem for some custom styles
- .
- boring specific code changes:
- wrote widgets to edit invertible preds and OR preds
- pulled the messy rating code out of the rating system predicate ui code to their own widgets
- wrote some special predicate ui definitions and initialisation handling for OR preds and grouped 'multiple' preds (for ratings)
- refactored search and predicate ui code to a new 'search' module
- refactored collect and sort widgets away from search code
- misc layout improvements for system pred edit ui
next week
I was not able to get to the siblings/parents logical bugs, nor was I able to catch up with my messages like I wanted. These will be top priority next week, and then I am going to start planning the network update that I hope to finish the year on.
Version 421
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.macOS.-.App.dmg
linux
I had a good week. I fixed some important bugs, and cleaned up some core access and maintenance code. This week's changelog gets pretty technical, which you can safely ignore. Database works better™.
highlights
I fixed a problem importing files with 'only add tags that already exist' checked in tag import options. Sorry, this was a stupid typo. I added a unit test here to ensure it doesn't happen again.
When you search for potential duplicates from the duplicates page, this now happens in a non-interrupting popup in the bottom-right. You can keep searching and browsing while it works. All duplicate pages sync with each other better, too, and share more CPU work.
Thanks to a user's contribution and help, the 'discord drag and drop' BUGFIX mode, which is really a 'export files to a temp location before starting the drag and drop' mode, now supports the same 'filename pattern' you see in the export files dialog. Check it under options->gui. This is experimental, but the first step to automatic export with nice filenames. If you care about export filenames, let me know how it works for you, and what sorts of filenames you want to make in future. I really want to update the old 'filename pattern' system to something with conditional 'if' statements.
If you are a new user and had trouble syncing to the PTR in the past week with an error about an invalid tag, you got caught by my new 'invalid tag catching' code being overzealous. This is now fixed, so if your PTR automatically paused due to the error, please hit services->review services->PTR and then unpause it. When a repository gives an invalid tag (we think there are about 17 really broken tags in the PTR atm, from older days), the client now recovers.
I played around with core database modes this week. Default values are now tuned a little better for modern use. Unfortunately, the planned new mode for HDD users did not pan out due to architectural problems, but I think that HDD users will experience better write performance nonetheless. For those who want to experiment more, I have finally properly documented the different launch switches for hydrus here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html
full list
- misc:
- thanks to a user's contribution, added the export 'filename pattern' to the discord drag and drop mode, under _options->gui_. this lets you auto-rename files in this export mode. I like how this works, but the overall pattern-based filename creation system really needs updating. let me know how this works for you, and I'll finally start the job to update filename generation
- fixed a bug when importing files with the 'only add tags that already exist' filter active, and added a unit test so this should not fail due to a typo again
- fixed an issue where ctrl-selecting on taglists was weird, where any mouse movement during ctrl+click would deselect. drag select and deselect can now only start when the drag crosses two indices
- prototyped a basic profile mode for the client api. it is insufficient (due to the asynchronous nature of twisted), but a start
- when the client catches an invalid tag with the new error handling code, when it shows you that bad tag in a popup, it now clips that to 24 characters (some PTR invalid tags are just a few hundred null characters in a row, wew lad)
- the client now recovers from a repository giving it a new invalid tag definition. all such tags are, for now, called 'invalid repository tag'. a plan to auto-hide these tags clientside and fully eliminate them serverside will come later
- the clipboard url watcher settings should stick a bit more firmly. those users who had trouble, please let me know how you get on
- fixed an issue editing duplicate action options when they contained tag or rating preferences for services that no longer exist
- I think I fixed some issues getting autocomplete results when you type the whole namespace before moving on to the subtag. when you hit 'namespace:', it should invalidate the old cache and start a new search
- when the database is given content updates for services that no longer exist, those content updates filtered out of UI update broadcast
- fixed an issue where URL status check could fail when the url map contained orphan hash_ids. proper orphan clearance will come later
- reduced overhead of tag filtering, which should improve display speed of taglist for very large pages
- parents should now work through repository processing faster. periods of 2 rows/s at the end up of updates should be up to 100 times faster
- .
- duplicates search improvements:
- potential duplicate search now works in the background! it will not interrupt you and is easily cancellable. duplicate search pages disable their search buttons while it is going
- the search distance in duplicates pages is now synchronised across all pages–when one updates, they all do
- all the updates to potential search maintenance numbers are now routed through one cached manager. updates here are repeated less often
- misc cleanup for duplicates page
- .
- database modes:
- a new 'program launch arguments' help page now talks about all the available command line switches, here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html
- added the '–db_journal_mode' launch switch to set the SQLite journal mode. default is WAL, permitted values are also TRUNCATE, PERSIST, and MEMORY
- ensured –db_synchronous_override was hooked up correctly
- the old disk cache options under _speed and memory_ are removed, along with various deprecated disk cache load calls and code
- fixed some shutdown maintenance check logic that was saying 'I think a vacuum is due' when it wasn't actually true
- db_journal_mode, synchronous value, and no_db_temp_files is now shown in _help->about_
- .
- technical database nonsense:
- PERSIST is new to hydrus, and _may_ in future versions of SQLite be boost performance for HDD drives with larger databases (e.g. those that sync to the PTR), although unfortunately in our case (which uses multiple ATTACH databases), it seems current SQLite must ultimately treat this as DELETE, as here https://sqlite.org/atomiccommit.html#_clean_up_the_rollback_journals. damn
- hydrus now tries to always trim WAL (and PERSIST, if it worked) journal files down to 1GB after commits (which happen every 30 seconds), so giganto WALs should clear up promptly after big work is done
- hydrus no longer refreshes the database connection every thirty minutes, meaning WAL journal files will persist (and hopefully regularly clip back to 1GB when exceeded), which should improve some elements of long-running write performance, but may result in some surprise memory issues, we'll see
- in lieu of the db connection not refreshing, the memory database now reattaches every ten minutes, which _should_ stop it leaking in certain situations
- when in WAL journal mode, the hydrus db now cleans up any lingering checkpointing work every half hour
- after testing and feedback from users, the database is now default SQLite synchronous 1 (down from 2) when in WAL. the db is still consistent, so sudden program stop (crash, power cut) should not result in software-caused corruption, but the database may lose more than just the last 30 seconds of work. this speeds up tag processing in an SSD test environment by approx 33%
- the 'no_wal' (TRUNCATE) and 'db_memory_journaling' (MEMORY) launch switches remain valid but are now deprecated
- improved launch switch code generally
- boosted cache size for each of the four db files to ~200MB-this will likely become a launch argument in future, along with some other specific db values
- the client and server no longer disconnect from the db to check whether it is possible to vacuum databases
next week
There are two more work weeks in the year. I will continue working on small jobs and cleanup, but the main focus is now to update ancient network and service code to improve client-PTR sync and communications. I'll fix up buggy janitor tools and hopefully add some nice filters so clients and servers can better manage what they sync.
Since a 'next big work' poll is coming up on Christmas, on Saturday I'll post here and in discord about the current big list and ask for more things to add.
Version 420
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v420/Hydrus.Network.420.-.macOS.-.App.dmg
linux
I had a great week fixing a whole bunch of bugs.
bugs
I fixed taglist drag-select, which was not moving the to-be selected indices down with the scroll. Sorry for the trouble here. You can now also ctrl+drag-select to deselect.
There was a bug in the new virtual siblings and parents lookup system that meant some grandparents and siblings were not appearing. For instance, for parents, 'samus aran' might have 'metroid', and 'metroid' would have 'nintendo', but 'samus aran' would not have 'nintendo'. Thanks to help from users, I was able to reproduce it and fix the problem. When you update, the client will spend a few seconds regenerating the lookups and finding the missing links. It will queue up a bit more work for the background display sync to do later on. In my test situation, the PTR went from 189,000 sync rows to 192,000.
Autocomplete results in manage tags now list parents beneath every tag they apply to. Previously, parents could only exist in the list once, so they were accidentally de-duping and only ending up beneath the last applicable tag. Now you get plenty.
Also, these 'write' autocomplete results now show sibling and parent data for the tag that matches your input text even if that tag has no count. When that tag has sibling data, all the siblings are loaded as well. This sounds obscure, but you'll notice when you next start entering tags on a service with a lot of siblings. It should make it easier to quick-type complicated tags in manage tags.
Typing to get autocomplete results in search pages with thumbnails is now significantly faster and more responsive when 'searching immediately' is turned on. This routine gathers count-accurate results based on the thumbnails in front of you and then sends that data down to the database for sibling population. This suddenly got laggy with the new virtuals system, but I have improved how it schedules its searches and performs its tag lookups, and it should be much better now.
An unusual situation with a 'null character' tag that could not be displayed in the client should be better. This tag is detected, the rendering problem caught, and the user notified. The database routine that corrects bad tags now also fixes this tag and forces a tag presentation refresh once it is done.
speed
I spent a bunch of time in the siblings and parents system this week, and autocomplete and tags in general, running profiles on complex data and optimising various calls. I also sped up a critical new optimisation used across the program. Tag searches, autocomplete lookups, tag processing, tag display sync, and anything that runs frequently in small amounts should now be a bit faster. The only thing that may run a bit slower is tag display sync for very complicated tag parent situations, due to the more thorough logic in the fixes above. Thank you to users who helped with profiling here.
Tag display sync now only tries to run every 30 seconds (up from every ~3 seconds) when allowed to run in 'normal' time. It also takes breaks when it thinks a bunch of other work is going on. It was running hotter and faster than we needed, and it made the client too laggy, so I am pulling it back until I can make it more intelligent. I am also considering breaking the main display sync maintenance job into even more granular pieces. I will keep working.
full list
- misc:
- fixed the bad position indexing when drag-selecting taglists that were scrolled down. this also caused some weird selection when scrolled and clicks included a little mouse movement. sorry for the trouble!
- ctrl+drag-select now deselects!
- fetching tag autocomplete results when you have thumbnails and 'searching immediately' on, which has been way too slow recently, now cancels much faster. in some large page situations, it was adding multi-second lag on the first character-press. it also runs faster overall
- hydrus should now deal better with invalid tags that contain the null character (there it one we know about on the PTR, from a decode of botched Shift JIS, which could crash the client from too many errors during critical paint periods). when a tag like this turns up in a taglist, thumbnail, or canvas background, it now renders as an appropriate 'invalid tag' string, and a one-time 'woah, bad tag, run fix tags now' popup appears
- regular tag cleaning now looks for and removes null characters, so all new sources of these bad tags should now be eliminated
- _database->check and repair->fix invalid tags_ now fixes tags with the null character. it also fixes tags so broken that after cleaning they have no subtag left. it also now forces a full media tag reload when it is done for all media
- the 'regen storage mappings', 'regen display mappings', and repopulate from cache' database routines now have an additional step where you can order them only to work on one tag service, so regenning or repopulating local tags, which usually takes a couple seconds, doesn't need to wait two hours for the PTR to go as well
- added some menu help to the 'profile modes' debug menu, and gave 'reducing program lag' help page a pass
- fixed virtual display regeneration on service delete
- .
- parents and siblings:
- fixed situations where some grandparent and sibling relationships would not appear in the virtual system. it was a bug when certain links of a multi-part display 'chain' were updated at different times. when repopulating chain data, the sibling and parent update routines now correctly chase their complete chains both when wiping ideal data and repopulating from raw data, hitting all levels of the chain, ensuring to go back up and down chains when there are multiple grandsiblings/children/parents, and chasing parents where one or both members have better siblings. thank you to the several users who reported and helped figure out this problem, which was not simple to reproduce (issue #725)
- your ideal display data will be regenerated on update, which should not take more than a couple of seconds. it will likely correct some siblings and add some grandparents to be filled in by the siblings/parents sync. my PTR test environment went up from about 189,000 display rows to 192,000
- while sibling and parent lookup is more thorough (and hence more expensive), I also optimised many parts of lookup week. I believe tag display sync and tag processing will be much faster for tags with simple sibling and parent relationships, and slightly slower for tags with complex relationships and many instances to files on your drive. as always, let me know what sort of processing speeds and lag you get, and if you know how to make a db profile, please send them in when it gets bad
- when a 'write' autocomplete results list includes parent expansion rows (as in _manage tags_), parents now show duplicated and properly for all the tags that have them, including siblings and other children/grandchildren (previously, a parent label could only exist once in a list, which meant parents were ending up hanging off the last valid tag for which they applied)
- 'write' autocompletes now show results that exactly match the text entry, and all their siblings, when they do not have count but do have sibling or parent data. so, if you type in 'samus aran', and it has a sibling to 'character:samus aran', but 'samus aran' doesn't actually have count, you now get it and all siblings anyway. this may need tuning, but it solves a persistent and annoying lookup and quick-sibling-access problem in _manage tags_
- copying tags and their indented parents now removes the parent indent whitespace
- tag sync display now takes way longer breaks (now 30 seconds, was 2.5) between 'normal' background work periods. this thing was hammering people far harder than needed and could clog up db write/commit time and nobble UI responsitivity when big bumps collided
- the tag display maintenance manager now also tries to detect when many siblings or parents are streaming in (from a migration or a repository process with a heap of data), and pauses work while that continues
- greatly sped up mass imports of sibling and parent data, either from tag migration or big dialog pastes. what was 40 rows/s should now be about 1,000 rows/s
- fixed the database menu's 'regenerate tag parents lookup cache', which wasn't hooked up
- .
- boring changes:
- gave tag parents and siblings update, regen, and chain fetch a full pass, correcting bad queries to fix the above, fixing raw pair chain level navigation and parent-sibling idealisation, and optimised these lookups as well
- fixed some tag_id vs ideal_tag_id nomenclature (and related bugs) in tag parents cache
- optimised 'all known tags' autocomplete count fetching a little. tag autocomplete and search should be a bit faster in this domain
- reduced display sync pre-processing overhead by about 30% with a better random pair sampling routine
- reduced the overhead of my now very commonly used single integer memory table select optimisation. this now recycles tables after use, which reduces overhead about 50% in small number scenarios. all features of the database will enjoy this speed improvement, particularly small repetitive tag lookup jobs (such as the new display sync and repository tag processing)
- reduced overhead on some sibling chain lookup code
- reduced overhead on the sibling lookup used by manage tag dialog taglist
- reduced overhead on some parent chain lookup code
- tiny optimisation on single sibling chain lookup
- sped up the ancient OG single tag->tag-id fetching routine, seems to work about twice as fast now
- more misc optimisations, mostly list/set/dict comprehension rewriting to reduce overhead, across virtual sibling and parent code
- added a full combined siblings and parents unit tests for the main missing parent chain link problem solved this week
- added a full combined siblings and parents unit test for large real world data added in multiple pieces
- 'a file identifier was missing!' critical errors now print a stack trace to the log for further debugging info
- updated the 'help my db is broke.txt' document with a couple new comments
next week
I want to do some code cleanup, catching up on bad old code and making the duplicate potentials search non-interrupting. I'll also prototype a database mode that may improve performance on HDDs. Other than that, I want to properly plan and start work on the big network improvements that I will finish the year on.
Version 418
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v418/Hydrus.Network.418.-.macOS.-.App.dmg
linux
I was ill this week, so I didn't have as much work time as I wanted. I did get some basic system predicate editing done though, if you want to try that out!
editing system predicates
This works, but it is basic, not super polished.
If you shift+double-click on a system predicate that is in your active search (like 'system:filesize>200KB'), it should now launch a dialog to edit it in place. You can also edit several at once. This should make doing several 'time imported < 7 days ago' … '6 days ago' … '5 days ago' searches in a row to chase something down a bit easier!
If a predicate is too simple for a panel but has an 'invert' version (e.g. archive/inbox, has audio/no audio), then it flips to that invert on a shift+double-click. This also works for regular tags, flipping between 'tag' and '-tag'.
As I was short on time, I did not finish everything and could not add too many bells and whistles. The stacked edit panel can look a little ugly, there is no menu command to edit, selection disappears after an edit, I have no edit support for 'system:rating' or 'OR' predicates, and I was unable to reintroduce optional default values for system predicates. I'll go at this work next week.
In any case, give it a go and let me know what works well and what gives you trouble. I am not super enthused about shift+double-click as the edit shortcut, but it is a good compromise. I'll likely end up adding it to the shortcut system so you can edit it.
BTW: A couple of clever system pred options that were accidentally hidden are reintroduced to the new options->search page today.
full list
- almost all system predicates are now editable if you shift+double-click them! you can also edit several at once in the same dialog
- if you double-click on any predicate type that is not editable but does have an inverse version (e.g. archive/inbox, has audio/no audio, and tag/-tag), the inverse version(s) will be swapped in
- all legacy custom system predicate defaults are eliminated this week. the panels now show a fixed default on launch, and will get a flexible favourites system in future, along with 'recently entered' quick buttons
- restored the 'show system:everything' and 'hide archive/inbox' options, which were inadvertantly hidden when file system predicate defaults were hidden, to the new _options->search_ panel
- fixed the borked list height for the file viewing statistics system pred panel checkbox lists
- fixed an issue where namespace:anything predicates would not propagate to new pages on 'open page with these tags' commands
- .
- boring code specifics:
- updated almost all the system predicate panels to take arbitrary initialisation values, and wrote a 'can I edit this' test for all predicate types to help some finnicky which-panel-and-pred-to-use issues
- wrote some new filtering code and a little UI to handle editing of system preds
- cleaned up some of the taglist item activation code
next week
I am feeling better already, so I think I'll be good for a normal work week. I'll try to polish these editable predicates off, and then I want to deal with some reported bugs in the parents and siblings logic.
Importing Issue
I'm having an issue importing files from Sankaku Complex. When I tried to import new favorites from the account I have on there (using the tag "fav:[account name]") it ignores all of the files. It should only be ignoring what I've already imported. Why is this happening? I'm logged in through Hydrus Network as well.
Version 417
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v417/Hydrus.Network.417.-.macOS.-.App.dmg
linux
I had a great week working on a variety of small jobs.
The hydrus network version updates this week, so any clients that currently sync with the PTR will need to update to continue syncing. There is no rush for this.
all misc this week
First of all, the hydrus network version increments from 18 to 19 this week. Clients and servers can only talk to each other if they are running on the same version, and the PTR will update today, so if you want to keep syncing with the PTR, you will have to update. This version update was to ensure that all clients syncing with tag repositories are now on the new virtual sibling and parents systems. There is no rush to get this done, and no penalty if you delay–you will just get a polite popup saying there is a version mismatch, and your PTR service will be paused.
If you get a popup about the network version mismatch in an older version, after you update to 417, go to services->review services and make sure the PTR service panel is unpaused. You might also want to hit 'refresh account'.
I added some shortcut actions to the 'main window' shortcut set–one for 'run all export folders now', and several for navigating the current page selection. You can navigate left, right, and to the leftmost or rightmost tab of the current row. Unlike (shift+)ctrl+tab behaviour, the left/right actions jump through page of pages when you are at a boundary, letting you step through all of your tabs in a depth-first manner.
I also fixed some macOS shortcuts. Control and Command should now both be recognised and labelled correctly, and trackpad scrolling through the media viewer should be at a more appropriate pace.
The Newgrounds downloader is updated. It should now be able to get all the content for artists with more than 60 or 70 items in their galleries. Also rolled in is an updated parser for realbooru, who apparently recently changed their site format.
I messed up the UPnP error parsing last week–sorry if you got any of this! I cleaned it up, wrote some overdue tests so I won't make this mistake again, and gave the network->data->manage upnp dialog a full pass. It should do all its long-delay jobs without blocking the UI.
Advanced: URL Classes now have 'header override' settings. Any time an URL is hit, those headers will be inserted into the request! I needed this to get the updated Newgrounds gallery parser to work, if you want to see an example. Be careful with this, but it may just solve several of our more tricky problems. Also, sorry, the URL Class edit UI is increasingly a hellstack, I will rework it at some point into tabbed pages or something.
full list
- the hydrus network version is increased this week from 18 to 19. clients and servers can only talk to each other when they are on the same version, so please update your clients if you wish to keep talking to the PTR, and your own servers if you have a home network setup or similar. if a server and client are on different versions, you will get a polite error when they next try to talk, and sync will be paused
- added 'run all export folders now' shortcut command to 'main window' shortcut set
- added shortcuts to the 'main gui' shortcut set for navigating the currently selected page. you can move left, right, to the leftmost on the current row, or to the rightmost. the left and right will cycle up a page of pages layer when at left/rightmost boundaries, letting you iterate through all pages in a depth-first manner
- updated the default newgrounds parser to deal with artists with more than 60/70 items in one art gallery (essentially, some clever 'next page' fetching now occurs to get older info that in your browser is drawn in as you scroll down). if you have some subscriptions for artists where you know this is true, try doing a full reset on them
- added realbooru to the hydrus defaults. they also apparently just switched away from a gelbooru 0.2.x site, so if you have a gelbooru parser with a realbooru example URL, I remove that example URL
- updated the page initial media load routine to my new async job
- updated the imported file presentation page-publish routine to my new async job
- when drag and drop or import file presentation now wants to add files to a page that is not yet fully loaded (rare, but possible for large sessions), that page now remembers the files it should add and appends them once load is done. these files-to-be-added are also preserved through a session save, if the client is closed before this long-loading page is initialised
- updated windows mpv, the reported api version is now 1.09
- updated windows ffmpeg to 4.3.1
- updated windows release to sqlite 3.33.0
- updated windows opencv to 4.4.0
- just a little thing–I took the source links out of the release post. anyone running from source is probably pulling straight from the github repo anyway
- cleaned up some misc inelegant string code
- misc other cleanup
- .
- macOS shortcuts:
- the client's shortcut system now detects macOS-specific 'scroll start/end' states, and will not spam scrolls or errors when these states are held
- the client's shortcut system now attempts to detect artificial trackpad scroll/wheel events, and adapts the relative speed of scroll event generation according to the respective trackpad velocity. let me know how the hell this works for you in media viewer etc… (issue #710)
- the client's shortcut system now detects Control and Command as separate and reliable modifiers in macOS, with correct shortcut string rendering (issue #717)
- .
- upnp:
- fixed the awful typos in the upnp add-mapping error handling I changed last week. I am sorry for this!
- improved the async mappings and external ip fetch routines in upnp dialog. closing the dialog while a job is going on should now be completely ok
- upnp dialog add, edit, and delete actions are now async (they won't hang the UI while they work)!
- all the upnp async jobs should now disable the main list controls while they work
- fixed the 'edit' action on upnp dialog to correctly remove old and existing mappings depending on what was edited
- when adding a mapping for an (external_port,protocol) that is already mapped, the upnp dialog now asks if you want to overwrite, rather than just failing with a notification
- after an async action in upnp dialog, and a mappings refresh triggered, the cached external IP should now be properly restored to the status area
- pulled parsing code out of upnp code and wrote some proper unit tests for this so stupid typo errors should not happen again
- .
- parsing:
- subsidiary page parser separation formulae that throw an exception will now be ignored, as if they parsed nothing. in the weird case that you might receive json or html, you can now create subsidiary parsers for both types, and the one that fails will do so gracefully and silently
- URL Classes now have a key->value 'header override' value. any time one of these URLs is hit, these headers are added!!! be careful with this, but it may solve some tough problems. also, sorry, the URL Class UI is becoming a hellstack, I need to break it into tabs or similar
- .
- client api:
- added documentation for the new add_files commands, delete_files, undelete_files, archive_files, and unarchive_files
- added unit tests for the new commands
next week
I am due a 'medium size job' week, so I would like to make system predicates and OR predicates editable after they are entered.
Version 416
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v416/Hydrus.Network.416.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v416.tar.gz
I had a great week catching up on a mix of smaller work.
highlights
The manage tags dialog is now more efficient at making changes for very large numbers of files. My test application of 6 tags to 10,000 files went from 52 seconds to 4.8! A variety of other tag presentation updates should also benefit from the optimisations here.
I reworked my hacky multi-list sizing code. The various instances of the final column sizing weird should be much better. The horizontal scrollbar is enabled again.
A new checkbox under options->files and trash now lets you prefix any copied hash with the hash type, like "md5:2496dabcbd69e3c56a5d8caabb7acde5". If you often paste hashes into boorus or other search engines that take this format, this should make it a bit quicker!
I am rolling in default thread watchers for warosu, prolikewoah, and smuglo.li. Warosu seems to have some CloudFlare stuff going on, so your mileage may vary.
full list
- misc:
- the new siblings and parents taglist menus now copy just the actual tag when you click, excluding the 'ideal/child/parent:' prefixes
- added a checkbox to _options->files and trash_ that allows you to automatically prefix hashes copied to clipboard with their hash type in a booru-lookup friendly manner, such as "md5:2496dabcbd69e3c56a5d8caabb7acde5"
- the media viewer now remembers if it was previously maximised when you set it to un-fullscreen (before, it would always restore-window-ise)
- fixed the 'test address' button in _manage services_ for hydrus administration services
- improved the 'add upnp mapping' error handling to better catch 'already mapped' error, with separate errors for redundant, already-on-but-wrong-port, and already-on-another-computer
- improved error handling when saving objects to the database, particularly for encoding or giganto-size-session errors
- rewrote my tag sibling lookup unit tests to deal with more situations
- wrote similar fairly comprehensive tag parent lookup unit tests
- .
- new downloaders:
- rolling in a user-created thread watcher for warosu. it may be CloudFlare hampered depending on your situation
- rolling in a prolikewoah thread watcher
- rolling in a smuglo.li thread watcher
- .
- multi-column lists:
- spent a bunch of time cleaning out how I calculate multi-column list preferred initial width/preferred current width/minimum width, and made the final column more flexible in its resizing. instances of dialog suddenly getting gigantic because of a final column that wants to size itself at 1,000px should be completely gone, and lists that are shrunk due to non-last-column resizing will now adapt to this situation and not try to flex back to total initial width.
- multi-column lists now have horizontal scrollbars again for those situations where the parent window is thinner than their (now better calculated) minimum size
- improved the multi-column list num_rows height calculation, it should have less empty space at the bottom for lists that grow as items are entered into them (such as in the download pages)
- .
- manage tags megajob speedup:
- sped up manage tags final application step when entering many tags for many thousands of files at once
- optimised UI-side per-file tag cache (re)generation, reducing overhead and surplus work
- granularised UI-side per-file tag cache (re)generation based on the four current tag display contexts–now, if a system (e.g. manage tags dialog) only needs storage tags, the different display tags do not need to be regenerated
- optimised all tag filtering, which is also used in UI-side tag cache regen
- overall, giganto manage tag dialog jobs should now be faster in several ways. on my dev machine, adding 6 tags to 10k reasonably tagged files went down from 52s to 4.8. even larger jobs will still need a lump of CPU time, but they should scale more efficiently (what was previously O( num_tag_changes x num_total_mappings ) is now O( num_total_mappings ), and better at that)
- when a huge number of tags is added at once in the manage tags dialog, 'recent tags' is now populated more carefully
next week
I want to do more of this work, catching up on older items and things that have piled up in the past few weeks. There are eight weeks left in the year. I want to do one large round of network updates, but otherwise just more like this!
I will be updating the 'network version' of hydrus next week, which will mean any client that talks to the PTR will need to update to keep syncing, or it will get a polite popup message telling them to. This will guarantee that all PTR clients will be on the new virtual sibling and parent caches, containing the problem of sibling and parent based surplus/faulty tags being uploaded. There should be another version update before the end of the year when I commit the planned network changes.
Version 415
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v415/Hydrus.Network.415.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v415.tar.gz
I had a mixed week. I greatly reduced the lag that 414 could get after large sibling/parent changes, but unfortunately I could not get much else done.
all misc this week
I had wanted to do a bunch of small work this week, but my schedule fell apart a bit, and then the 'one last thing' for the new parents cache, which was rejiggering the background maintenance code to do smaller jobs at once, ended up being very difficult. I worked hard on figuring out a method, and then when I was ready to move on, I realised there was a design flaw and had to re-do a large part of it! In any case, it now works much better–even if you have thousands of tags with one common parent, they can now calculate one at a time, nice and quietly in the background. Some million-strong siblings and parents might still be a bit too laggy for an HDD to calculate in normal time, but an SSD should be generally ok. If it still lags for you, you can make it only work in idle time under tags->siblings/parents sync.
In that vein, the new tag display dialogs now have a nice green/red text label to describe the current maintenance options, so you better know when to expect any changes to be calculated.
I fixed and improved a couple small things, but other than that, the main highlight is options->gui pages now lets you put the main window's page tabs on top/left/right/bottom (previously it could only do top/left). This now also updates as soon as the options dialog is applied.
full list
- in _options->gui pages_ you can now set the main window's page tab alignment to top/left/right/bottom (previously it was just top/left). this property now updates for all page of pages on options ok, it no longer needs client restart (issue #642)
- the maintenance task that migrates tag display from the current values to the ideal application now works in significantly smaller steps. big lag from adding hundreds of childen to one parent (or similar siblings) should now be radically reduced
- rejiggered some layout in the new tag display dialogs
- added green/red texts to the new tag display dialogs to talk about when sync can work atm and how fast to expect changes to apply
- reordered the new tag 'siblings/parents info' right-click menu so the dynamic 'has x siblings/parents' submenus are on the bottom
- added basic client api calls for /add_files/…, delete_files, undelete_files, archive_files, and unarchive_files. they take 'hash' and 'hashes' parameters. I am throwing these out at the end of the week, so they don't have documentation or proper unit tests, but feel free to play with them (issue #393)
- sped up some UI refresh on content update for very large sessions
- sped up right-click tag/file menu any/all select actions on very large file sessions
next week
I'll try to get some more small things done, ha ha. I have IRL election day stuff to do, so it'll unfortunately be a slightly light week again. Release should be on Wednesday as normal, maybe a little late.
Version 414
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v414/Hydrus.Network.414.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v414.tar.gz
This is the first release of a complicated new system. If you are a cautious user, feel free to wait a week for me to iron out any surprise bugs.
I had a good four weeks converting tag parents to the new 'virtual' database cache.
The release does not need to do a big update this week. It will ask you if you have an SSD or an HDD (EDIT: with a special caveat for PTR users, see below), but that is it. All the mandatory heavy work that came in with 407 now happens in the background.
parents
tl;dr: You don't have to do anything. If you haven't heard of a tag parent before, no worries. The database should work better now.
I did the same to parents as I recently did with siblings. They are now virtual, which means when the database needs to 'fill in' a missing tag because of a parent relationship, that new tag now does not actually exist in storage, only in a special computed database cache. If parents change, the implications in this cache can be recalculated with no permenant changes made. Parents are now undoable.
Parents are also comprehensive. Previously, due to different logical holes, not every parent was filled in correctly. There were several retroactive holes. Parents now always apply to all their children, and they adjust to siblings and sibling changes.
Also, you can now apply parents across services. This matters less than siblings, but if you have particular parent needs, you can now set your 'my tags' parents to work on the PTR, or remove all parents entirely, under the new tags->manage where tag siblings and parents apply dialog.
This took a bunch of work. I am happy with all of it except how parents now display in manage tags. It was complicated to insert 'ghost' tags to represent parent implications and I just ran out of time, so for now, tags with parents just get a '(2 parents)' suffix after their tag label. You can now right-click on tags in most places to see all their sibling and parent data.
smoother sibling/parent calculation
LATE EDIT: It seems the PTR has one very large parent structure that causes a one-time lag of a few minutes even for SSD users. This is happening because we are catching up on all the parents at once. You will get a popup when you update asking if you are on SSD or HDD. If you select SSD, be aware the client is likely to lock up for a few minutes soon after boot. Just let it do its work, or select 'HDD' on that update dialog and it will happen when you are not looking.
I was not happy with the large lag introduced when I added siblings. Ten minutes to see a new PTR sibling application is too long, and HDD users were almost completely nuked. Rolling parents into the same system would have made it worse. So, another aim of this big work was to reduce that calculation time and smooth it out in the background.
This is now done. Applying changes to siblings or parents should be very quick in all cases, but those changes may not instantly appear in your actual tags. The client now keeps track of what it currently has and what it should ideally have, and migrates silently in the background, in pieces. Most of the time, changes should appear within a second or two, but if a whole bunch just changed, it will take longer. You can check what is going on under tags->sibling/parent sync.
I also improved the recalculation code. The sibling and parent systems are now unified and expressed in a more granular 'tag implication' algebra, and rather than clearing bad numbers and regenerating them from source, the client now defines the difference between the current implication structure and the ideal, and only applies those differences. Reducing all structure changes to 'add implication' and 'remove implication' allowed me to implement this migration tech and optimise the queries significantly.
This is an entirely new system. If it causes lag for certain situations (e.g. if you have a 100k+ file session open), let me know. You can always turn it off under the sibling/parent sync menu. If you select 'hdd' during update, the migration will only happen in idle time, but I'd be interested to know how it performs in active time as well, if you are feeling adventurous.
If you are running a pre-407 client, you should be able to update straight to this week and skip almost all that work. Should™.
full list
- tl;dr: you don't have to do anything. if you haven't heard of a tag parent before, no worries. the database should work better now
- .
- top level:
- parents are now completely virtual! this means that when you add a tag parent, the tags that 'fill in' to make it show do not really exist in storage, only in a computed cache. if you decide to undo the parent, the implications are recalculated and the virtual tags disappear, with no permanent changes made. also, petitioning a parent will 'preview' the delete, just as siblings now does
- siblings and parents are now unified, and the logic is improved. all parents apply to all siblings, so no more worries about retro-active filling-in. the siblings and parents code is now basically 'nice'. this was a lot of quite complicated work, and it solves a number of lingering issues from the original prototypes I made several years ago. I will still do some smaller work and little fixes I am sure in the near future, but the 'big' siblings and parents work is done!
- like with the recent siblings change, the client no longer needs to do the 'loading parent tags' step when booting–everything is now handled at the db level
- like with the recent siblings change, you can now edit which services apply their parents to which service, now under _services->manage where siblings and parents apply_
- in the _manage tags_ dialog (and some other places), tags with parent implications now show a '(x parents)' after their label, much like the 'will display as' sibling suffix. I do not like this, but I ran out of time. I hope to add a more advanced actual listing of virtual tags with a nice 'ghostly' colour or similar in future
- right-clicking on a tag in a specific tag domain now shows a 'siblings and parents' submenu with detailed info on all known siblings and parents in that domain
- 'tag' menu entries are moved from the top 'services' menu to a new 'tags' menu. 'pending', when available, is also moved right
- the process of changing siblings or parents, or which services apply where, is no longer a CPU-laggy process! actual changes, however, may not appear immediately. a maintenance task now tracks what is currently applied and what is 'ideal', and slowly migrates to the ideal in the background in little chunks. in most situations, the changes are very quick, but if you are behind due to big recent changes, they may be delayed. you can manage when this maintenance runs and see the current status under _tags->siblings/parents sync_. this is an entirely new thing, so feedback on IRL work would be appreciated–there may be some kinds of siblings or parents that cause a whole bunch of annoying lag
- the PTR has a lot of non-virtual parents that were hard-added in older versions over the years. most are fine, but some are like the 'shadow'->'shadow the hedgehog' debacle. now the source of the problem is fundamentally solved, this problem will reduce over time. with luck, before the end of the year, no more will be added at all, and thanks to the janitors, the worst offenders should be chipped away
- during all this work, a bug with tag siblings and parents repository processing has been revealed (some users do not 'get' all siblings/parents for some reason). now the system is nice and undoable, this will be more easily addressed in coming weeks, with automatic retroactive fixes rolling out to all clients
- .
- boring details:
- like with siblings, wrote a parents structure object that constructs the parents tree without loops more simply and reliably. it populates a new parents quick-lookup table in the database, for which a full suite of lookup and maintenance methods are written
- parents and siblings virtual tag presentation is now unified into a single 'display' (i.e. vs 'storage') system with a more granular tag implication algebra (essentially 0-n rows of 'if A is in storage, show B in display' for every tag) that can calculate new and updated display tags and counts without having to do the expensive 'clear-and-regen' that 408-413 used
- wrote functions to quickly add or remove a display implication to the 'all known files' or specific file service tag display cache
- migrated all the combined and specific tag display cache update code (add/remove files, add/remove mappings, add/remove sibling/parent, add/remove sibling/parent application, and misc regen maintenance calls) to use the implication system instead of the sibling 'ideal' system (basically moving from 1->1 to 1->n)
- completely rewrote the complicated 'all known files' cache 'with tags' and 'with and without tags' lookup routines to use much less overhead in general and to use a single, albeit complicated, count-based query that carefully chooses whether to select the 'with tags' and 'without tags' portions using tags or files where available as the primary selector based on existing autocomplete count data
- replaced all usage of the old ui-side 'tag parents manager' object. as parents pop in virtually and do not need to be bundled intentionally to various content updates, this was mostly just clearing now-surplus code, but for instance in 'write' autocomplete searches, the parents that appear below search results are now generated at the db level on first search, rather than looked-up live in UI time
- the parents and siblings lookup tables are now split into two views: what the display cache currently holds, and what it ideally should hold. when adding new sibling or parent data, only the fast ideal table is changed
- a new complicated maintenance function now takes actual and ideal data for a particular unsynced tag, hashes out the implication changes needed to effect a migration, and performs it
- a new maintenance manager and accompanying db code now track and manage calls to migrate actual to ideal display presentation, and to update UI afterwards
- as tag display changes are now more frequent, I have made the routine that refreshes tag UI after sibling/parent changes more efficient. tag display now only refreshes for files that have the affected tags in a particular change
- wrote the UI panel and dialog to show and hurry up current sync status, and all the background hooks for that
- added 'tag parents lookup' entry to the database 'regenerate' menu. this routine and the 'siblings' variant are now very quick thanks to the new actual/ideal maintenance system
- updated my sibling unit tests to deal with the new actual->ideal syncing
- improved the speed of mappings cache updates when deleting files
- deleted all the old combined/specific 'regen chain' code and the sibling-based 'get sole/any tagged files' search code
- optimised and generally cleaned a bunch of the new cache code, particularly cutting out overhead for unusual/small situations
- fixed a counting bug with 'all known files' tag counts when rescinding pending tags
- fixed a bug in the siblings display code where deleting or pend-rescinding all of the multiple tags that have the same ideal sibling in the same transaction (e.g. if both A and B sibling to C, and a file has both A and B, and you remove them in one manage tags dialog apply) would not remove the current/pending ideal (issue #571)
- the 'add_siblings_and_parents' parameter on /add_tags/add_tags client api command is now obsolete! the help is updated to reflect this
- cleaned up just a bunch of db/ui/tag code mate
- .
- the rest:
- fixed an issue where long-running 'similar files' search was not cleaning its memory use properly as the job was going on, resulting in out-of-mem errors on very large clients (issue #669)
- thanks to user submission, rolling in a fix for the default pixiv tag search downloader
- cloudscraper updated to 1.2.48
- removed surplus executables from linux and macOS builds (win32 upnpc exe was causing anti-virus false positive on mac lmao)
next week
Back to small work. There are a thousand things to catch up on, so I'll mostly grind away at that. This work was big and difficult, and some IRL stuff hit me for the first couple of weeks, so the year is running out. My overall plan is still to have some network upgrades done and then a 'what to work on next' poll up for Christmas, but I will take a few weeks of simple work first.
Hydrus On The Go
I don't know about you guys, but wouldn't it be nice to have a mobile client for this shit? Or at least a mobile-friendly UI for the discerning user who is gonna buy a Librem 5 from Purism or roll their own portable device running on linux (or some kind of BSD?)
Paying someone to tag all of my files
I'm dead serious. $10 for 10 Gb worth of files (Price is non negotiable). Will send them to you via MEGA.
Post your discord ID here if you're interested.
I'll give you more instructions on how to tag them in DMs, like which tags to prioritize etc…
Version 413
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v413/Hydrus.Network.413.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v413.tar.gz
I had an ok week. I mostly fixed bugs to make a 'clean' release before I go for the tag parents database cache.
It is all misc work this week, no highlights.
full list
- added 'sort by number of files in collection' file sort type. it obviously only does anything interesting if you are collecting by something
- when you enter a tag from a manage tags suggested tags column with a double-click, the tag input box is now immediately focused. entering it with a keyboard action does not move the focus
- wrote a new routine for the 'check and repair' database menu that scans for and fixes invalid tags. this might be some system:tag that snuck in, superfluous unicode whitespace, or some weird website encoding that results in null characters, or any other old tag that has since become invalid. tag translations are written to the log
- added an experimental 'post_index' CONTEXT VARIABLE to subsidiary page parsers–whenever a non-vetoed post has pursuable URLs, this value is incremented by one. this is an attempt to generate a # 0,1,2,3 series. feedback on this would be appreciated, so I can formalise and document it
- added 'no_proxy' option for the options->connection page. it uses comma-separated host/domains, just like for curl or the NO_PROXY environment variable. it defaults to 127.0.0.1. in future, options will be added to auto-inherit proxy info from environment variables
- fixed an error when subscriptions try to publish to a page name when a 'page of pages' already has that name
- activated some old 'clean url' parsing tech I wrote but never plugged in that helps parsing urls from source fields on sites that start with non-url gubbins
- fixed the v411->v412 update step to account for a tags table that has duplicate entries (this shouldn't ever happen, but it seems some legacy bug or storage conversion indicent may have caused this for some users). if a unique constraint error is raised, the update step now gives a little message box and does dedupe work
- fixed an issue where the 'will display as' tag was rendering without namespace when 'hide namespace in normal views' was on
- fixed a recent character encoding routine that was supposed to filter out null characters
- fixed some UPnP error reporting
- _may_ have fixed an odd and seemingly rare 'paintevent' issue when expanding the popup toaster from collapsed state–it may also have been a qt bug, and fixed in the new qt:
- updated qt to 5.15.1 for windows and linux builds. it fixes a couple of odd issues like 'unclicking' to select a menu item (issue #296)
- added session save to holistic ui test suite
- misc code cleanup
- .
- client api:
- wrote a client test for the help menu so I can test some basic functions holistically, hoping to stop some recent typo bugs from happening again
- did a couple of hotfixes for v412 to deal with some client api url pending bugs. the links in the 412 release now point to new fixed builds
- fixed an issue setting additional tags via the client api when the respective service's tag import options are not set to get anything
- fixed a 500 error with /add_tags/add_tags when a tags parameter is an empty list
- fixed the /manage_pages/get_page_info client api help to show the 'page_info' key in the example response
next week
I am now going to write the tag parents database cache. This will basically be the same as I just did for tag siblings, making parents applicable cross-service and completely virtual. I will also improve how the application works, making it pause/resumable in non-interrupting parts in the background, like how repository tag processing works now.
I expect this to take two weeks, possibly three, so please do not expect a release next week. I'll make an update on Tuesday evening as normal.
64 bit
>"The hydrus releases are 64-bit only."
That was really bad news after my half-day reading about and planning a future, and how I will co-develop the hydros engine or a fork of it…. :-(
Why???….
The Python is a multiplatform and hi-level language (even a script language).
Why you all gaming maniacs make 64bit-only software?..
Of course, I will not change my tuned computer for one of 64bit-only apps. Several months of work to tune up Windows again, to test some app?.. No, thanks.
Ability to see all of my tags when adding new tags?
Ok so without this, the process of adding tags is kinda lengthy.
When adding new tags I want to be able to switch from "recent" tags, to seeing all of my tags. Doesn't seem like a difficult feature to add, and it would make adding new tags a lot easier since I keep forgetting the tags I have available.
The autocomplete feature does not compensate for this, as remembering the exact spelling isnt the problem, its remembering that the tag exists in the first place.
Version 412
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v412/Hydrus.Network.412.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v412.tar.gz
I had a great week catching up on smaller jobs, improving search speeds, and adding a 'lite' 407->408 update mode for HDD users who sync with the PTR. There are also a couple of new applications for the Client API.
Update this week will take a few seconds to a few minutes as new database indices are created.
sibling and search speeds
Thanks to feedback from some PTR-syncing HDD users, the new siblings update code, most importantly in step 407->408, takes way too long for them - perhaps more than 24 hours. I have written a little yes/no dialog popup into the update step that talks about this and optionally activates a 'lite' mode that does not apply siblings. This still requires some basic cache copying work, but it is significantly less. If you are still on 407 or before and have been waiting to update, please give this a go and let me know how it works for you.
The 'manage where tag siblings apply' dialog now has some red text to warn about the high CPU/HDD of applying many siblings to a large number of tags. I am still not happy with the 'monolithic' way this db work goes, so when I get stuck into the parents cache, I will write an asynchronous system that does this work in the background, pause/resumable, without interrupting browsing and so on, much like I did with repository processing.
Some things were working slow since siblings (e.g. in a search, mixing wildcard tags with regular tags), but I went through every instance of my new optimisation code, fixing bugs, testing it at large scale, and smoothing spikes out further. Tag, namespace, wildcard, tag presence/count, known url, and file note searches should all be more reasonable. A neat new tag search pre-optimisation routine that checks autocomplete counts for expected result size before deciding how to search now works for more sorts of tags and also kicks in for namespace and wildcard searches, which now break their work into smaller and simpler pieces. I also added and reshaped some database indices, which will ensure that more unusual search types and general operations can still run efficiently. The update will take a few seconds to a few minutes as tag indices are regenerated.
I have learned a bunch about speeding up multi-predicate searches recently - how to get it wrong and how to get it right. I have a plan to speed up rating and known url results, which are still generally not able to speed up with multiple predicates on large clients.
new client api applications
A user has been working hard at making a web browser for the client via the Client API, called Hydrus Web. It is now ready at https://github.com/floogulinc/hydrus-web ! If you have a bit of networking experience, please check it out - it allows you to browse your client on your phone!
Also Anime Boxes, a booru-browsing application, is adding Hydrus as an experimental browseable 'server' this week, also through the Client API. Check it out at https://www.animebox.es/ !
I also updated the Client API help to talk more about HTTPS and connections across the internet, here: https://hydrusnetwork.github.io/hydrus/help/client_api.html
full list
- client api:
- added Hydrus Web, https://github.com/floogulinc/hydrus-web, to the Client API page. It allows you to access your client from any web browser
- added Anime Boxes, https://www.animebox.es/, to the Client API page. This booru-browsing application can now browse hydrus!
- the /add_urls/add_url command's 'service_names_to_tags' parameter now correctly acts like 'additional' tags, and is no longer filtered by any tag import options that may apply. that old name still works, but the more specific synonym 'service_names_to_additional_tags' is now supported and recommended (issue #456)
- the /add_urls/add_url command now takes a 'filterable_tags' parameter, which will be merged with any parsed tags and will be filtered in the same per-service way according to the current tag import options.
- the client api help is updated to talk about this, and the client api version is now 14
- updated client api help to talk about http/https
- .
- the rest:
- the 407->408 update step now opens a yes/no dialog before it happens to talk about the big amount of CPU and HDD work coming up. it offers the previous 'full' version that takes all the work, and a 'lite' version that applies no siblings and is much cheaper. if you have been waiting on a PTR-syncing HDD client, this should let you update in significantly less time. there is still some copy work in lite mode, but it should not be such a killer
- the 'manage where tag siblings apply' dialog now has big red warning text talking about the current large CPU/HDD involved in very big changes
- a bunch of file-location loading and searching across the program has the opportunity to run very slightly faster, particularly on large systems. update will take a few seconds to make these new indices
- namespace and subtag tag searches and other cross-references now have the opportunity to run faster. update will take another couple of minutes to drop and remake new indices
- gave tag and wildcard search a complete pass, fixing and bettering my recent optimisations, and compressing the core tag search optimisation code to one location. thank you for the feedback everyone, and sorry for the recent trouble as we have migrated to the new sibling and optimisation systems
- gave untagged/has_tags/has_count searches a similar pass, mostly fixing up namespace filtering
- gave the new siblings code a similar pass, ensuring a couple of fetches always run the fast way
- gave url search and fetch code a similar pass, accounting better for domain cross-referencing and file cross-referencing
- fixed a typo bug when approving/denying repository file and mapping petitions
- fixed a bug when right-clicking a selection of multiple tags that shares a single subtag (e.g. 'samus aran' and 'character:samus aran')
- thanks to some nice examples of unusual videos that were reported as 1,000fps, I improved my fallback ffmpeg metadata parsing to deal with weird situations more cleverly. some ~1,000fps files now reparse correctly to sensible values, but some either really produce 1000 updates a second due to malformation or bad creation, or are just handled that way due to a bug in ffmpeg that we will have to wait for a fix for
- the hydrus jpeg mime type is now the correct image/jpeg, not image/jpg, thanks to users for noticing this (issue #646)
- searching for similar files now requires up to 10,000x less sqlite query initiation overhead for large queries. the replacement system has overhead of its own, but it should be faster overall
- improved error handling when a database cannot connect due to file system issues
- the edit subscription(s) panels should be better about disabling the ui while heavy jobs, like large subscription resets, are running
- the edit subscription(s) panels now do not allow an 'apply' if a big job is currently disabling the ui
- cancelling a manage subscriptions call when missing query logs were detected no longer causes a little error
- if a long-running asynchronous subscription job lasts beyond its parent's life, it now handles errors better
- .
- boring details:
- improved a pre-optimisation decision tool for tag search that consults the autocomplete cache for expected end counts in order to make a better decision. it now handles subtag searches and multiple namespace/subtag searches such as for wildcards
- wrote fast tag lookup tools for subtag and multiple namespace/subtag
- fixed some bad simple tag search optimisation code, which was doing things in the wrong order!
- optimised simple tag search optimisations when doing subtag searches
- polished simple tag search code a bit more
- added brief comments to all the new cross joins to reinforce their intention
- greatly simplified the multiple namespace/subtag search used by wildcards
- fixed and extended tag unit tests for blacklist, filterable, additional, service application, overwrite deleted filterable, and overwrite deleted additional
- added a unit test for tag whitelist
- extended the whole 'external tags' pipeline to discriminate between filterable and additional external tags, and cleaned up several parts of the related code
- moved the edit subscription panel asynchronous info fetch code to my new async job object
- cleaned up one last ugly 'fetch query log containers' async call in edit subscriptions panel
- moved the edit subscription(s) panels asynchronous log container code to my new async job object
- misc code cleanup
next week
More small jobs and other bug fixes. Nothing too huge, so I can have a 'clean' release before I go for the big parents cache in 414. I am starting to feel a bit ill, so there's a chance it will be a light week.
Version 411
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v411/Hydrus.Network.411.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v411.tar.gz
I had a good week. The siblings work is complete.
siblings
The last areas that used the old siblings systems are now migrated to the new fast cache. The 'loading tag siblings' step of client boot, which for PTR users could be several seconds of work, is no longer needed!
These last parts are all user-facing UI code. Mostly tag autocomplete behaviour and the 'will display as' labels you see in tag edit contexts. Everything now goes through the database, and all in a nice asynchronous way, so (fingers crossed) it will not add noticeable lag.
As I unified code here and moved to nicer logic, I was able to clean up some behaviour. Matching siblings based on what you type (for instance, if you have 'lotr' siblinged to 'series:lord of the rings', text-matching the latter even if you type the former) should be cleaner and more reliable. Also the 'will display as' labels should appear in all the tag suggestion columns, and some other 'edit tag' places.
There is still more I could do here. Using 'manage tags' on thousands of files at once can be a bit laggy, both in working and once you hit apply, and I could make '(will display as xxx)' labels colour according to the xxx namespace to make them 'pop' a bit, and I would still like to ultimately replace the individual 'will display as' labels with something neater for workflow. But I have been focusing on siblings for more than a month. Now the main virtualisation and boot lag elimination objectives are done, I should move on to other work.
If you have been waiting to update from 407, this is a fine week to do it. There will still be a whack of update CPU if you sync to the PTR, but on an SSD it shouldn't be more than a one-time shot of 10-30 minutes.
full list
- misc:
- fixed the 'system:(like/dislike) rating = x' search predicate string, which was saying 'unknown' rather than 'like/dislike' in several cases
- fixed a 'current_count' error in the new file search optimisation code for tag searches where the tag did not exist for any files in the domain (i.e. autocomplete count=0). thank you to users for helpful reports here
- fixed the recent file search optimisation code to handle 'system:time imported' when it was mixed with tags or search predicates that would pre-populate the query file pool with file domain cross-referenced files. sorry for the trouble!
- the forced delay overhead for table analysis is reduced from 0.1s to 0.02s. whenever many mostly empty tables need to be analyzed (like on first boot shutdown, when it is usually 100+ tables), it now zips by
- .
- siblings/tag improvements:
- typing a shorthand sibling like 'lotr' into an 'all known tags' 'read' autocomplete - like on a default search page - now reliably discovers and matches text entry to ideal sibling results like 'series:lord of the rings'. this was previously buggy and unreliable–it now allows the match using better db knowledge, even when the merged 'all known tags' services involved disagree on siblings
- when typing tags into a 'searching immediately' page that has media, the autocomplete count results that only refer to that media will now match shorthand sibling inputs to the ideal result. media-populated tag search now takes a little bump of extra CPU to fetch results (they are now passed through the db to get nice siblings info), so it is also cached for the duration of your typing (previously, the counts were re-computed on every new keystroke, so this should be significantly smoother to work with on large pages even if that first keystroke takes a moment to give results)
- when typing into a 'write' autocomplete, like in manage tags, the process that promotes the entry text and known siblings to that entry and a potential ideal sibling result to the top of the list is now more sane. it now also only uses results with nonzero count. we'll see how this last change works out IRL
- when typing into a 'read' or 'write' autocomplete, the pre-search tag insert no longer has sibling insertion/swapping. it was unreliable before, with weird sibling-swapping in the short moment before real results returned. if you have slow results and often quick-add tags into search pages or manage tags, let me know how this works for you
- the 'additional tags' tag input dialog off the tag import options edit panel now shows the 'will display as' label
- the 'favourite', 'file lookup', and 'recent' tag suggestion panels now show the 'will display as' label
- the 'related' suggestion panel, which works on a slightly different system, now shows the 'will display as' label
- the 'tag suggestions' options panel's 'favourite tags' edit lookup and list now displays 'will display as' labels and correctly finds service-specific siblings in its results (e.g. you type 'lotr', it also finds 'series:lord of the rings')
- all autocomplete tag filtering should be just that little bit faster as you type
- filtering cached autocomplete results based on subsequent search text is now faster
- autocomplete inputs should no longer return 'ghost' results that have no current/pending count when one of the 'include current/pending' buttons is deactivated
- the new database autocomplete predicate generation routine now checks for 'cancel search' signals, saving CPU time as you type
- the slow 'regen chains' maintenance tasks now process sibling chains in random order, smoothing out the 500/100,000 progress label, which previously took about 80% of time on the first 20% of ids due to IRL tag distribution
- .
- the last UI-side siblings work is cleared:
- the UI-side tag siblings cache is no longer used. the sometimes multi-second 'loading tag siblings' step of boot no longer happens!
- media autocomplete fetches are now asynchronously populated with siblings data via the db
- the exact-match and sibling 'insert' predicates at the top of pre-load and post-load read and write autocompletes now rely exclusively on db data for sibling matching
- taglists now present 'will display as' labels asynchronously and are better about updating those labels when the list's underlying tag service changes
- the taglist right-click menu that shows siblings to copy now fetches that submenu's contents asynchronously from the database
- the test panel on a blacklisting tag filter now asynchronously fetches tag siblings to test against from the database
- the actual blacklist tag filter test now fetches tag siblings to test against from the database
- reworked my custom tag listbox to handle asynchronous text decoration, and unified sibling decoration for media taglists and string taglists
- updated my old async updater class to be more flexible for different job types, and cleaned the code that already used it
- wrote a simple class for one-shot async jobs
- wrote a simple db lookup for UI-side tag sibling chain members
- wrote a simple db lookup for UI-side tag ideal siblings
- bunch of misc sibling, db, and ui work and cleanup to make all this work
next week
I was going to go straight onto the parents database cache, which is basically a smaller repeat of this siblings work, but I think I will insert a couple of weeks of smaller work first. I'll actually catch up on github issues and also stop myself from going crazy.
I have two more big jobs for the year: this upcoming parents cache, and a network update iteration to improve client-repository communication and control. I'd like to have a new 'big job' poll up for Christmas.
Version 410
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v410/Hydrus.Network.410.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v410.tar.gz
I had an ok week. I wasn't as productive as I hoped, but I am happy with the mostly optimisation work.
optimisations
After some more profiling in IRL situations, and with more helpful info from users, I have done another round of profiling for the new sibling cache, and more besides. A database technique I use for many purposes is now more reliable (fewer lag spikes), and has less CPU overhead. If you found some systems (like the 'related tags' suggestions in manage tags dialog) sometimes took a few seconds to work in the past couple of weeks, they should now be fast again. And you should find many types of file search, particularly those with multiple search predicates, and general tag processing, should be faster than before.
My dev machine went from about 3-8k rows/s processing speed in a test environment up to 8-20k rows/s, which is faster than it was before the siblings cache was added.
full list
- general work:
- fixed a bug in the new file service filtering code that was stopping file upload commands to file repositories or ipfs services from sticking
- fixed an issue with the export files dialog auto-close-when-done function
- I think I fixed a possible bug in the boot file location repair/recovery dialog sometimes not saving corrected paths on unusual file systems
- file migration cancel button and shut off timer should work a bit more reliably, more to come here
- copying subscription quality csv info to clipboard no longer does nice human numbers (you now get 1234, not csv-breaking 1,234)!
- may have fixed a very rare 'or predicate' error when opening a dialog with a 'read' autocomplete input, like export folder or file maintenance jobs dialogs
- all pages are better about dealing with missing (i.e. recently deleted) services on load, and autocompletes also
- error handling from servers with strange character encodings should be better about dealing with null characters
- cleaned up the combined display regen chain code
- deleted some obselete db code
- .
- optimisation review:
- after more profiling, and thanks to additional input from users, I have done another round of optimisation for the new caches. using a new technique, more than just mappings are sped up this week - a number of queries that were prone to lag spikes should now have much more reliable speed and also be faster when hammered often
- .
- join and analyse db optimisations:
- these are mostly forcing table join orders, which reduces lag spikes, and reducing some related pre-query analysis overhead, which speeds things up more the faster your drive is (up to double processing speed on an ssd). they will affect different clients to different extents, but if your 'related tags' were taking more than a second to load, it should be sorted this week. systems affected:
- archiving files
- fetching 'related' suggested tags
- tag siblings regen/update in about ten places
- all mappings processing
- additional mappings processing for add/delete, pend/rescind_pend
- importing or deleting files that have tags
- loading medias' tags for the first time or on regen
- loading any media for the first time
- num notes searches
- similar files search tree maintenance
- many general file hash lookups
- many general tag lookups
- .
- other optimisations:
- mappings processing
- sibling processing
- wildcard tag searches, with and without namespaces, particularly when mixed with other search terms
- 'tag as number' searches, with and without namespaces, particularly when mixed with other search terms
- searching for tags when mixed with other search terms
- has notes/no notes
- searching files on 'all known files' with general file metadata system predicates (like size, filetype)
- url class, url domain, and url regex file searches, particularly when mixed with other search terms
- num tag file searches when mixed with other search terms
- has/not has tags file searches when mixed with other search terms
- sped up specific display chain regen significantly, with similar separate current/pending optimisations as last week's for combined
- converted specific display cache overall regen to use a copy followed by the new chain regen rather than additive file import
- sped up combined display chain regen a little bit
- the splash window now updates itself with less UI overhead, so spammy updates (like the new tag regen code) use a little less CPU and fewer UI context switches
next week
I had some IRL fall on my head this week. There's a bit more next week. I still have many small jobs and github to catch up on, and also some final tag siblings lookups to migrate to the new system to eliminate the 'loading tag siblings' step on boot. I'll just keep pushing. I'm scheduled to start the parents cache the week after, so it would be great to have all the siblings changes squared away.
Version 409
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v409/Hydrus.Network.409.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v409.tar.gz
I had a great week fixing some bugs and optimising the new tag siblings cache. The new code works much faster now.
siblings
I am very happy that there do not seem to have been any obvious errors with the new sibling database cache. Unfortunately, a couple of areas were working inefficiently, which IRL testing helped to diagnose. I put a lot of time into this this week and was very successful - some sections take 10% less time, some 90%, and one critical query now takes 99% less time. It depends on many factors, but many things are faster overall. In particular, tag processing speed, which took a real hit, is back up to good speed, and setting new tag sibling application rules now only needs to regenerate for changed siblings, so if you add (or remove) your own five 'my tags' siblings onto the PTR, the client now only has to do two seconds of work, not ten minutes.
I made some progress on the final awkward things to migrate. Most autocomplete results you see are now able to give themselves the 'will display as xxx' label when needed and match against sibling input (e.g. having an input of 'lotr' match 'series:lord of the rings' due to siblings) on their own, which should save some CPU time when typing. There is still more to do, so I'll keep hammering at it for the next two weeks and see if I can get rid of 'loading tag siblings' on boot before I start on parents db cache.
If you have been waiting for faster code before you update, you might want to wait another week. I just did a test re-do of the 407->408 update step in IRL conditions, and it was not as fast as I wanted it. I'll keep pushing at this.
I am increasingly looking forward to doing that parents db cache, which will extend this work in a new dimension. That will be v412, which I am very confident will be another two-week release. This is some of the most intricate work I have done.
full list
- siblings:
- the slowest of the new sibling regen & update code has received a full optimisation pass. some sections take 10% less time, some 90%, and one critical query takes 99% less time. overall, several big jobs work much faster, and ptr processing, which slowed significantly for many users, should be back up to a good speed. uploading pending tags (which tend to be for local files) should be much faster in particular. let's do another round of IRL observation and profiling this week, and I'll keep at it
- the various 'display' regeneration routines now provide more progress status text, drilling down to the x/y siblings being collapse-counted, or number of files added to a cache, and generally all tag sibling regen got a status update polish pass
- optimised the way tag sibling application is set–now, only the tag siblings that are changed need to have their counts regenerated. hence, if you just apply (or remove) your own five 'my tags' siblings onto the PTR, the client now only has to do two seconds of work, not ten minutes
- .
- the rest:
- fixed the annoying issue with media viewer mouseovers stealing focus/activation from the manage tags dialog. this can now only happen if current focus is on a hover window. sorry for the delay!
- updated manage tag parents dialog to state the pairs being petitioned on the 'petition reason entry' dialog
- updated manage tag parents and siblings dialogs to have appropriate 'reason' suggestions for petitions (previously, they were inheriting the same suggestions as for add)
- ipfs network jobs now have a minimum 'reply' connection timeout of two hours (so giganto directory pushes won't throw an error). connection timeout remains the same, so if the server is hanging on that, it'll still notice
- fixed the 'test address' button on the IPFS manage services panel
- petitioning an IPFS file when there is no IPFS multihash entry in the db no longer causes an error. now, in this case, the file entry is removed with no change made.
- when pending to or petitioning from a file service, a quick filter is now applied to discard invalid files (i.e. (not) already in the service). any weird logical holes where this might occur should now be fixed
- export folders now catch and report missing file errors more nicely
- export folders now remember the last error they encountered and report that in the edit export folders dialog
- .
- boring tag siblings optimisations:
- optimised the tag manager generation routine to use any common file domains for fast cache lookup for any subset of the files available, rather than falling back to 'all known files' domain when there is no single common file domain
- optimised the new 'all known files' display autocomplete cache to use similar faster specific files cache lookups when available
- optimised how the 'all known files' display cache regenerates tag sibling chains. it now takes a shortcut when given non-sibling tags and tags where all but one sibling member have zero count, and it can count current and pending counts separately according to the most efficient counting method (e.g. most pre-display pending counts are 0 across the board, so even if current count is a million, the pending count can often be assumed without lookup overhead). furthermore, the 'clever' count has better query planning and less non-sqlite data overhead, and with experimental data is now chosen more carefully. what was previously a 22s job on a test database now takes 5s
- deduplicated how new mappings are filtered to all the specific cache domains, significantly reducing overhead
- massively optimised a critical - and the slowest - part of the new 'combined' cache that handles add/pend mappings pre-insert presence testing, speeding up the core query about 100x!
- reduced some overhead when doing file service_id normalisation in repository processing
- split up specific chain regen into groups to reduce memory usage
- optimised specific display tag cache 'add file' updates, and thereby basic cache regeneration, to be just a little faster for files that have multiple sibling tags
- all predicates made in the database are now populated with ideal and chain sibling information, and this is used for '(will display as xxx)' labels and autocomplete tag search filtering (e.g. you type in 'lotr', it matches an autocomplete result of 'lord of the rings'). there are still some ui-made predicates to figure out, so the old system remains as a fallback
- related tags lookup is a tiny bit faster and now populates its predicates with ideal and chain sibling info at the db level
- cleaned up some 'fetch related tags' code, might make it a bit faster for large tag counts
- cleaned up the way some mapping tables are fetched
- unified table/table_name nomenclature in the db code
- updated an old data->ui status presentation method (it typically does stuff like "regenning some stuff: 500/10,000"), to not hog so much UI time and not yield worker threads so often when new statuses are coming in real fast
- several late optimisations based on IRL testing
next week
Next week would normally be 'cleanup', but all the optimisation I did here kind of counts as that, so I'll make sure to do some small jobs, just so I am not neglecting other things. Github issues and other non-sibling work.
Version 407
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v407/Hydrus.Network.407.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v407.tar.gz
I had an ok week. I did some prep work for siblings, and fixed some bugs.
misc
The right-click menu off a normal taglist now provides a whole bunch of different copy options, depending on whether you want all or selected, tags or subtags, and with or without counts.
I fixed a bunch of weird hover window issues in the media viewer. On a thin media viewer, the top-right window now squeezes into the smaller space better, and if it can't fit, it bumps down so you can still use it. The center-right duplicates window also positions correct on a thin window. And now moving your mouse off a clicked hover window will immediately restore focus to the main media window, which should eliminate some annoying click focus issues (usually when you click quickly after moving your mouse off, when the click can be swallowed).
A user has written a simpler getting started guide here: https://github.com/Zweibach/text/blob/master/Hydrus/Hydrus%20Help%20Docs/00_tableOfContents.md This is now linked at the top of my help as well. If you have had trouble with the denser official help, or you know someone else interested in hydrus who is similar, please give that a go.
siblings
This is about a somewhat advanced change to how siblings and parents will work at the database level. I'm doing big work behind the scenes soon, but nothing huge will change at the front end, feel free to skip.
v408 will introduce a database level cache of your current tags as they appear with siblings. This is a long-planned and large job that will improve accuracy, speed up a bunch of siblings stuff, and enable some neat features like namespace renaming. I did various prep work for this this week, but unfortunately it does not do much visible yet.
I also prepared some new options, but the back-end isn't ready to accept them yet. If you want to check them out, they are under services->manage where siblings apply. Instead of the old 'siblings only apply to their own service' vs 'siblings apply everywhere' option, we'll be moving to a neater per-service system, where you can say things like 'apply "my tags" and "PTR" siblings to "my tags"' or 'do not apply any siblings to service x'. The same will be true of parents, but I'll get siblings working first.
Also as part of this update, I have decided to make siblings and parents completely 'virtual', which means if a tag is changed by a sibling or added by a parent, it will appear as normal in any UI view, but it will not actually exist on the main store. Such tags will not be shared on the PTR, for instance, which means you can set up whatever situation you like, and it won't affect anyone else. It also means they will be completely undoable, so if a parent or sibling is wrong or reassigned, or if you want to stop or change which siblings apply to your view of the PTR, the virtual tags can just be recalculated from the unchanged main store. I know hard-replace actions are important to some users, so I will re-introduce this later under a modified or entirely different system–trying to support both virtual and non-virtual in the same pool has proved too messy to manage at large scale.
full list
- sibling prep:
- I am preparing for a new siblings database cache for v408. this will ultimately make siblings (and parents) faster, more accurate, more powerful, and simple to undo. I have decided, as part of it, to make siblings and parents completely virtual (i.e. the tags won't exist for real, they'll be implied). better tools to manage hard-replace siblings and parents will come later, as trying to support both situations at once has not been excellent
- .
- created options to hold per-service sibling and parent preferences, so you'll be able to set up '"my tags" siblings and then "ptr" siblings apply to "my tags"' or 'no parents apply to this service'
- wrote UI for the sibling options under 'services->manage where tag siblings apply'. you can play with it if you like, and it saves values, but it is not plugged in yet and makes no changes
- siblings logic is a little tighter. the db and gui side of siblings structure calculation is more unified, petitioned siblings are discounted properly on all generation, and the db side now resolves conflict decisions the same on every regen. the gui-side still runs on an older structure, but will be updated to exactly mirror the db
- updated and unified how large numbers of raw tag siblings are fetched in the database. it also supports fast tag slicing, speeding up sibling cache maintenance. the siblings lookup cache now uses this method for regeneration and update calls
- .
- the rest:
- tag right-click menu copying now supports all combinations of selected/all, tags/subtags, and no_count/with_counts where appropriate (issue #325)
- if the media viewer is too thin for the top hover window to fit into its space, the top-right hover now drops down below it. I don't really like how this looks, and will probably instead figure out a flow layout so the toolbar buttons always fit, but at least they are now accessible (issue #388)
- altered the above fix–if the top-right hover window can be shrunk to fit in the available space, it will now squeeze in, only bumping down if it can't
- moving the mouse off an activated (e.g. clicked) hover window now instantly activates the main canvas. this should fix up some fast swallowed clicks and annoying click-to-activate issues with the center-right duplicates hover window, which does not hide (issue #384)
- the duplicates hover window now positions correctly if its min size is too wide to fit in a thin media window
- if you make changes to a parser or content parser, there is now a yes/no confirmation when trying to cancel the dialog
- fixed an issue where 'queue' listboxes with no edit button would throw an error on double-click. now double-click in this case deletes
- fixed a couple of timestamp convertions that were doing YYYY/MM/DD instead of the more ISO-nice YYYY-MM-DD. also, when in UTC, they'll correctly say UTC now instead of GMT (issue #369)
- fixed some borked centered text layout on ratings dialog and import folder dialog
- fixed the manage services dialog's wrong headers for type/name columns
- added links in the official help to the new user-written simple help guide at https://github.com/Zweibach/text/blob/master/Hydrus/Hydrus%20Help%20Docs/00_tableOfContents.md
- moved object tag and ratings code to a new client module, 'metadata', and pulled various ratings gui code into a new separate file
- refactored some more manager code around to generally more sensible locations
- did a bit more work chasing down the highlight-downloader ui deadlock, which unfortunately still exists
- reduced the number of db hits some paged downloaders need, particularly on highlight and init
- updated some test code to support cleverer db testing
- updated mpv for windows build. api version is now 1.109. this fixes at least one weird linux vm audio driver issue
next week
So, I will try to get this cache working. I need to essentially mirror the existing tag and autocomplete caches, but after siblings collapse, ensure that is maintained through tag or sibling changes, and then plug it in to everything. There is a bunch of ancient code to update around here, and other auxiliary system to improve, so this may end up being a two week job. I'll update on Tuesday night as normal.
Version 408
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v408/Hydrus.Network.408.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v408.tar.gz
I had a great couple of weeks moving tag siblings forward. The update this week will take some time as a new cache is generated. On an SSD with the PTR synced, about 10 to 20 minutes, depending on how many files you have. Some of the new tag sibling application rules may have 5-10 minute delays on edit as well. If you have a large/PTR client and are on an HDD, or you would rather just wait for the kinks to be ironed out, you might like to wait a week or two for me to further optimise this new code.
siblings
tl;dr: Siblings are faster and better now, you don't have to do anything. Some parents will not appear with new downloads - don't worry about it, they will all fill back in nicely soon.
I have finished the first version of the database 'display tags' cache. This is a long-planned system that keeps track of how tags are 'collapsed' by siblings on an ongoing basis. Until now, siblings were always calculated on the fly at the UI level whenever media was loaded, which lead to slow and inaccurate sibling-based search results and a variety of logical problems. Search results generally load much faster now, particularly on subsequent loads.
On the front end, tags appear exactly they did before, but when you do tag-based searches or tag autocomplete, the files and counts that come back should be more accurate. For instance, if you have a file tagged 'metroid' that appears as 'series:metroid', and you search 'series:anything', that file now shows. Siblings logic is also tighter, and changes apply faster.
And now, if you go to services->manage where tag siblings apply, you can now decide which services' siblings apply where. If you do not like the PTR's siblings, or you have a couple of different preferences you want to add on top, you can say things like 'apply siblings from "my tags" then "PTR"', or you can turn them off entirely. This powerful service-based system replaces the old 'apply all siblings to all services?' checkbox that used to be in the options.
This is some complicated code affecting core systems. I have tested it extensively, but IRL situations, particularly with large sessions or storage, will be more complicated and useful than what I have here. Please let me know where this new system is laggy or miscounts anything. And of course, if anything outright breaks, let me know!
The help on siblings is going to be increasingly outdated as this work continues. If you don't know what siblings and parents are, please wait until I am done with this multi-month job, and I'll have some nice new help written.
sibling and parent virtualisation
On the back end, the important change is that siblings are now virtual, which means that when the tag 'samus_aran' comes in, while it may appear as 'character:samus aran' in all normal views, the old systems that would substitute the 'good' tag in for the bad at the storage level (i.e. what you see in manage tags) no longer occur. This means that all siblings going forward are completely undoable–if you want to try something different out, you can just change the current siblings or application rules and it will all recalculate from the untouched (albeit uglier) source.
This 'virtual' change is needed now. Siblings can come from anywhere (particularly with the new service application rules), and trying to manage a mix of 'soft' and 'hard' siblings has proved too difficult to manage in one system. Parents are not yet virtual, but they will be soon - I hope in four or five weeks to do the same for them as I have done here: virtualisation, improving search and application logic, permitting complete undo, and allowing service-based parent application.
We are therefore in an interim period. Parents are still hard-added, so I have disabled sibling-based parent application (so if 'character:samus aran' has 'series:metroid' as a parent, a file that only gets 'samus_aran' will not get that parent). This unfortunately means that workflows for the next month may be a little awkward, as parents will not fill in for certain downloads. Please hang in there - I am still working on this, and when parents become virtual, all the missing parents, old and new, will be filled in (and finally undoable!). I want to improve some of the autocomplete sibling-add workflow, something like grouping siblings together, a bit like how parents hang off a tag that will add them. Let me know what you think.
I know there are uses for 'hard' sibling and parent rules that will actually change and add to storage tags. Trying to support both in the same system has proved unworkable, so I will bring hard-replace and -implication in as separate systems in future.
full list
- tag siblings cache:
- tl;dr: siblings are faster and better now, you don't have to do anything. some parents will not appear with new downloads - don't worry about it, they will all fill back in nicely soon
- wrote the first version of a 'tag display' cache, which stores not your tags as they are, but how they appear after display rules such as siblings, parents, and filtering are applied, meaning this data need not be calculated every time on thumbnail load. this week marks the first concrete step forward in an improvement of siblings and parents storage, and begins with just siblings. all siblings and front-end tags work should be generally faster and more accurate
- part one is for tag domains cross-referenced with file domains. it maintains virtual sibling-collapsed mappings and autocomplete counts through mappings added, deleted, pended or pend-rescinded, files added/deleted, and siblings added/removed
- part two is for tag domains on the 'all known files' domain (i.e. no file domain). it maintains virtual sibling-collapsed autocomplete counts through mappings added, deleted, pended or pend-rescinded, and siblings added/removed
- both parts also support full table drop/regen (under the new database->regenerate->tag display mappings cache) for when my logic inevitably miscounts something. the existing regen 'tag mappings cache'/'tag siblings lookup' commands also regen the display mappings cache, since it relies on them
- when tag siblings on a repository are petitioned to be deleted, they are now instantly discounted from tag sibling application (previously, they had to be uploaded and committed to count, now both pending and petitioned offer a quick preview of outcome)
- the display cache supports the tag sibling service application rules under the 'services menu', and regen when that changes, so you can now turn siblings on and off, and apply them across services. as a result, the old 'apply all siblings to all services' option is now gone! as parents will undergo a similar change soon, and the siblings changes this week may lead to some undesired parents in the interim, the 'apply all parents to all services' option is also gone
- tag autocomplete counts in the form (x-y) due to siblings are eliminated. it will still do it when combining the same merged tag across different services, or when an unnamespaced tag includes how many potential namespaced will also be found
- the following search types now obey tag sibling application rules accurately: number of tags search, namespace:anything search, wildcard search, tag search (on a per-tag-domain basis, previously it was globally hacked to all siblings), tag-as-number search. for instance, if you search series:anything, a file that has 'metroid' tag-siblinged to 'series:metroid' will now correctly appear.
- the above search types are now exact to how the tag displays. if you have for files that are tagged 'samus' on either tag service A or B, and service B has a sibling for that to 'character:samus aran', searching for 'samus' gets the results in A, 'character:samus aran' gets the results in B. previously it was an expensive logical mish-mash of 'sure, try and get everything behind the scenes'. now it searches what you see
- searching for files in the advanced 'all known files' domain currently has no sibling support for the above search types. autocomplete counts should be good, and the results that come up should have the correct tag display, but the actual results are calculated based on storage tags. getting this to work without doubling the size of the db is tricky, so it will have to be ongoing work
- all tag siblings are now completely virtual. this means that when a tag comes in via a downloader or other means, it will not be automatically coerced to its ideal sibling in the actual db storage tables (the true tags you see in manage tags dialog), but remain as it is. there is no change in sibling appearance in normal operationit still _displays_ and searches as its ideal sibling. the same will happen to parents in the coming months, and in the interim period, parents no longer apply across siblings. as siblings can come and go from anywhere, they are now divorced from actual stored tag mappings. in a similar way, the manage tags dialog no longer supports the 'hard-replace siblings and parents' command, nor the 'auto-replace with sibling' command. this may be jarring to workflows and preferences, so please bear with me and let me know what feels particularly bad. and please don't worry too much about parents not always being added in the meantimeI hope to do the same transition for them in four of five weeks, and all gaps will be filled in. also, in the coming weeks, I expect to improve manual tagging workflow by indent-grouping edit-view siblings together (ditching the old 'will display as' text) for easier review and selection, a bit like parents. actual 'hard' siblings and parents that do always get irreversibly renamed/added in storage will come in the future as a separate system
- the 'add_siblings_and_parents' client api parameter no longer adds siblings, and soon will be retired completely
- I had wanted to completely eliminate the old UI level siblings manager this week, but there are still some systems, mostly tag autocomplete work, that need it and are tricky to swap. I stripped it down, at least, and reduced its update delay to 2 seconds. therefore, the 'loading tag siblings' step of boot still occurs, albiet a _little_ faster. I hope to have it gone soon
- this is some complicated code affecting core systems. almost everything 'siblings' is now different. there are likely to be laggy parts, awkward new workflow, and possibly some update or miscount bugs as I iron it out. the good news, now they are all virtual, is that problems are undoable. please report any issues, and I will work on it as I keep pushing on this and on parents
- please expect your client.caches.db file to expand in size about 10-30% or so this week. the update itself will take a few minutes as the improved tag lookups and new caches are regenerated from empty
- .
- boring mostly db optimisation list:
- after some thought, moved those new options for tag sibling application down to the db. previously, they were stored in an UI object for convenience, but since everything is going down to the db, it is worth doing it properly down there. thus they reset this week to the default
- I also removed that complicated 'all known tags' page in the tag sibling application options–it wasn't doing enough to justify itself
- tag siblings lookup cache now obeys the tag sibling application rules and regenerates the appropriate cache when the options change
- tightened up the db tag siblings lookup cache and wrote more tools for it. it had a couple of blind spots for getting all siblings in a chain. also optimised the lookup for en-masse tag operations
- tightened up my tag sibling structure builder object, which was not eliminating loops but collapsing them to (generally harmless, but not desireable) (A,A) pairs
- extended mappings and siblings lookup caches to perform various sorts of tag sibling collapse filtering, to determine files that do or do not have another tag mapping on a tag sibling chain
- optimised the existing mappings cache in several ways
- optimised cross-domain file cache mappings filtering, and cleaned the code
- optimised autocomplete count fetching from the mappings cache, particularly for large result sets
- optimised how the combined autocomplete count generates from nothing
- optimised how tags are loaded for search results (thumbnails)
- optimesed basic tag search
- greatly optimised how the mappings caches request cross-domain file cache filtering
- broke up the rescind_pending/add mappings job into simpler separate parts, which was needed for accurate display cache counting. this may end up fixing the other weird pending miscount bug we had
- the cached 'display' tags are now loaded with regular media results, not generated on the fly on first request (unless in the advanced 'all known files' domain, where it is done quickly on first load at the db level)
- converted the db over to using its local sibling lookup cache for all sibling jobs
- all data-level content updates to media result objects now occur in the database loop, reducing lag and eliminating a single UI event loop gap when the objects the UI relies on were desynchronised
- optimised how the tag and hash id-to-definition cache maintains itself
- cleaned up cache code generally
- wrote a ton of unit tests to cover construction, tag, and tag sibling operations on the siblings and display caches
- wrote a second optimised method for regenerating 'all known files' display cache autocomplete counts from nothing, which, when multiple siblings have wildly different counts (e.g. 50, 100, 200000), instead of counting them all, counts the smaller tags sans the largest, and adds this to the already pre-computed largest count
- the old ui level siblings manager has been pared down to some final tools that will be trickier to replace
- .
- the rest:
- fixed the stupid manage tag siblings dialog input/ok bug I introduced last week
- fixed the pair preview label in manage tag siblings dialog when it asks to enter a reason for a remove petition
- I believe I fixed the annoying recent bug where the top-right hover window would sometimes not position itself correctly on a window size/move until the top hover was shown once
- fixed a bug where the 'do you want to do shutdown work?' dialog was not abandoning shutdown if cancelled (rather than yes/no)
- updated the 'has free space to do db transaction?' checker, which needs to test device partitions, to do two sweeps–first only fast local devices, then potentially mega laggy network discovery if the mount point is not found (hydev was wondering why it was suddenly taking nine seconds to close his test client!)
- fixed another issue with double-clicking some addremove/queue listboxes when no edit button is set–now in this case they all delete on a double-click
- fixed a little bad error handling on pending content upload. an error with petitioning certain IPFS uploads is not yet fixed
next week
I was unable to completely replace the UI-side siblings manager. It is a bit smaller and faster, but you will still get the slow 'loading tag siblings' on boot. There are about four or five things it still does that I still need to move down to the new cache, so I will keep hammering away. I'll also improve the speed of these new caches' regen. Other than that, I have two weeks of other jobs to catch up on, so just some misc work.
I just realised I forgot to fix the annoying manage tags/media viewer focus stealing! I'll fix it for 409!
hydrus+hyve locally hosted booru
I would like to set up a locally hosted booru for my (rather large) image collection. Having used hydrus before, I would like to use it in conjunction with hyve (https://github.com/imtbl/hyve) as a Web UI accessible by many devices, instead of relying on a danbooru instance, for example.
Some things that I'm thinking about are:
- I mostly use Docker images on a Debian-based headless system. hyve provides a Docker image for hydrus server to use so I think my plan will be to use the server version as host, use the client to manage the database, and use hyve's web UI to access the database on the Android devices which don't have the client.
- I'm not very knowledgeable so I'm mostly relying on using others prebuilt images.
- I'll probably want to test this out on a smaller imageset first as I'm well aware of how long it will take to set up a database of 50,000+ images.
I posted this here because I'm curious:
1. Does anyone else have a similar set-up to this and if they do, do they have any comments on how it's working?
2. Is there a better solution than hyve that would suit my purposes?
3. Would it make sense to find (or try to build) an up-to-date Docker image of the client to run on the host?
Thanks in advance for any replies, even if you just want to call me a fag.
Version 406
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v406/Hydrus.Network.406.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v406.tar.gz
I had a good week. It is now easy to deduplicate queries in subscriptions.
subscriptions
Managing multiple subscriptions can get complicated. Figuring out which queries are in which sub, whether and where you have duplicates, is tedious. A recent bug with query pasting also could have introduced some duplicates within the same sub. This week should make it simple to manage.
The main manage subscriptions dialog now has a 'deduplicate' button. It lights up any time your subs of a particular downloader have the same query, whether that includes dupes within the same sub or across different subs. Click it, and it launches a thorough process to determine what you want to do:
- First, it asks if you want to deduplicate by case or not–e.g. whether "Samus_Aran" and "samus_aran" should be considered dupes, or only exact matches.
- Then it summarises which downloaders appear to have dupes, letting you choose which to work on.
- Then you have the option of selecting which of the queries you wish to deduplicate–e.g. if you want to do two runs to dedupe, say, artists to one sub and characters to another.
- Then you select which subscription(s) will retain the deduplicated queries.
Every step explains itself. Mostly it is simple, but there are some odd situations. When a sub dedupes queries within itself, it keeps the one(s) with the most files. I hope I have accounted for everything reasonable.
Adding to caseless matching, also added is a 'lowercase' button, which will coerce all the queries in the selected subscriptions to lowercase, just to help keep things neat. And lastly, when you paste queries into a sub, the 'already in the sub' test is now caseless.
full list
- subscription management:
- the manage subscriptions dialog now has a 'deduplicate' button. it is enabled whenever your subs of a particular downloader contain duplicate queries. it launches a semi-bananas but thorough 4-step process to ask if you want to do upper/lower-case deduplication, then which downloader, then which queries, then which master sub(s) to retain the queries
- subscription dedupe within the same sub keeps the query with the most files
- the manage subscriptions dialog also now has a 'lowercase' button that coerces all queries of the selected subs to lowercase
- when pasting a list of queries into a subscription, the 'already in sub' test is now caseless. pasting "Samus_Aran" into a sub already with "samus_aran" will not add anything
- .
- misc:
- url classes now have a checkbox to keep fragment data (e.g. "#kwGFb3xhA3k8B") during URL normalisation. this data is not sent to the server and is not useful for almost all sites, but for sites like Mega, it contains useful clientside javascript navigation or access info if you open the URL in your browser
- fixed video resolution parsing for some unusual SAR files. this stretches a video slightly, usually when the vid was created or converted with older analog tech (e.g. NTSC)
- fixed rating system predicate label for 'rated/not rated'
- the issue where miscounts in pending upload data would persist, sometimes leading to an annoying 'pending (13)' style menu that would not clear without debug action, is now fixed in a cheap way. on any upload action, this cached count is reset. a fix for the actual unusual miscount situation will have to come later
- the different in-memory manager objects now save changes at different time intervals–lightweight things like favourite searches still save not long after any change, but column widths, network sessions, and bandwidth use now save only every ten minutes
- I _may_ have fixed an issue with favourite tags not sticking correctly or resetting when added en masse via the tag right-click menu
- I believe I fixed a rare but permanent ui hang on highlighting a gallery or watcher when that same downloader was spamming through a largely 'already in db/previously deleted' list
- copying tags 'with counts' now works correctly for simple tag views (previously, it only worked for 'predicate' views)
- copying tags now preserves the tag order as in the list (previously, it did a human sort)
- to stop status-sorted gallery and watcher list entries bouncing around so much, they now just say 'working' in their status column when they are working. the highlight panel still reports granular file/gallery info. galleries also now say a more solid DONE when complete, to spot them more easily
- the gallery and watcher search/checking column now includes stop status in sort
- fixed the dowloader link in the help to https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Downloaders
- added that same link to the Lain dowloader import panel's help button
- updated cloudscraper to 1.2.46
- updated cloudscraper interfacing code to adapt for new reCaptcha->Captcha object names
- .
- boring code cleanup:
- refactored downloader gui code to its own file
- refactored network gui code to its own file
- refactored service gui code to its own file
- finished import reordering. now all files import in a cleaner order
- further reworked all hydrus imports to be more breadth-first, loading core modules earlier and catching potential errors in nicer places
- checkbox selection is now wrapped in the 'quick' dialog system, and all checkbox selections now use this single method
- simplified and unified a variety of layout code, and fixing some odd layout expanding bugs
- misc code cleanup
- deleted some old unused ui code
next week
Just doing small jobs. I'd like to catch up on the github issue tracker, and I'll maybe do some sibling prep work for a bigger job in v408.
Version 405
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v405/Hydrus.Network.405.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v405.tar.gz
I had a good week. 'system:number of tags' now supports namespace filtering.
number of tags
The 'system:number of tags' predicate now lets you attach a namespace, so you can search for 'files with two character tags' or 'files with more than one creator', or any other combination you can think of. Also, all 'number of tags' queries have been optimised, and are now typically much faster, particularly when mixed with other tags. They are also quickly cancellable, so if you do enter a very slow one, it should respond fairly quickly if you hit the 'stop search' button that appears after a delay.
I updated a bunch of the database code here to support new types of search. With the namespace searching, you can enter a blank namespace to search for 'unnamespaced' tags, and if you have a very specific need, you can now enter a wildcard namespace, like 'crea*', perhaps to help search for typos.
Unfortunately, I am removing 'system:number of tags' from the 'default system predicates' options panel. EDIT: I am hiding the whole panel now. This horrid stack of UI runs on ancient and difficult-to-update code, so rather than wrestle with it more, I plan to retire it and replace it with a clean system that also allows for editing search predicates in place (e.g. right-clicking an active search predicate and changing the >2 to >3 or similar).
I am really happy with how namespace counting turned out, and I have wanted it myself for a long time. Let me know how it works for you.
duplicate subscription queries
I believe the subscription data storage rewrite introduced a bug where when you paste queries into a sub and it says 'x y z are already in the sub and will not be added', they were being added anyway!
I fixed it. If you paste into subs a lot and get this message regularly, please check your subs for dupes! Now we have faster and more flexible subs, some users have also asked for broader de-duping tech that spans across multiple subs or does upper/lower case deduping, so I also expect to write buttons to do this in the near future, so you may also just want to wait for that.
ptr parents
Please note that the public tag repository is turning off tag parent submissions for regular users for a bit. If you haven't seen it already, the 'ptr' tab in manage tag parents will disappear in a few days. The task of cleaning up old and ongoing mistakes is proving too awkward with the current tools, so it is halted until I have a 'virtual parents' rework done, which will allow for more cleanly undoable parents (and hence less contagious bad ones). If you have seen 'shadow the hedgehog' or other bizarre tags appearing in odd places, this is what we hope to fix.
I have scheduled my next 'medium-size' job week, 408, to be for a 'presentation' tag cache, which will allow fast and accurate searching and loading of the tags you actually see on the front end. It will start with siblings, extend to tag filtering/censorship, and then parents will be made virtual and moved to it as well. This was a priority at the end of last year, before Qt and 2020's fun appeared, so I would now like to focus on it again.
full list
- tag search:
- system:number of tags now supports namespaces, for example 'find files with two character tags'! (issue #280)
- it also supports wildcard namespaces, as now do regular namespace search predicates. both run faster. "crea*:anything" is now possible
- system:number of tags has been optimised, and in many cases is now ten to a hundred times faster
- system:number of tags still does not support siblings, something I hope to start correcting as of v408
- both tag existence (numtags =0 or greater than 0) and tag count database routines now respond quickly to 'cancel search' commands, so if you do run a slow query (a bare 'has creator tag' search on 'all known files' on the PTR, for instance), you can now back out quickly after the 'stop' button appears
- note that 'system:number of character tags greater than 0' and '= 0' are equivalent to +/-character:anything, which will be swapped in if you enter these. also, +/-unnamespaced:anything can now appear
- the program is a bit better about determining =0 and greater than 0 and less than 1 being 'none' and 'any but none', when it needs to determine optimisations and special labels
- unfortunately, I am taking away the default value for system:num tags in the options page (edit: I am killing the whole panel now). this old ugly mess of stacked predicate edit panels works on ancient, difficult to update code, so I will retire it and replace it with a unified system that is easy to use, supports in-search system predicate editing, and keeps up with changes automatically
- system:number of tags is now comfortable with redundancies–if you add >2 and >4, it now knows that >4 is the true lower bound (previously, the one used was random)
- boring code changes here:
- updated tag existence and tag count searches to take advantage of the tag cache when in a specific file domain (which is pretty much all the time), which should speed them up significantly
- updated tag existence and tag count searches to more carefully plan their queries, speeding them up both in advantageous and difficult situations
- cleaned up tag existence and tag count code significantly
- updated all edit system predicate panels to return full predicate objects, a step towards decoupling them and allowing in-place system predicate editing
- wrote a new number test object to hold and help with number range test values. num tags now uses it, and eventually all range predicates will too
- the namespace existence search code ('anything' queries) is now folded into the new generalised tag existence search code
- streamlined how the search context propagates through all database tag searchingnow, most queries do not know or care about domain or current/pending statusthey just iterate over n tables as determined by a specialised routine
- added a handful of unit tests for the new namespace num tag searching
- .
- database repair:
- the database menu has a new entry, 'repopulate truncated mappings tables', under the newly renamed 'check and repair' submenu, which will try its best to 'fix' a client.mappings.db file that has been truncated due to hard drive fault by repopulating from the local-file-only tag cache. do not run this unless you know you need to
- the 'help my db is broke.txt' document has a full update pass. the language is clearer, common issues and questions are better addressed, two new recovery routines are added, a section on the stages after boot recovery (like the new repopulate job above) is added, and I added my stock 'now become a backup patrician' nag at the end
- the debug routine to clear cached service info numbers is now moved to the 'regenerate' database menu. this thing fixes hanging incorrect 'pending' counts until I can fix it properly
- .
- the rest:
- fixed an issue where when you pasted queries into a subscription, those that were already in the sub (and got the dialog saying so), were being added anyway! I believe this bug came in the last few weeks, after the data storage rewrite. please check your pasted-into subs for dupes
- fixed tab double middle-click behaviour (so you can spam page close), which I thought I had fixed last week but actually messed up completely right at the end (issue #314)
- cleaned up some more of the page tab event code–it was a mess all around. should all be on Qt now, no wx hacks
- network jobs will no longer wait for and consume bandwidth start tokens while all network traffic is paused. all bandwidth competition now halts. (previously, they would continue to consume tokens according to current rules and then all rush to start as soon as traffic was resumed)
- fixed some client booru/client api requests to correctly 404 on missing file results, rather than 500
- cleaned up some file sort code and fixed the sort string conversion, which was rendering the opposite sort direction (asc/desc) in summary labels (e.g. on manage favourite searches)
- cleaned up some ui layout stretching code, including some borked tag import options expand sizing
- improved some button and padding layout definitions, and improved, slightly, the way the top-right media viewer hover window lays itself out and changes its size on media change
- improved some review services layout. should be fewer weird heights and widths in unusual situations, and the new multi-column list fits better
- the manage subs dialog now saves its changes to db more cleanly and atomically
- updated the default derpibooru parser to pull species tags. ten points if you can guess what that is most of the time
next week
Next week is cleanup. I did some layout/sizing work this week, and it was nice to clear out some cobwebs, so I'll keep pushing on that. I'd like to bring back system predicate defaults as well, with a cleaner system.
Version 403
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v403/Hydrus.Network.403.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v403.tar.gz
Despite losing a couple days to sickness, I had a great two weeks doing small jobs.
shortcuts
I reworked some of the behind the scenes of shortcut commands, most importantly cleaning up a bad decision I made when I first moved to the new system. There are about 120 'simple' commands in the system, and now all of them are defined in a neater way that allows me to change and expand things more easily in future.
Every shortcut now has a nicer 'label' when you select it in the edit shortcuts UI (and anywhere else the command is shown). Rather than the old 'focus_media_viewer', you'll get something like 'keyboard focus: to the media viewer'. Rating shortcuts will also display their service's rating as it understands it, like 'set to 3/5' instead of the ugly internal data of 'set to 0.6'.
Although I have been wanting to add more actions to the systemmpv commands have been delayed far longer than I wantedI have been putting that off until this was done. I hope to push on this more in the near future.
the rest
The options->gui pages page now has options to set the height of the autocomplete dropdown results lists, which were previously hardcoded. It only applies to new lists atm, so restart the client once you find a nice height for new pages.
All 'tag import options' attached to downloaders now have an explicit optional 'tag whitelist'. It is just below the blacklist button. If the whitelist is populated, then any file that does not have at least one of the given tags at the download source will not be imported. This is a specific tool, generally not as commonly useful as a blacklist, but it does allow you to filter downloaders that only let you download by an artist username (e.g. "I want Artist A's Evangelion and Metroid content, but nothing else.").
Right-clicking on many files should now give combined 'times viewed' and 'total viewtime' info lines on the menu. It is cool to see on a big selection of favourites!
An important issue where the Client API could duplicate-add tags to multiple tag services on an 'add url' call is fixed. Thanks to a user for finding the solution here.
Thanks to a user's work, there is a new stylesheet, Hydracula, under options->style. It is a darkmode theme.
The hydrus client no longer does the 'background disk cache' job by default. This routine has become less important over time, and as more users have moved to SSDs anyway, where it has little to no benefit, I am now relegating it to a special mode just for HDD users with larger dbs who need it. All users will be reset to not using it on update, so if it is still important to you, please check options->speed and memory to set it back to what you want.
I neatened layout and menu code in several different places. The 'gui' and 'gui pages' options pages had some attention, and the 'network' menu particularly have been shuffled around. The duplicates page now starts on the 'prep' tab if work needs doing, review service panels now highlight repository errors in red text, the edit subs panel is cleaned and has less spammy help, migrating your database files now hides the main gui window while it works, and the 'restore from backup' command now works with and reports text updates to the exit splash.
A handful of users have had trouble in recent versions getting the windows hydrus executable to run. There are different problems for different users with both the old and new versions of the exe bundler I use, and I am moving back up to the latest this week, as it seems to be less bad overall. If you had trouble booting since v400 and v403 again does not boot for you, please roll back to the last version that did work and let me know. There does not seem to be a great solution here, but maybe with more info I can figure something out, or at least properly diagnose it.
client api for advanced users
We need to experiment with this more, so feel free to play with this if you know what it is about, but for now this is mostly just for Client API developers, so we know what we need to do to get this working nicely.
The Client API (and the ancient and less-used local booru) now tentatively support https. You turn it on under services->manage services. It works like the hydrus server, and the same caveats apply: the certificates behind the https are 'self-signed', which means they work but are imperfect, and any program that wants to talk to them may have to access in a 'do-not-verify' way, or only after adding a specific exception rule, like you sometimes see in web browsers. If your Client API is actually hosted on a domain and you have a proper cert for it, feel free to replace the client.crt and client.key files that will be created in the db dir–the client should pick them up and use them no problem when it next boots.
full list
- shortcuts:
- shortcuts have a backend update this week. a bunch of hacky stuff is now cleaner behind the scenes, and the related UI has some cleanup as well
- converted all 100-odd simple shortcut commands from hacky text ids to a proper enumerated id system, and across every single instance across the program
- wrote nicer descriptive labels for all simple shortcuts. gone is 'focus_media_viewer', now is 'keyboard focus: to the media viewer'
- if you have no like/dislike or numerical services, the respective application command edit panels now say so and do not allow an ok action
- like/dislike rating sub-panels now start with 'like' checked
- when a like/dislike or numerical rating sub-panel is set to 'remove', the action dropdown is set to 'set' (rather than flip) and is disabled, as is the numerical slider
- application commands now state better "3/5" information about rating actions, rating than the underlying "0.6" float implementation
- all application commands existing in shortcuts or elsewhere are updated to the new enumerated id system
- refactored ApplicationCommand (the side of shortcuts that holds the actual action to be done) and its edit UI to new separate files
- completely refactored the application command edit panel, pulling the simple/tag/rating sub-panels into their own decoupled classes, simplifying the tangle and permitting easier future expansion
- rearranged some application command functions and contant definitions to more appropriate locations
- improved how application commands are interrogated by the objects that process them
- added plenty of type hinting around application command processing code
- cleaned up a bunch of shortcut and application command code, including some wx->Qt updates as well
- .
- menu and UI cleanup:
- removed an old wx hack that prohibited last-second ui updates. the exit splash screen now reports final db shutdown info
- if a service or account is currently non-functional (e.g. all repositories are paused), the appropriate status text is now in red
- if there is work to do the first time a duplicate page is opened or looked at, it now moves to the 'preparation' tab
- doing a 'migrate database' file migration now temp-closes the migrate db dialog and hides the main gui while it goes on
- brushed up the tag filter ui a bit–now only one of the tag_filter/blacklist test phrases only show up, in the appropriate context, and the test text input now supports multiple newline-separated tags (e.g. if you want to paste a bunch)
- every panel on review services now has a refresh button to force an update
- the 'clear trash' button on the trash review services panel is now disabled when there is nothing to clear
- updated edit subscription panel to point to the main html help and brushed up that help to talk about file limits more, also the earlier downloader help has a little section to highlight subscriptions and their use
- reworked the 'restore from db backup' command–it is now integrated into client shutdown proper, and reports its basic restore progress to the exit splash screen
- reorganised the 'network' menu. manage subs is now up top, downloader submenus are now split better into high-level vs component-level, and login stuff is pulled to its own submenu
- put 'network traffic' at the top of the network->pause menu
- rearranged some of the 'gui' and 'gui pages' option pages and tucked everything into box sections for clarity
- the search pause/play button on search page tag autocomplete now has a simpler 'search paused' label when paused. the code has a similar nomenclature change, and eventually this will turn into a simple pause/play icon button or similar
- fixed some weirdness with floating autocomplete dropdowns sometimes not appearing on dialogs on first load
- fixed some focus logic so set-focus calls on downloader pages should work again on the query input text box and elsewhere
- unified all numerical rating->stars and stars->rating calculations across the program. this may have fixed some edge-case bugs
- unified all rating string generation across the program
- .
- the rest:
- the disk cache options under _options->speed and memory_ are now default off and force-set off for all users on update. as more users are on decent ssds where these options are of limited value (and sometimes negative value), I now only recommend them for users on HDDs
- added two options for autocomplete results list height to 'gui pages' option page, under the new 'controls' section
- fixed a critical issue where the client api could duplicate-add tags with url imports to multiple services. the potential service duplicate cascade order was pseudorandom and particular to a client. thanks to a user for figuring this issue out (issue #317)
- added a 'tag whitelist' to downloader tag import options. its edit button is below the blacklist. when there are no tags in the list, it does nothing, but if tags are added, then files that do not have at least one of the given tags at the download source will not be imported. for instance, if you have a username-based downloader (where you can't add more tags to the query to filter serverside), and you only want their metroid content, you can now filter it simply hydrusside (issue #279)
- if you are both in advanced mode and a mad lad, the basic blacklist tag filter now allows you to show the 'whitelist' and 'advanced' panels again, if you have a complicated blacklist to set
- the local booru and client api now support the same https as the hydrus server, using self-signed certificates stored in the db directory. just set the checkbox in manage services and you should be good. self-signed certificates are free and will work on a server hosted off an IP address, but they are imperfect. they are also likely to require special permission to be accepted by the web browser or whatever you want to talk to the https service. however, if you host your client from a real DNS domain and have your own fully signed cert+key files, you can swap them in no problem
- local booru and client api urls adjust scheme for the new option, and unified and cleaned up how booru share urls are generated internally
- the way cert+key files are generated is moved from server code to common hydrus code
- cleaned up how additional db files like certificate files and the mpv conf are managed for backup/restore operations
- cleaned up some ancient http urls to https. mostly stuff like the regex tutorial links
- when files are appended to a regular search page (e.g. from a subscription publish to an existing page, or from a mouse drag and drop), the search context will now pause. this is to stop accidental F5 or mass refresh signals wiping out the changed page
- to break advanced-case gallery search loops, gallery url jobs now have a 'run' identity token. galleries pass their token down to 'next page' or 'sub-gallery' urls they generate, meaning all urls of a particular search run share the same url. gallery logs now ignore to-be-added urls that already exist for their token, terminating loops. new tokens are generated if a search is restarted or similar, meaning duplicate urls can exist in a gallery log, just not from the same starting point (issue #302)
- improved simple gallery url deduplication in several stages of the downloader pipeline
- when right-clicking on multiple thumbs, the info lines off the top menu item now list the files' combined viewtimes (this previously only showed when one file was selected)
- fixed some error reporting problems with adding urls to import via the client api–some url class exceptions were being converted from 400 to 500 errors unintentionally
- a new stylesheet, 'Hydracula', is added to the default install. check it out under options->style. thank you to a user for contributing this
- subscriptions are better about calculating a 90 second forgiveness window for bandwidth rules. they should schedule and startup more effectively, and the edit subscriptions and single edit subscription panels should also no longer show bandwidth delays below the next 90s, which are often a technical situation of regular work breaks that are better ignored for the purposes of the dialog
- went back up to pyinstaller 3.6 again on windows, as 3.5 caused its own Qt bindings dll problems. if you had trouble with 3.6 (401), let me know how this works for you, as there are additional dll-finding fixes included (issue #329)
- fixed an issue where under some conditions, file save dialogs were only happy with filenames that already existed (issue #319)
- fixed an issue with the 'client already running' system sometimes not closing the client process correctly when told to cancel the boot
- bumped the 'space needed for vacuum' estimate up to 120% (was 100%) of estimated final file size, just to catch some edge cases
- rolling out updated danbooru parsers that pull associable urls correctly, thank you to a user for this fix
- rolling out an updated deviant art parser that finds some unusual file urls when other methods fail, thank you to another user for this fix (issue #295)
- upgraded cloudscraper to 1.2.42
- improved some type hinting
- fixed up some unit tests for new command and rating data
next week
Next week is a 'medium sized job' week. I want to write a 'list manager' to manage and remember presentation options for all the multi-column lists across the program. The ideal here will be to have selectable columns and columns that remember their last width. It will open the program to supporting clever and specific columns, initially hidden due to size constraints, for things like the edit subscriptions list in future. There are many lists in the program, and a lot of code here, so I am not sure how far I will get, but I think I'll be able to do column width memory at the least.
A few requests/suggestions
But there are a few improvement suggestions I'd like to make, if that's ok? They're pretty small (except for the last one) but I have no idea how to code anything so maybe it's harder than I think… but regardless, here they are:
-It would be very convenient for editing tags if the selection for editing tags was not inside any arrow side menus. So if you right click, the option for "Edit Tags" would just be at the top of the menu, instead of having to hover over "Manage". The attached picture to this post will show my personal ideal right click menu.
-I have a LOT of alternate pictures. Instead of having to go through the right click menus to check if there are any alternates, could there be an icon that tells you whether a picture has alternates or not on the thumbnail? Similar to the Notes icon that was added not long ago.
-In addition to the previous one and like the first request, could the alternate grouping option be put in a more convenient spot in the menu instead of having to go through "manage > file relationships > set relationships > set alternate"? Maybe I'm the only one that has a lot of alternates (I get a lot of art from pixiv and artists love to have multiple images per post with slight variations between them)? But imo this is a very tedious way of setting images as alternates. It would be awesome if the button was just there when the menu is opened, instead of having to hover and guide the mouse cursor over a bunch of side menus.
-To add a new page, could there be a "+" button, similar to adding a new tab in an internet browser?
-And my biggest and most difficult request: having a sort of "folder" system. This would allow folders or archive files of manga to be added to the database. You could make it so that you add all the pages normally, but then you can add a grouping that turns it into a folder of sorts. Maybe when you decide to view the 'folder', it opens a new tab with all the pages? This would be extremely cool, as I like to collect digital manga as well as pictures, and as of now, I cannot sort my manga beyond making folders in Windows Explorer. Which isn't a very good way to sort manga…
If you take the time to read this and consider it, I thank you very much. And I also thank you for working on and updating this program, it has been a game changer for me.
Version 404
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v404/Hydrus.Network.404.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v404.tar.gz
I had a good week working almost entirely on the client's multi-column lists. They now remember their column widths and sort.
multi-column lists
All the tables across the program are now plugged into a unified options structure. This remembers the lists' columns' widths, so if you resize the manage subscriptions list a little tighter, then the next time you load that dialog, it should come back the same. The lists across the program that automatically start and maintain a sort (e.g. manage subs, or the query list in a downloader page), also remember if you change the sort.
This sounds simple, but it involved rewriting critical sections for every list, and there are about 66 different ones! I am happy with the change, as it means a load of duct-tape mess is unified and easier to maintain and extend in future.
I built the system with several near-term extensions in mind: having all lists sort cleanly, quickly, and by default, and show sort up/down arrows; allowing columns to be rearranged; and allowing columns to be hidden, which will also allow me to add complicated columns (e.g. statistically expensive subscription quality info) that start hidden by default.
Although the system is now easier to check for errors, the sheer number of lists and amount of code I changed this week means there is probably a typo somewhere in a column name or initial size. I have checked the main lists closely, but if you encounter anything odd, or even a list that does not boot at all, please let me know. There are still a couple of sizing calculation peculiarities I need to fix.
full list
- column lists:
- all multi-column lists across the program now remember the widths of their columns when they are next recreated
- the last column of any list is now universally the 'stretching' column, which should correctly initialise with its preferred/previous size, but also grows and shrinks with the parent window
- while all lists retain their initial rows height, and those in the gallery and watcher management panels will continue to grow and shrink in a fixed way, all lists in dialog windows can now be shrunk down to four rows
- the minimum size of any column is now much smaller, about three characters
- all column headers now tooltip their name
- lists should be better about sizing in non-100% OS UI scalings
- the lists that are automatically sorted (e.g. the download pages, and manage subs) now remember the last sort you gave them
- future plans, now within reach:
- all lists will sort, sort arrows will appear on the header, and sort will be faster
- columns will be rearrangeable
- columns will be hide/showable, and initially hidden complicated columns will be available
- there will be some, maybe optional, capability to have lists sync live, so if you edit one, the others do the same
- num rows height memory, maybe–we'll see how the above shakes out first
- boring code changes:
- moved list code to a new sub-module
- wrote a status object for column list current columns, widths, and sort, and plugged it into list code
- wrote a manager to handle column statuses, and plugged it into the main controller and db
- wrote definitions for every list (66 or so different lists!) and all their columns, and unified width, name, default sort, and future hideabality and default hide/show status to that one easy to edit and extend location
- rewrote list column and sort initialisation to work off the new status object and added hooks for list sorting and column resizing to save new status back
- rewrote every list column instantiation to use the new system
- numerous misc column list code cleanup
- .
- the rest:
- double middle-clicking on the page tab bar should now correctly close two tabs in a row (rather than opening the rename page dialog on the second)
- entering an odd number of hex characters into system:hash no longer causes an error. this will be changed in future to properly highlight and explain badly pasted or incorrect-length hashes in future
- the new red text for non-functional status texts in review services now properly re-colours itself between normal/red when an error or resolution occurs while the panel is open
- hydrus now knows if it is running in the Haiku operating system and has preliminary platform specificity. if you are interested in helping to get hydrus running properly in Haiku, please join in with github issue #358
- cleaned up a mix of smaller code, unused variables and imports and so on
next week
Back to small jobs next week. This list work monopolised my time, so I want to catch up on messages and some bug fixes, and I'd like to also slip in namespace-specific system:number of tags (like "files with more than one series: tag") as well, if the search logic is not too hellish.
EDIT: I messed up the double middle-click on page tab. I’ll fix it next week!
Version 402
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v402/Hydrus.Network.402.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v402.tar.gz
I had a great week just doing cleanup and other small work.
all misc this week
Most of the pages in the client now have special status bar text for when they have no files. If your search results in no files, instead of the old '0 files', you'll get 'no results for this search', and 'search cancelled!' if the search was cancelled, and so on. It just adds a bit of quick feedback to some actionslike highlighting a gallery query with no new fileswhere the new blank page may just be replacing another blank page.
I added 'copy_xxx_hash' shortcuts to the media shortcut set, for 'md5', 'sha1', and 'sha512'. All the hash-copying code is neated up as well, so if hashes can't be found for some reason, you'll still get what could be found and get nice info on how many were missing and why.
The client's shutdown code is untangled. A variety of unusual shutdown situations should be smoother, less buggy, maybe a little faster, and when errors do occur, they should be caught and reported more gracefully.
Gallery and watcher pages' list right-click menus now have direct links to the selected query or watcher's file and gallery logs. It is just an easier shortcut than highlighting and finding the button.
Forced file maintenance jobs now list a correct total x/y progress on their popups. Previously, they would show x/256 in the batches of 256, with no total summary.
Most of the lists with clipboard/png import/export buttons now also support json files, which is just a straight copy of what would go to your clipboard, but to a file. You can drag and drop compatible json files on the lists as well, just like with pngs.
I have updated the Windows and Linux client to Qt 5.15, up from 5.13. We had a variety of problems with 5.14, but this version seems to be working well. Let me know if you have any trouble, or if any problems are magically fixed.
full list
- in many situationssuch as a search result that gives no results, or a search cancel, or a downloader page cleared of a highlightpages will now report a special status text rather than '0 files', such as 'no results for this search' or 'search cancelled!' (issue #277)
- new pages, and the first page of a loaded session, should now correctly publish their status text to the status bar immediately after initialisation, (previously blank until first change)
- clicking the 'searching immediately' button while a search is ongoing now correctly cancels a search, cleaning up status and page and buttons, rather than just stopping current work immediately
- added 'copy_xxx_hash' shortcuts to the media shortcut set for 'md5', 'sha1', and 'sha512'
- when copying file hashes to clipboard, a popup appears for two seconds to verify what happened
- when copying file hashes to clipboard, recovery from missing hashes is more graceful, with multiple error report states
- the way the client shuts down is untangled. the order in which the gui, managers, threads, database are shut down is smoothed out, with better error handling and fewer potential logical holes
- the 'should I do shutdown work?' dialog is now only presented in the clean shutdown pipeline
- menu labels now elide at 128 characters, extended from 64 previously. hopefully this strikes a better balance between fixed texts we do want to read while still not letting long dynamic texts go nuts (issue #276)
- gallery and watcher pages now have 'show file/gallery log' on their menus, which directly zoom in to the edit dialogs for the top-most selected query or watcher (issue #256)
- when file maintenance is forced to run from the thumbnail menu or file maintenance job panel, it now provides x/y progress text and gauge based on total jobs, e.g. 1,234/10,000, rather than out of the 256-job batches (issue #264)
- the simple downloader page now updates its pending jobs list more efficiently, and supports multiple selection, and presents a yes/no confirmation on delete
- most lists with clipboard/png import/export buttons can now also do .json files. they also accept json files in a drag and drop. you can mix json and png files in a multi-file drag and drop
- when selecting a parser for a url class in 'manage url class links', those parsers with example urls that match the url class are now separately listed at the top of the choice dialog
- in the recent autocomplete rewrite, the hidden repository update file domain was accidentally exposed in the file domain button. after some testing, it actually works(!), but as this is an advanced topic, it is now hidden behind advanced mode
- the way services are deleted or completely reset is now changed to what should be a significantly faster and smaller operation
- the latest user-made nitter/twitter downloader is rolled in to the update. some little fixes and adds support for mobile.twitter.com url imports
- fixed an issue where uninitialised repositories thought they were caught up
- to reflect that it does nothing in this case, the mouse shortcut edit panel now disables the press/release choice on double-click or scroll
- fixed file save dialogs not filling in the default filename properly
- removed an old wx safety hack where new pages would silently not create while the client was minimised. this fixes issues with large session loading and subscriptions publishing files to page names that do not yet exist while the client is minimised
- removed an old wx safety hack where some tag lists would not regen their current tag display while the client was minimised
- in lieu of a future better bit of html subscription help that I link to from the subscription panel, the 'file limits' help button has temporarily briefer text so it doesn't make such a giant popup
- moving back to pyinstaller 3.5 (from 3.6) for the windows build, which appears to fix some dll loading for some users (issue #244)
- the windows and linux builds are updated to Qt 5.15 (from 5.13.2). it does not seem to have the odd problems 5.14 gave us. let me know if you have any trouble or if any weird graphical issues magically fix themselves
- .
- client api:
- the /get_files/file_metadata call has a new true/false parameter, 'detailed_url_information', default false, that adds 'detailed_known_urls' structure to list the known urls results as in /add_urls/get_url_info. it has a help example and a unit test and everything (issue #235)
- the client api version is now 13
- .
- boring cleanup details:
- reshuffled the shutdown code. now the controller takes the lead, booting splash as appropriate and commanding gui to save and close, and then proceeds to other shutdown
- fast and normal shutdown code is unified, just run differently
- shutdown calls should now always be idempotent
- a catch for some OS-level shutdown commands, normally user log-off, also hooks into the newer UI-free fast shutdown
- SIGINT and SIGTERM also hook better into the new shutdown, and are thread safe
- performing multiple SIGINTS on shutdown should no longer throw an error after the gui is deleted
- more potential startup/shutdown errors are now caught and presented to the user and saved to log, with subsequent shutdown urgency accelerated afterwards
- critical errors on a fast shutdown no longer present to the user–they just save to log
- updated how an emergency shutdown state is tested
- updated how a 'clean exit complete' state is set and tested
- various unusual shutdown states now skip human interaction and jump straight to guaranteed fast shutdown
- refactored splash window to its own file
- wrote a new qlistwidget subclass to do some common data storage/retrieval/selection. it will eventually replace most lists across the program
- the 'queue' list widget that has up/delete/down and add/edit buttons beside a list has nicer backend code and now initialises with its buttons correctly disabled due to no selection
- the similar 'add/edit/delete' list widget is updated to use the nicer backend
- some wx->Qt list hacks, which were themselves using borked old display-string-based indexing, are deleted
- the repository download/process daemon has been moved to the newer job scheduler. it should start up and close out on program exit a bit more neatly
- untangled some messy value-change radio button code in the shortcut edit panel
- updated the way page status text propagates up from the thumbnail grid to the main gui to Qt signals instead of the old inefficient pubsub
- all UI file hash clipboard copying code is now unified and improved
- added a new subscription file publish debug test to help->debug->gui
- refactored some client specific time delta rendering code out of core to client
- misc event cleanup code
- misc code style cleanup
next week
Next week is a small jobs week. I'll push like this week, just clearing out small items. I'd like to neaten shortcuts as well, dropping_the_long_command_names for a proper shortcut action object that'll be easier to work with in future and have a nice name. I didn't get to string parsing improvements this week, so I'll see if I can work on that as well.
The new Github Issues workflow we set up last week worked well, I think. The users managing it helped me focus on some nice issues, including a couple that had previously fallen to the back burner, and I was comfortable with the balance of my time on it. As a reminder, if you are comfortable with Github Issues, you can find it here: https://github.com/hydrusnetwork/hydrus/issues
Version 401
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v401/Hydrus.Network.401.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v401.tar.gz
I had an ok week, although I was a little short on work time. This is mostly a bugfix release that tweaks how the new subscriptions work and fixes some downloader update issues. There is also an important new way to make feature suggestions for hydrus.
mostly misc work this week
Now that subscriptions can run in small efficient pieces, I removed several old limiters that stopped the old system running too often. Subscriptions should boot with less delay when needed (e.g. after the client loads or when you ok the dialog after changes), and an old system that batched queries together is gone, so if a query is due, it should run generally as soon as it can.
I also re-added the import/export/duplicate buttons to the manage subscriptions dialog. It also supports importing (and converting to the new format) the old subscription format, so if you have an old .png sub, or you want to reimport the text that was backed up to your database folder last week, this is now possible. Exporting or duplicating a very large subscription may now need a few seconds to do some database work, but the dialog will warn you beforehand.
As users have made and shared more custom downloaders, I have had reports here and there about new default downloaders sometimes messing up the custom links, unintentionally resetting back to those defaults or other unusual link remapping. I have tightened up all this code. Please let me know if you have any more trouble.
And speaking of new defaults, I am rolling out new objects for nitter, derpibooru, and e621 again this week to improve object linking and URL matching. If you had a subscription complain last week about an e621 gallery URL not matching, this should now be fixed.
github issue tracker
As you doubtless know, I have difficulty keeping up with messages. It is great that people like hydrus and think of all sorts of ways to make it better, but as the program has become more popular, feature suggestions in particular has grown into a tsunami that I cannot give the time they deserve. My todo list has about 3,000 items in it. My list is private, so dealing with duplicate requests and keeping track of discussions in threads and chat conversations has been inefficient.
So, with the aid of a team of volunteer users, I am turning the Hydrus Github Issue Tracker back on. I turned it off about three years ago, as it never fit my workflow and I could not keep up with discussions, but I have been talking with some users, and I agree that for the people it does work for, it is excellent.
We are still figuring out the workflow, but the idea is the team of users will label and sort things, participate in discussions about features, and forward to me the most urgent and popular items. I will not be actively managing it, but I'll read the issues I am forwarded and likely close things out once I have them done. If you are comfortable with Github, please feel free to participate yourself. If you have already requested a feature to me, you can still repeat it on the tracker, which will help deduplication for other users and provide a clean space to discuss it with others if you would like. The program and the html help has all been updated with info about and links to the tracker. It is on the main repo, here:
https://github.com/hydrusnetwork/hydrus/issues
I am grateful for the volunteers who are taking on a job my mind just doesn't gel with.
You are still welcome to contact me on any subject however else we used to talk. This is simply a new way to report, particularly for feature suggestions. If you are uncomfortable with Github, no worries. Also, if you would like me to forward an item to the Tracker anonymously, please send me it and I will pass it along.
full list
- subscriptions:
- as subs can now load more flexibly, previously hardcoded waits are now eliminated:
- - the subscriptions manager now only waits three seconds after initial session load to boot (previously 15)
- - the subscriptions manager now wakes as soon as the subscriptions dialog is ok'd or cancelled
- - a timing calculation that would delay the work of a sub up to five or fifteen minutes if more queries would come due for sync in that time window (in order previously to batch to reduce read/write) is now eliminated–subs will now start as soon as any query is due. if you were ever confused why a query that seemed due did not boot after dialog ok or other wake-up event, this _should_ no longer happen
- re-added the import/export/duplicate buttons to manage subs. export and dupe may need to do db work for a couple of seconds and will have a yes/no confirmation on larger jobs
- the import button on manage subs accepts and converts the old 'legacy' subscription object format, including a copy/paste of the objects backed up to disk in the v400 update
- fixed an issue where creating a subscription query and then deleting it in the same manage subs dialog session would result in surplus data being written to the db (which the next dialog launch would note and clear out)
- an unusual error with pre-run domain checking, exposed by the new subscription code and e621 subs, where the gallery url has also recently changed, is now fixed
- .
- issue tracker:
- the Github issue tracker (https://github.com/hydrusnetwork/hydrus/issues) is turned on again! it is now run by a team of volunteer users. the idea is going to be to try to merge duplicate feature suggestions with the proper platform and put some discussion and cognition and prioritisation into idea development before it gets to my desk, so I can be more focused and productive and so 95% of feature suggestions do not simply get banished to the shadow realm of the back of my todo
- this is mostly intended for wishlist and other suggestions, as the tsunami was just getting too much for me to handle, but we'll see how it goes for things like bug reports as well. I'll still take any sort of report through my normal channels, if you are uncomfortable with github, or if you wish for me to forward an item to the issue tracker anonymously
- the website, help documents, and hydrus help menu links have been updated regarding the issue tracker
- .
- the rest:
- improved how the database 'update default downloader objects' job works, ensuring that new defaults are better at simply take the place of existing objects and do not break/reset existing url class to parser links
- tightened up how automatic url class to parser linking works, eliminating some surplus and potentially bad data related to api links. furthermore, whenever the links between url classes and parsers update, existing surplus data, which may creep in when api links change, is now cleaned from the data structure
- rolling out updated e621 url class and parser to deal with their alternate gallery url format
- rolling out an updated derpibooru parser that will link to the new api class correctly
- thanks to a user's submission, rolling out updated versions of the new default nitter parsers that pull creator:username tags
- before every subprocess launch, and when waiting for all subprocess communication (e.g. to ffmpeg), now tests regularly for program shutdown. if an unusual situation develops where a subscription is doing a file import job while the OS is shutting down, and that system shut down would hang or is hanging on a 'ffmpeg can't be launched now' dialog, the hydrus client should now notice this and bomb out, rather than going for that never-running ffmpeg. this may not fix all instances of this issue, and further feedback on the client not closing down cleanly with the OS is welcome.
- when adding a new path to the 'migrate database' panel, any symbolic links will be converted to canonical equivalents
- added some location checks and appropriate errors when the database is doing file storage rebalancing
- fixed an issue uploading swfs, video, or audio to the server when it is launched from a frozen executable build
- misc code cleanup
next week
Next week is cleanup. I want to untangle some of the convoluted shutdown code, push more on wx->Qt cleanup, and revisit the new text parsing so I can get closer to multiline note parsing for the downloader.
Version 400
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v400/Hydrus.Network.400.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v400.tar.gz
🎉🎉🎉 MERRY v400! 🎉🎉🎉
I had a great week of vacation, and then a great week finally getting the subscription data overhaul done.
subscriptions
When I first wrote subscriptions, they could only hold one simple query each. Queries have become much more complicated since then, and subscriptions can of course hold many queries at once, sometimes hundreds. The old monolithic method of storing and loading subs was creaking at the seams. This week fixes it, and subscriptions should now load and operate quickly for all normal operations.
Subscriptions are now broken into pieces. Essentially, instead of one thing holding everything, they now store each query as a separate object and load and save each from your database as they are needed. The 'top' of a subscription is now always in memory and allows the manage subscriptions dialog to start instantly. Subscriptions can also boot quickly, and will cause less lag as they finish up. It all saves time and database read/write. The old '200,000 files' limit for subscriptions is gone. I wouldn't advise you make a sub with 10,000 queries just yet, but you do not have to worry about the size of any one sub too much any more.
There are no significant changes to how subscriptions look or are edited. All your existing subscriptions will be converted to the new format on update. However this is a big change behind the scenes, and if you have big subs, it may take a minute or two to update your databases. Your old subscription objects will also be backed up to a new subdirectory in your db directory, just in case anything goes wrong now or in the near future.
Unfortunately, as subscriptions are now more complicated, I did not have time to write new import/export system for them. The duplicate/import/export buttons on the manage subscriptions dialog are hidden for now.
This took a lot of planning, prep, and work. I hope you find your subscriptions work nicer, and if you have any trouble, please let me know.
new downloaders
Twitter retired their old API on the 1st of June, which broke our downloader. There is unfortunately no good hydrus solution for their new API, but thanks to a user's efforts, I am rolling in a parser for nitter, a twitter wrapper, this week. It has three downloadersone for media posts, one for retweets, and one that does bothso please play with it and then move your twitter subscriptions over to it.
Also fixed should be derpibooru search and the md5 hash parsing of the danbooru downloader (which speeds up some downloading).
full list
- subscription data overhaul:
- the formerly monolithic subscription object is finally broken up into smaller pieces, reducing work and load lag and total db read/write for all actions
- subscriptions work the same as before, no user input is required. they just work better now™
- depending on the size and number of your subscriptions, the db update may take a minute or two this week. a backup of your old subscription objects will be created in your db directory, under a new 'legacy_subscriptions_backup' subdirectory
- the manage subscriptions dialog should now open within a second (assuming subs are not currently running). it should save just as fast, only with a little lag if you decide to make significant changes or go into many queries' logs, which are now fetched on demand inside the dialog
- when subscriptions run, they similarly only have to load the query they are currently working on. boot lag is now almost nothing, and total drive read/write data for a typical sub run is massively reduced
- the 'total files in a sub' limits no longer apply. you can have a sub with a thousand queries and half a million urls if you like
- basic subscription data is now held in memory at all times, opening up future fast access such as client api and general UI editing of subs. more work will happen here in coming weeks
- if due to hard drive fault or other unusual situations some subscription file/gallery log data is missing from the db, a running sub will note this, pause the sub, and provide a popup error for the user. the manage subscription dialog will correct it on launch by resetting the affected queries with new empty data
- similarly, if you launch the manage subs dialog and there is orphaned file/gallery log data in the db, this will be noticed, with the surplus data then backed up to the database directory and deleted from the database proper
- subscription queries can now handle domain and bandwidth tests for downloaders that host files/posts on a different domain to the gallery search step
- if subs are running when manage subs is booted, long delays while waiting for them to pause are less likely
- some subscription 'should run?' tests are improved for odd situations such as subs that have no queries or all DEAD queries
- improved some error handling in merge/separate code
- the 'show/copy quality info' buttons now work off the main thread, disabling the sub edit dialog while they work
- updated a little of the subs help
- .
- boring actual code changes for subs:
- wrote a query log container object to store bulky file and gallery log info
- wrote a query header object to store options and cache log summary info
- wrote a file cache status object to summarise important info so check timings and similar can be decided upon without needing to load a log
- the new cache is now used across the program for all file import summary presentation
- wrote a new subscription object to hold the new query headers and load logs as needed
- updated subscription management to deal with the new subscription objects. it now also keeps them in memory all the time
- wrote a fail-safe update from the old subscription objects to the new, which also saves a backup to disk, just in case of unforeseen problems in the near future
- updated the subscription ui code to deal with all the new objects
- updated the subscription ui to deal with asynchronous log fetching as needed
- cleaned up some file import status code
- moved old subscription code to a new legacy file
- refactored subscription ui code to a new file
- refactored and improved sub sync code
- misc subscription cleanup
- misc subscription ui cleanup
- added type hints to multiple subscription locations
- improved how missing serialisable object errors are handled at the db level
- .
- client api:
- the client api now delivers 'is_inbox', 'is_local', 'is_trashed' for 'GET /get_files/file_metadata'
- the client api's Access-Control-Allow-Headers CORS header is now '*', allowing all
- client api version is now 12
- .
- downloaders:
- twitter retired their old api on the 1st of June, and there is unfortunately no good hydrus solution for the new one. however thanks to a user's efforts, a nice new parser for nitter, a twitter wrapper, is added in today's update. please play with itit has three downloaders, one for a user's media, one for retweets, and one for both togetherand adjust your twitter subscriptions to use the new downloader as needed. the twitter downloader is no longer included for new hydrus users
- thanks to a user's submission, fixed the md5 hash fetching for default danbooru parsers
- derpibooru gallery searching _should_ be fixed to use their current api
- .
- the rest:
- when the client exits or gets a 'modal' maintenance popup window, all currently playing media windows will now pause
- regrettably, due to some content merging issues that are too complicated to improve at the moment, the dupe filter will no longer show the files of processed pairs in the duplicate filter more than once per batch. you won't get a series of AB, AC, AD any more. this will return in future
- the weird bug where double-clicking the topmost recent tags suggestion would actually remove the top two items _should_ be fixed. general selection-setting on this column should also be improved
- middle-clicking on a parent tag in a 'write' autocomplete dropdown no longer launches a page with that invalid parent 'label' tag included–it just does the base tag. the same is true of label tags (such as 'loading…') and namespace tags
- when changing 'expand parents on autocomplete' in the cog button on manage tags, the respective autocomplete now changes whether it displays parents
- this is slightly complicated: a tag 'write' context (like manage tags) now presents its autocomplete tags (filtering, siblings, parents) according to the tag service of the parent panel, not the current tag service of the autocomplete. so, if you are on 'my tags' panel and switch to 'all known tags' for the a/c, you will no longer get 'all known tags' siblings and parents and so on presented if 'my tags' is not set to take them. this was sometimes causing confusion when a list showed a parent but the underlying panel did not add it on tag entry
- to reduce blacklist confusion, when you launch the edit blacklist dialog from an edit tag import options panel, now only the 'blacklist' tab shows, the summary text is blacklist-specific, and the top intro message is improved. a separate 'whitelist' filter will be added in the near future to allow downloading of files only if they have certain tags
- 'hard-replace siblings and parents' in _manage tags_ should now correctly remove bad siblings when they are currently pending
- network->downloaders->manage downloader and url display now has a checkbox to make the media viewer top-right hover show unmatched urls
- the '… elide page tab names' option now applies instantly on options dialog ok to all pages
- added 'copy_bmp_or_file_if_not_bmpable' shortcut command to media set. it tries copy_bmp first, then copy_file if not a static image
- fixed some edit tag filter layout to stop long intro messages making it super wide
- fixed an issue where tag filters could accept non-whitespace-stripped entries and entries with uppercase characters
- fixed a display typo where the 'clear orphan files' maintenance job, when set to delete orphans, was accidentally reporting (total number of thumbnails)/(number of files to delete) text in the file delete step instead of the correct (num_done/num_to_do)
- clarified the 'reset repository' commands in review services
- when launching an external program, the child process's environment's PATH is reset to what it was at hydrus boot (removing hydrus base dir)
- when launching an external program from the frozen build, if some Qt/SSL specific PATH variables have been set to hydrus subdirectories by pyinstaller or otherwise, they are now removed. (this hopefully fixes issues launching some Qt programs as external file launchers)
- added a separate requirements.txt for python 3.8, which can't handle PySide2 5.13.0
- updated help->about to deal better with missing mpv
- updated windows mpv to 2020-05-31 build, api version is now 1.108
- updated windows sqlite to 3.32.2
next week
Next week is a small jobs week. I'll get subscription import/export working and then hack away at my bugs list and other little things to catch up on.
Version 399
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v399/Hydrus.Network.399.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v399.tar.gz
I had a great week tidying up smaller issues before my vacation.
all small items this week
You can now clear a file's 'viewing stats' back to zero from their right-click menus. I expect to add an edit panel here in future. Also, I fixed an issue where duplicate filters were still counting viewing time even when set in the options not to.
When I plugged the new shortcuts system's mouse code into the media viewer last week, it accidentally worked too well–even clicks were being propagated from the hover windows to the media viewer! This meant that simple hover window clicks were triggering filter actions. It is fixed, and now only keyboard shortcuts will propagate. There are also some mouse wheel propagation fixes here, so if you wheel over the taglist, it shouldn't send a wheel (i.e. previous/next media) event up once you hit the end of the list, but if you wheel over some hover window greyspace, it should.
File delete and undelete are now completely plugged into the shortcut system, with the formerly hardcoded delete key and shift+delete key moved to the 'media' shortcut set by default. Same for the media viewer's zoom_in and zoom_out and ctrl+mouse wheel, under the 'media viewer - all' set. Feel free to remap them.
The new tag autocomplete options under services->tag display and search now allow you to also search namespaces with a flat 'namespace:', no asterisk. The logic here is improved as well, with the 'ser'->'series:metroid' search type automatically assuming the 'namespace:' and 'namespace:*' options, with the checkboxes updating each other.
I fixed an issue created by the recent page layout improvements where the first page of a session load would have a preview window about twenty pixels too tall, which for some users' workflows was leading to slowly growing preview windows as they normally used and restarted the program. A related issue with pages nested inside 'page of pages' having too-short preview windows is also fixed. This issue may happen once more, but after one more restart, the client will fix the relevant option here.
If you have had some normal-looking files fail to import, with 'malformed' as the reason, but turning off the decompression bomb check allowed them, this issue is now fixed. The decomp bomb test was itself throwing an error in this case, which is now caught and ignored. I have also made the decomp bomb test more lax, and default off for new users–this thing has always caught more false positives than true, so I am now making it more an option for users who need it due to memory limitations than a safeguard for all.
advanced parsing changes
The HTML and JSON parsing formulae can now do negative indexing. So, if you need to select the '2nd <a> tag from the end of the list', you can now set -2 as the index to select. Also, the JSON formula can now index on JSON Objects (the key->value dictionaries), although due to technical limitations the list of keys is sorted before indexing, rather than selecting the data as-is in the JSON document.
Furthermore, JSON formulae that are set to get strings no longer pull a 'null' value as the (python) string 'None'. These entries are now ignored.
I fixed an annoying issue when hitting ok on 'fixed string' String Matches. When I made the widgets hide and not overwrite the 'example string' input last week, I forgot to update the ok validation code. This is now fixed.
full list
- improvements:
- the media viewer and thumbnail _right-click->manage_ menus now have a _viewing stats->clear_ action, which does a straight-up delete of all viewing stats record for the selected files. 'edit' will be added to this menu in future
- extended the tag autocomplete options with a checkbox to allow 'namespace:' to match all tags, without the explicit asterisk
- tag autocomplete options now permit namespace searches if the 'search namespaces into full tags' option is set
- the tag autocomplete options panel now disables and checks the namespace checkboxes when one option overrules another
- cleaned up some tag search logic to recognise and deal with 'namespace:' as a query
- added some more unit tests for tag autocomplete options
- the html and json parsing formulae now support negative indexing, to select the nth last item from a list
- extended the '1 -> "1st"' ordinal string conversion code to deal with negative indices
- the 'hide tag' taglist menu actions are now wrapped in yes/no dialogs
- reduced the activation-to-click-accept time that the shortcuts handler uses to ignore activating clicks from 100ms to 17ms
- clicking the media viewer's top hover window's zoom buttons now forces the 'media viewer center' zoom centerpoint, so if you have the mouse centerpoint set, it won't zoom around the button where you are clicking!
- added a simple 8chan.moe watcher to the defaults, all users will get it on update
- the default bandwidth rules for download pages, subs, and watchers are now more liberal. only new users will get these. various improvements to db and ui update pipeline mean the enforced breaks are less needed
- when a manage tags dialog moves to another media, if it has a 'recent tags' suggestion list with a selection, the selection now resets to the top item in the list
- the mpv player now tracks when a video is fully loaded and only reports seek bar info and allows seeks when this is so (this should fix some seekbar errors on broken/slow-loading vids)
- added 'undelete_file' to media shortcut commands
- file delete and undelete are no longer hardcoded in the media viewer and media thumbnail grid. these actions are now handled entirely in the media shortcut set, and added to all clients by default (this defaults to (shift +) delete key, and also backspace on macos, so likely no changes)
- ctrl+mouse wheel is no longer hardcoded to zoom in the media browser. these actions are now handled entirely in the 'all' media viewer shortcut set (this defaults to ctrl+wheel or +/-, so likely no changes)
- deleted some old shortcut processing code
- tightened up some update timers to better halt work while the client is minimised to system tray. this _may_ improve some users' restore hanging issues
- as Qt is happier than wx about making pages on a non-visible client, subscriptions and various url import operations are now permitted to create pages while the client is minimised to taskbar or system tray. if this applies to your situation, please let me know how you get on here, as this may relieve some restore hanging as the pending new-file jobs are no longer queued up
- .
- fixes:
- clicks on hover window greyspace should no longer propagate up to the media viewer. this was causing weird archive/delete filter actions
- mouse scroll on hover window taglist should no longer propagate up to the media viewer when the taglist has no more to scroll in that direction
- fixed an issue that meant preview windows were initialising about twenty pixels too short for the first page loaded in a session, and also pages created within nested page of pages. also cleaned up some logic for unusual situations like hidden preview windows. one more cycle of closing and reopening the client will fix the option value here
- cleaned and unified some page sash setting code, also improving the 'hide preview window' option reliability for advanced actions
- fixed a bug that meant file viewtime was still being recorded on the duplicate filter when the special exception option was off
- reduced some file viewtime manager overhead
- fixed an issue with database repair code when local_tags_cache is missing
- fixed an issue updating a very old db not recognising that local_tags_cache does not yet exist for proper reason and then trying to repair it before update code runs
- fixed the annoying issue introduced in the recent string match overhaul where a 'fixed character' string match edit panel would not want to ok if the (now hidden) example string input did not have the same fixed char data. it now validates no matter what is in the hidden input
- potentially important parsing fix: JSON parsing, when set to get strings, no longer converts a 'null' value to 'None'
- the JSON parsing formula now allows you to select the nth indexed item of an Object (a JSON key->value dictionary). due to technical limitations, it alphabetises the keys, not selecting them as-is in the JSON itself
- images that do not load in PIL no longer cause mime exceptions if they are run through the decompression bomb check
- .
- misc:
- boosted the values of the decompression bomb check anyway, to reduce false positives. it generally now has a problem with images with a bmp > 1GB memory
- by default, new file import options now start with decompression bombs allowed. this option is being reduced to a stopgap for users with less memory
- 'MimeException' is renamed to 'UnsupportedFileException'
- added 'DamagedOrUnusualFileException' to handle normally supported files that cannot be parsed or loaded
- 'SizeException' is split into 'TagSizeException' and 'FileSizeException'
- improved some file exception inheritance
- removed the 'experimental' label from sub-gallery page url type in parsing system
- updated some advanced help regarding bad files
- misc help updates
- updated cloudscraper to 1.2.40
next week
I am taking next week off. Normally I'd be shitposting E3, but instead I think I am going to finally get around to listening to the Ring Cycle through and giving Kingdom Come - Deliverance a go.
v400 will therefore be on the 10th of June. I hope to have the final part of the subscription data overhaul done, which will mean subscriptions load in less than a second, reducing how much data it needs to read and write and ultimately be more accessible for the Client API and things like right-click->add this query to subscription "blahbooru artists".
Thanks everyone!
Site Request
I'm a new user to Hydrus and I'm kind of confused about how you create the images to add a new downloader. If they can, can someone either tell me how to make it or even better post an image for it. The website I'm requesting is "https://booru.allthefallen.moe/". Call me what you want.
Version 398
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v398/Hydrus.Network.398.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v398.tar.gz
I had a good work week. Tag autocomplete gets some new search options, and advanced users who make downloaders get some new text processing tools.
tag autocomplete
When I recently overhauled the tag autocomplete pipeline, I eliminated some unusual logical hoops where you could accidentally fire off expensive searches that would fetch all tags. Now the code is clean, I am adding them back in as real options.
The main thing here is that services->tag display is now services->tag display and search. It has several new options to change search based on what the autocomplete's current 'tag domain' is (i.e. what the button on the dropdown says, "all known tags" or "my tags" or whatever else). The options are available for every specific tag domain and the "all known tags" domain, and only apply there.
There are three new search options: You can have full namespace lookup, so an input of 'ser' also finds 'series:metroid' and all other series tags; you can have an explicit input of 'series:*' show all 'series' tags; and you can have '*' show all tags. These queries are extremely expensive for a large service like the public tag repository (they could take minutes to complete, and eat a ton of memory and CPU), but they may be appropriate for a smaller domain like "my tags". Please feel free to play with them.
There are also a couple of clever options setting how 'write' autocompletes (the ones that add tags, like in the manage tags dialog) start up, based on the tag service of the page they are on. You can set them to start with a different file or tag domain. Most users will be happy with the defaults, which is to stick with the current tag domain and "all known files", but if you want to change that (e.g. some users like to get suggestions for "my tags" from the PTR, or they don't want tag counts from files not in "my files"), you now can. The old option under options->tags that did the "all known files" replacement for all write autocompletes is now removed.
I have optimised the database autocomplete search code to work better with '*' 'get everything' queries. In the right situation, these searches can be very fast. This logic is new, the first time I have supported it properly, so let me know if you discover any bugs.
string processing
This is only important for advanced users who write downloaders atm. It will come to the filename tagging panel in future.
I am plugging the new String Processor today into all parsing formulae. Instead of the old double-buttons of String Match and String Converter, these are now merged into one button that can have any combination of ordered Matches and Converters, so if you want to filter after you convert, this is now easy. There is new UI to manage this and test string processing at every step.
The String Processor also provides the new String Splitter object, which takes a single string like '1,2,3' and lets you split it by something like ',' to create three strings [ '1', '2', '3' ]. So, if your HTML or JSON parsing provides you with a line with multiple things to parse, you should now be able to split, convert, and match it all, even if it is awkward, without voodoo regex hackery.
I also did some background work on improving how the parsing example/test data is propagated to different panels, and several bugs and missed connections are fixed. I will keep working here, with the ideal being that every test panel shows multiple test data, so if you are parsing fifty URLs, a String Processor working on them will show how all fifty are being converted, rather than the current system of typically just showing the first. After that, I will get to work on supporting proper multiline parsing so we can parse notes.
the rest
Double-clicking a page tab now lets you rename it!
system:time imported has some quick buttons for 'since 1/7/30 days ago'.
I cleaned out the last of the behind-the-scenes mouse shortcut hackery from the media viewer. Everything there now works on the new shortcuts system. There aren't many front-end changes here, but a neat thing is that clicking to focus an unfocused media window no longer activates the shortcut for that click! So, if you have an archive/delete filter, feel free to left-click it to activate it–it won't 'keep and move on' on that first click any more. I will continue to push on shortcuts in normal weekly work, adding mouse support to more things and adding more command types.
You can now enter percent-encoded characters into downloader queries. A couple of sites out there have tags with spaces, like '#simple background', which would normally be broken in hydrus into two tags [ '#simple', 'background' ]. You can now search for this with '#simple%20background' or '%23simple%20background'. Generally, if you are copy/pasting any percent-encoded query, it should now work in hydrus. The only proviso here is %25, which actually is %. If you paste this, it may work or not, all bets are off.
I am rolling out updated Gelbooru and Newgrounds parsers this week. Gelbooru searching should work again, and Newgrounds should now get static image art.
full list
- new tag search options:
- there are several new options for tag autocomplete under the newly renamed _services->tag display and search_:
- for 'manage tags'-style 'write' autocompletes, you can now set which file service and tag service each tag service page's autocomplete starts with (e.g. some users have wanted to say 'start my "my tags" service looking at "all known files" and "ptr"' to get more suggestions for "my tags" typing). the default is 'all known files' and the same tag service
- the old blanket 'show "all known files" in write autocompletes' option under _options->tags_ is removed
- you now can enable the following potentially very slow and expensive searches on a per-tag-domain basis:
- - you can permit namespace-autocompleting searches, so 'ser' also matches 'ser*:*', i.e. 'series:metroid' and every other series tag
- - you can permit 'namespace:*', fetching all tags for a namespace
- - you can permit '*', fetching all tags （╬ಠ益ಠ)
- '*' and 'namespace:*' wildcard searches are now significantly faster on smaller specific tag domains (i.e. not "all known tags")
- short explicit wildcard searches like "s*" now fire off that actual search, regardless of the 'exact match' character threshold
- queries in the form "*:xxx" are now replaced with "xxx" in logic and display
- improved the reliability of various search text definition logic to account for wildcard situations properly when doing quick-enter tag broadcast and so on
- fixed up autocomplete db search code for wildcard namespaces with "*" subtags
- simplified some autocomplete database search code
- .
- string processing:
- the new string processor is now live. all parsing formulae now use a string processor instead of the string match/transformer pair, with existing matches and transformers that do work being integrated into the new processor
- thus, all formulae parsing now supports the new string splitter object, which allows you to split '1,2,3' into ['1','2','3']
- all formulae panels now have the combined 'string processing' button, which launches a new edit panel and will grow in height to list all current processing steps
- the stringmatch panel now hides its controls when they are not relevent to the current match type. also, setting fixed match type (or, typically, mouse-scrolling past it), no longer resets min/max/example fields)
- the string conversion step edit panel now clearly separates the controls vs the test results
- improved button and summary labelling for string tools across the program
- some differences in labelling between string 'conversion' and 'transformation' are unified to 'conversion' across the program
- moved the test data used in parsing edit panels to its own object, and updated some of the handling to support passing up of multiple example texts
- the separation formula of a subsidiary page parser now loads with current test data
- the string processing panel loads with the current test data, and passes the first example string of the appropriate processing step to its sub-panels. this will be expanded in future to multiple example testing for each panel, and subsequently for note parsing, multiline testing
- added safety code and unit tests to test string processing for hex/base64 bytes outcomes. as a reminder, I expect to eliminate the bytes issue in future and just eat hashes as hex
- cleaned up a variety of string processing code
- misc improvements to string processing controls
- .
- the rest:
- double-clicking a page tab now opens up the rename dialog
- system:time imported now has quick buttons for 'since 1/7/30 days ago'
- all hydrus downloaders now accept percent-encoded characters in the query field, so if you are on a site that has tags with spaces, you can now enter a query like "simple%20background red%20hair" to get the input you want. you can also generally now paste encoded queries from your address bar into hydrus and they should work, with the only proviso being "%25", which is "%", when all bets are off
- duplicates shut down work (both tree rebalancing and dupe searching) now quickly obeys the 'cancel shutdown work' splash button
- fixed a signal cleanup bug that meant some media windows in the preview viewer were hanging on to and multiplying a 'launch media' signal and a shortcut handler, which meant double-clicking on the preview viewer successively on a page would result in multiple media window launches
- fixed an issue opening the manage parsers dialog for users with certain unusual parsers
- fixed the 'hide the preview window' setting for the new page layout method
- updated the default gelbooru gallery page parser to fix gelb gallery parsing
- updated the newgrounds parser to the latest on the github. it should support static image art now
- if automatic vacuum is disabled in the client, forced vacuum is no longer prohibited
- updated cloudscraper for all builds to 1.2.38
- .
- boring code cleanup:
- all final mouse event processing hackey is removed from the media viewers, and the shortcut system is now fully responsible. left click (now with no or any modifier) is still hardcoded to do drag but does not interfere with other mapped left-click actions
- the duplicates filter no longer hardcodes mouse wheel to navigate–whatever is set for the normal browser, it now obeys
- cleaned up some mouse move tracking code
- clicking to focus an unfocused media viewer window will now not trigger the associated click action, so you can now click on archive/delete filters without moving on!
- the red/green on/off buttons on the autocomplete dropdown are updated from the old wx pubsub to Qt signalling
- updated wx hacks to proper Qt event processing for splash window, mouse move events in the media viewer and the animation scanbar
- cleaned up how some event filtering and other processing propagates in the media viewer
- deleted some old unused mouse show/hide media viewer code
- did some more python imports cleanup
- cleaned up some unit test selection code
- refactored the media code to a new directory module
- refactored the media result and media result cache code to their own files
- refactored some qt colour functions from core to gui module
- misc code cleanup
next week
I will be taking my week vacation after next week, and I don't want to accidentally create any big problems for the break, so I will try to mostly do small cleanup work and bug fixes.
Version 397
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v397/Hydrus.Network.397.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v397.tar.gz
I had an ok week. I mostly did small fixes and improvements.
highlights
I added 'system:has/has no note with name xxx' to search for files with specific note names. This is bundled into the same dialog as last week's 'number of notes', for the new generic 'system:notes' entry in the normal system predicates list.
When you have many page tabs in a row, they will now middle…elide their name text to try to fit better. You can turn this off if you like under options->gui pages.
When files in the duplicate filter are pixel duplicates, the file size and import time comparison statements now have no 'score' and colour blue. Pixel duplicates are often a different judgment call, so until I can get more flexible scoring options here, I am hardcoding it to be less visually green/red.
If you need to run a gallery download with a search term that includes '#', this should now work.
I updated the IPFS code to work with the new IPFS 5.0 daemon. I think I have things mostly correct, but the new daemon has some new CORS restrictions that may need configuration on the IPFS side, depending on your situation, to give hydrus permission to talk to it.
The left page panel's position should set itself more reliably for users who start the program minimised to system tray. If you have had trouble here, please let me know how you get on. I believe the changes here will also reduce some UI lag for many other page adding and moving operations, particularly for large sessions.
full list
- regular changelog:
- added 'system:has/has no note with name xxx' to search for specific note names
- in the normal system predicate list, the notes pred is now the generic 'system:notes' to launch a combined dialog for both num notes and named notes
- favourite tag suggestions are now sorted in manage tags dialog according to the default tag sort
- page names will now middle…elide when there are too many to fit into a row (and normally left/right buttons would be added). if the elided tabs still do not fit, the buttons will pop up as before. added a checkbox to options->gui pages to turn this text eliding off
- pulled the 'page name' options on that panel into their own box and added some text regarding the 'my big row of import page tabs keeps scrolling weird' issue
- when files are pixel duplicates, the filesize and age comparison statements will now have 0 score and thus be coloured neutral blue
- the standard text entry dialog now always selects any default text it starts with, so you can now type to immediately overwrite. see how you like it and if there are some places where you think an exception should be made
- updated the IPFS interface to work with the new IPFS 5.0. all api requests are now POST so it doesn't 405, and the User-Agent is overridden to one that IPFS will not 403 at, and I fixed a typo the new api is more strict about
- a hack to get page splitters to lay out correctly on session load is rewritten from a hammer to a scalpel. pages now set their splitter positions on their first individual visible selection. this both reduces some minor ui lag on session/page load and improves splitter positions for clients that open minimised to the system tray
- a long-time odd issue where loaded sessions would initially select the top-left-most non-page of pages is fixed. now the bottom-left-most page of any kind is selected
- fixed tag autocomplete selecting the bottom-most pre-loading result. it now correctly selects at the top
- fixed an issue setting certain values (typically loading a default) to a tag import options panel
- the client is now more aggressive about clearing subscriptions from memory when they are finished running
- in windows, the main method that copies files now checks for modified time of the source file. if it is before 1980-01-01 UTC, it does not copy the file metadata, as some Windows has trouble with this lmaoooo
- cleaned up how some thumbnail 'current focus' media determination code works. should have fixed some weird errors when hitting certain shortcuts on collections
- cleaned up basic list/sort code across the program
- the 'queue' and add/edit/delete listboxes now emit change signals when new items are added or imported
- pyparsing, a helper for cloudscraper, is now correctly bundled in the built releases. a new line in help->about displays this
- help->about now lists cloudscraper version
- updated the discord link to the new https://discord.gg/wPHPCUZ
- .
- upcoming string processing changes for advanced users:
- I extended string parsing code this week, but I am not yet ready to turn it on. when it does come on, it will change all formulae from the fixed string match/converter pair a combined general string processing 'script' of n steps
- wrote a new 'string splitter' object that takes one strings and splits it into up to n strings based on a separator phrase (such as ' ,')
- wrote an edit panel for string splitters
- wrote a new 'string processor' object that holds n ordered string match/converter/splitter objects and filters/converts/splits x strings into y strings based on those steps
- wrote an edit panel for string processors. it has a notebook that live updates with test results for each step on every update
- wrote unit tests for string match
- wrote unit tests for string converter
- wrote unit tests for string splitter
- wrote unit tests for string processor
- refactored string conversion edit panels to their own file
- refactored string conversion controls to their own file
- misc string processing cleanup and labelling improvements
- .
- technical url parsing stuff:
- urls are now stripped of leading and trailing whitespace during normalisation, just in case a paste contains some extra whitespace. previously, it would sometimes throw a 'doesn't start with http' error
- the hydrus url normalisation process now normalises the hostname according to the NKFC unicode format, meaning unusual characters like ？and e◌ are now replaced with their normalised visual equivalent ? and é, and hence these urls will no longer throw errors when they are added
- if '?' or '#' end up in a hostname (which are invalid characters), it is now converted to _, just to stop complete parse mangling when weird urls are submitted. this character replacement may become more sophisticated in future
- the hydrus downloader should now support search terms that include '#'
- download query parameters that contain '%23' ('#', encoded) are now not unquoted in url normalisation
next week
Next week is a cleanup week. I will push on more wx->Qt code cleanup. I would also like to try to turn on some of the background text/notes work I have been doing for the past couple of weeks.
I will be taking my week vacation in two weeks, in the 399->400 gap from the 27th to the 3rd.
Version 396
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v396/Hydrus.Network.396.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v396.tar.gz
I had a good week. The client now has better file 'notes' support.
There are several database updates this week. Users with large databases can expect the first boot to take a few minutes.
notes
A long time ago, I added prototype 'notes' to files. You can access it under a media right-click->manage->notes. It was a bit hacky, functioning just like a plain .txt attached to files, but some users really picked it up. This week it becomes more mature.
Firstly, files can now have multiple notes, each with their own name. Existing notes will get the simple default name of 'notes'. Also, files that have notes now show a notes icon on their thumbnails and in the media viewer top-right hover window. Clicking the hover window icon quickly opens the edit notes dialog.
And to find files with notes, there is now 'system:number of notes'. Its panel has a couple of quick buttons for finding files with any notes or no notes. Notes are stored in a faster and more space-efficient way, so this should work fairly quickly for most situations.
The next step here is to add note parsing to the downloader system. The main thought is to get artist post text and similar imported with nice 'blah gallery artist comment' style names as you download files. I had hoped to figure something out for this this week, but I didn't have time to do it properly, so I focused on polishing the above. I would like to fold parsing in in upcoming normal weekly work.
The new storage system for notes also builds a fast search cache for their text. It will not be extremely difficult to add some sort of 'system:note has text xxxxx' in future as well, if you want to search for parsed mega links or whatever else. Beyond that, I'd love some note 'preview' on the media viewer and comprehensive import/export/Client API support for notes.
unusual tag searches
I am really happy with the new tag autocomplete system. It is so much nicer to work on now it is all unified. This week I fixed a bunch of unusual searches.
A good example is '/f/', which according to hydrus 'special character' rules also matches 'f'. Short text queries like 'f' and '/f/' that do not do full lookups will now match all special character results, including if they have a namespace, so all combinations of 'board:/f/' and '/f/' and 'board:f' and so on will now match 'board:/f/'.
Also, wildcards are now unified. 'Complex' wildcards that have inserted *, like 's*s ara*', are now treated logically the same way as 'simple' ones that only have an * on the end like 'sam*', and should more reliably return all the correct results. They ~also~ support the special character rules as above, so 'int*fies' should match '[intensifies]'.
And wildcard file search predicates, where you might search for 'character:samus* (wildcard search)' to return all files that have any tag that matches that wildcard, now support the special character rules in all cases. When you include special characters in the wildcard text, such as 'e*mple-tag*', that '-' character is now preserved (previously, it was being replaced with space, which is behind-the-scenes hydrus technical trickery).
A new tag search cache supports and speeds up these logical improvements. Users with large databases can expect the first boot into 396 to take a few minutes.
the rest
I fixed the sort in the tag suggestion boxes!
You can now click the 'inbox' icon in a media viewer top-right hover window to archive a file.
The file->shortcuts dialog now has descriptive, sorted names for the reserved shortcut sets.
'system:dimensions' now has some quick-click buttions for common ratios and resolutions.
'system:known url' is better about fetching both www and non-www versions of URLs.
full list
- notes:
- the file notes system is more mature. files now store multiple named notes
- the edit notes ui is now a tabbed window with add/edit_name/delete buttons
- media results now load with their notes, so note access is instant
- thumbnails now show a notes icon when they have notes
- the media viewer top-right area shows a notes icon when the current file has notes
- clicking the media viewer top-right notes icon opens edit notes
- the edit notes menu entry now lists the number of current notes if there are notes
- added a 'system:number of notes' predicate. it has easy 'has/no notes' buttons for quick filtering
- the file notes database table will be updated on update, it shouldn't take long. existing notes will get the default 'notes' name
- duplicate notes now share the same storage space in the database
- in prep for a future search expansion, notes are now cached in the database for fast text search
- in prep for note parsing, wrote a 'note import options' object. it doesn't do anything in the program yet, but it supports multiple note conflict resolutions, note extension detection, and global and specific note renaming
- wrote unit tests for the new note import options
- .
- some tag search stuff:
- hydrus now maintains an internal mapping of direct 'searchable' versions of tags to the tags themselves, which allows it to now do fast exact-match (short search) and complicated wildcard lookups of tags with unusual characters. 'f' and '/f/' will now return '/f/' and 'board:/f/' quickly, 'board:f' and 'board:/f/' will return 'board:/f/' quickly, and 'te*a*' will correctly return 'test-tag'
- it will take a few minutes to regenerate this new cache on update
- complex wildcards like 's*m*' are now treated the same as simple ones like 'sam*' and should match unusual subtag characters in all cases
- wildcard tag file search predicates are now plugged into the new cache, so the search preds '/f/*', 'board:/f/', 'board:/f/ast;', 'b*d:/f/' and 'b*d:/f/*' now all match files with 'board:/f/', as do wildcards that include replacement characters, so the same should be true above for 'f' instead of /f/' in all cases
- new wildcard search preds do not collapse their characters for their presentation string, so 'date:2*-01-01' now renders like that, not 'date:2* 01 01'
- wildcard file search predicates are now faster for simple (just an asterisk on the end) subtag wildcards
- the fts search cache is moved from 'master' to 'caches' db this week, it will take a few moments on update
- the 'repopulate tag search cache' db regen job now repopulates the fts cache, the new 'searchable' cache, and the integer tag cache
- the database repair code now checks for the fts cache and new 'searchable' cache on boot and, if they are missing, warns the user and creates empty tables
- .
- improvements:
- fixed the unsorted tags in tag suggestion boxes
- clicking the inbox icon in the top-right hover window now archives the file
- system:dimensions now has quick buttons for 16:9, 9:16, 4:3, 1:1, 1080p, 720p, and 4k
- system:known url searches are now better about fetching www and non-www urls for the domain or url class
- the edit shortcut sets panel now has nicer english names for reserved shortcut sets, and also sorts them in a more logical way
- you no longer have to be in advanced mode to copy file hashes from thumbnails
- users in advanced mode can copy the internal file_id of files from the thumbnail/viewer copy menus (this is most useful for the client api)
- system num_frames, num_words, and num_notes now display alternate 'has/no xxx' labels when they search for =0 or >0
- you can now search for 0 with system:num_frames
- .
- fixes:
- users who could restore from system tray using the menu but had trouble with clicking _should_ now have better luck with clicking
- fixed some instances where fps could be calculated as 0, which would lead to other problems down the line. now a missing or 0 fps is remapped to 1
- fixed system:framerate for '<' queries
- the status bar cells now get expanded tooltips to describe what they do
- fixed some media result caching code that could in rare cases cause an error in content update processing when the result disappeared from the cache during processing
- the 'hard-replace siblings and parents' button on 'manage tags' now makes a submenu so its actions' long labels show better
- fixed a handful of tables that were not starting sorted
- a variety of credential parse and other server failures that were formerly returning 403 now properly return 400 and 409
- in order to improve default 'open externally' behaviour on Linux/macOS, if the environment variable XDG_DATA_DIRS is not preserved through a hydrus build launch env, hydrus now sets a simple 'default' value for this before running xdg-open
- if the client is booted from a windows shortcut to a built release, the program restart command is slightly more reliable
- .
- misc:
- cleaned up some db update error reporting code, it should now more reliably make an english-friendly popup text box before splurging technical info
- refactored some media object code, cleaned some class definitions, and added typing hints
- misc code cleanup
- the 'getting started' help files now have anchor definitions, so their sections can now be #linked to
- added several links in the 'getting started' help to the user-created video guides here: https://github.com/CuddleBear92/Hydrus-guides thank you for making these!
- added a link to the help for the user-made 'other archiving software' guide here: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/wiki/0-Alternative-Programs-and-Resources#software thank you for making this!
- fixed link to AUR package in the help
- updated cloudscraper in all builds to 1.2.36
- updated windows mpv to a significantly newer dll, it now reports api version 1.108
- included libgpg-error.so.0 in Linux build, which will improve some Linux situations (more reports from Ubuntu 20.04 or others about missing/conflicting .so files are welcome)
next week
Next week is a small jobs week. I want to catch up on bugs, see if I can move notes parsing forward, and also (as part of that) improve our text match/transform tools so complicated texts and filenames are a bit easier to parse. The new IPFS daemon is also broken for hydrus, so I will also be looking at that.
Version 395
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v395/Hydrus.Network.395.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v395.tar.gz
I had a good work week. I fixed a variety of bugs, improved the feel of the suggested tags columns, and made zooming in the media viewer nicer.
suggested tags quality of life
I recently did some work on the different 'suggested tags' panels the manage tags dialog can support. This week I worked on some of the feedback from that pass, hopefully making it a bit easier to get what you want done. As a reminder, you can set these columns up under options->tag suggestions.
The columns are now 'synced' with the current media. They all refresh/repopulate reliably when you move to another media in the media viewer. Also, all columns now hide tags that all the selected media already have, and they update this ~as you edit~, so if both 'favourites' and 'related' suggest 'character:samus aran', and you add that through any means, both lists will remove that tag! Furthermore, if you remove a tag that one of those lists would show, they now re-show that tag instantly, without having to do any new lookup.
Furthermore, the tag lists now try to retain the selection location when you add a tag through the keyboard enter key, so even though a tag you hit enter on will disappear, the focus should stay on the tag above or below. I hope this makes it easier to work with these columns with just the keyboard. As a second remind to keyboard users, you can currently set up some slightly mickey-mouse shortcuts to move focus around here under the 'main_gui' shortcuts, the actions starting 'show_and_focus_manage_tags_…'.
zoom center
When you zoom a file in or out, there is a 'centerpoint' about which the zoom happens. This position 'stays still', while everything else grows or shrinks around it. Until now, this has been hardcoded on the media center, which was sometimes unhelpful when the media had been panned. This centerpoint is now customisable under options->media. It now defaults to the media window centerpoint, which means the image will always grow and shrink around the center of what you are looking at. This makes it much nicer to pan and zoom in the duplicate filter. You can change it back to the media centerpoint, the media top-left corner, or even the current mouse cursor position, for RTS-vidya-like zooming.
fixes
'character:aran' matches 'character:samus aran' again in the Windows build (or anything else running python 3.7). It was a regex library version issue.
The hover windows in the media viewer are now aware of the new shortcuts system in Qt and should pass uncaught regular keyboard shortcuts up to the media viewer. So, if you click the top-right hover to set a rating or the center-right duplicates hover to do a skip, you should now be able to hit custom keyboard shortcuts to add tags or set ratings and so on without having to click back on the media viewer. Where this will not happen are simple things like the taglist on the left eating up/down arrow keys to do list navigation and a handful of Qt-hardcoded shortcuts like tab and space, which can intercept and do panel navigation and currently-focused-button pressing.
If your mouse has back and forward buttons, these should now be caught by the shortcuts system, for those shortcut sets that can currently do mouse events. I assume there is no such thing as a double back-click, but let's see what happens!
The client now tries to avoid large analyze and vacuum database maintenance jobs in the normal maintenance cycle. Too many users in unusual situations have been hit by unreasonably big jobs here, and the benefit is not worth it.
Dialog messages that intercept ok and cancel events, the 'hey, are you sure you want to cancel, there is some uncommitted stuff, yes/no' sort of thing, now have improved logic. They fire off in better order and shouldn't ever bother you twice in one action.
deleted tags overwrite
This is an important bug fix, but it is technical and not a priority for non-advanced users for now.
Some users recently reported some unusual deleted mappings counts on the PTR, not matching that of other users. I also had a job to check that deleted tags were being filtered correctly out of local tag domains, so I went into it this week. To my mild horror, I realised that deleted tags were not being filtered out of tag parsing at all, so any time files were being re-parsed, any previously deleted tags were being overwritten. In the case of tag repositories, deleted tags were still being discarded at the server level (and hence not overwritten for other users), but the overwrite was happening to the local client that did the 'late' reparsing.
I regret this oversight, and I am sorry for the transactions of inconvenience and difficulty this has likely caused, even without us ever realising directly what was going on. The issue is unusual, with numerous technical caveats that make it apply to different users in different ways, but I believe it has not hit most users too much. It is also retroactively fixable for tag repositories.
The parsing is now fixed. Tag import options will now filter out the currently deleted tags from their importee file's parsed tags. If you wish, you can force an overwrite using the tag import options cog menus. Hard drive imports and tag migrations continue to overwrite deleted tags, as these are still considered 'human, manual' actions that can take responsibility for overwriting a deleted tag.
For tag repositories, particularly the PTR, a 'reprocess content' run will fix the historical record, re-deleting things that should be deleted. Advanced users can do this now from the review services panel if they wish. This is an expensive job, so I have not queued it up automatically in today's update. I think I will write a separate targeted maintenance routine for it in the coming weeks.
full list
- some more suggested tags fixes/qol:
- favourite tags now correctly refreshes on new media
- the tag suggestion lists in manage tags now discard current and pending tags that _all_ the current media already have, and all tag suggestion lists update this filter any time the media gets a tag content update! they _should_ update live now
- all tag and predicate taglists now try to move the selection to a 'nice' neigbour when a keyboard enter activation results in the current selection being removed (e.g. as in these tag suggestion lists). the nice selection should be the tag after, before, or at the top of the list, and should make it nicer to keep navigating the list and add tags with your keyboard
- all tag and predicate taglists now try to preserve selection on simple clear-and-set data refreshes
- .
- deleted tags overwrite update:
- due to an unfortunate oversight, until now tag parsing has not filtered out previously deleted tags from the tags it parses and sends to the local database
- as the majority of downloaded files are parsed once per site per user and in a similar time window before manual editing ever occurs, and most non-tag-sibling-eligible bad tags are site specific or not parsed to begin with, and as these undesired tags were not broadcast up to the tag repository, this problem has not been very obvious and I believe has not affected most users too much. this is however a reason why some users who have more recently downloaded many older files are seeing smaller 'deleted mappings' counts on their ptr review panel (and some low quality tags in their db), as they have been re-adding previously deleted tags to their local store
- this has been fixed. tag import options now load the pending importee file's metadata before tags are filtered and discard currently deleted tags from those to be added or pended. this applies to parsed tags, additional tags, and those tags added through special other means, such as from a parent gallery page.
- if you do wish to allow parsed or additional tags to overwrite currently deleted tags for a particular job, the cog icons on the edit tag import options panel now allow you to permit overwrite for either
- tags added via hard drive imports or the migrate tags tool still overwrite deleted tags as before
- as this is a local-only problem, there is thankfully a retroactive fix for this issue for tag repository domains, involving a content reprocess run to re-apply deleted tags. I am not activating this automatically this week as this is a heavy job for the ptr and I need to study the true fallout of the problem more, but I may in future, likely as a smaller and more targeted maintenance job. advanced users can do it now under the ptr's review services panel
- I regret missing this, and I am sorry for any inconvenience. I only discovered it through the serendipity of some users recently reporting unusual deleted counts and a personal item in my todo to check the reliability of deleted mapping filtering for local tag domains–turns out it never got added, and we never specifically noticed, fugg
- there are now unit tests for the improved tag filtering pipeline and both of these new overwrite options
- .
- the rest:
- hydrus can now use several different zoom 'centerpoints' about which to expand and shrink a zooming file. this was previously hardcoded to the center of the media. under options->media, you can now set it to be the media window center (the new default, which feels much nicer after a pan), the mouse cursor, the old media center, or the media top-left corner
- cleaned up the related zoom positioning code, and removed the jarring old re-centering off-screen rescue hack when zooming out to canvas zoom
- added a warning about big zooms to the media options page
- fixed tag autocomplete filtering in python 3.7 so 'character:aran' matches 'character:samus aran' again
- when the hover windows on a media viewer have focus, they _should_ now pass up all options->shortcuts shortcuts to the media viewer
- mouse back/forward buttons _should_ now be supported in the shortcuts system, as much as your OS allows them to work like regular clicks
- fixed a rare crash with the 'clear trash' button
- the client will now not re-analyze tables that have been previously scanned with at least 100k rows in the normal 'soft' maintenance cycle, as this is an expensive operation with limited benefit
- the client will now not vacuum database files greater than 1GB in the normal 'soft' maintenance cycle, as this is an expensive operation with limited benefit
- the new 'cannot vacuum because xxxx' log entry is now only ever printed once per boot. however due to the above change, it likely won't appear in the normal maintenance cycle anyway now
- cleaned up some vacuum code
- reworked the panel system to better test data validity vs 'woah, you sure you want to do this?' tests and generally cleaned and simplified the canok/cancancel/isvalid testing logic for all panels. panels like manage siblings will now not produce two message boxes if you try to ok them on an uncommited pair and then back out of the ok
- refactored the top level window code and improved scrollable panel code typing
- more standalone gui function code refactoring
- fixed a click-selection-test bug when clicking on certain whitespace in certain predicate lists
- the text of the cloudflare-specific error when encountering a captcha page is improved
- cleaned up some tag list menu copy and select code, both the menu labels and the copy action, for unusual tags. the 'copyable tags' fetching code is now flexible and unified for menu and action
- cleaned up the taglist sibling copy code, eliminating the chance of dupes
- fixed a _little_ of the wording on the discard/exclude tag list menu labels for negated predicates, it still feels a bit awkward and I will keep working here
- cleaned up some old media metadata fetching code
- misc import code typing
- misc list/iterable typing improvements
- added some misc media-tag tool code
- unified the tag import options tag filtering pipeline somewhat to deal with the deleted overwrite situation
- improved a debug ui test to no longer need window focus
- misc help cleanup
next week
Next week is a 'medium-sized' jobs week. I would like to finally get to an expansion of the prototype 'file notes' system. I would ideally like multiple named notes, note parsing support (so you can pull an artist comment from a web page), note preview on the media viewer background, and note import/export. This is a ton of work, so I don't expect to get it all done in one week.
Version 394
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v394/Hydrus.Network.394.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v394.tar.gz
I had an ok week mostly cleaning code.
The bug from last week where subscriptions that had incomplete file queues were unable to start is fixed. I apologise, this was a typo that was not caught by my code checker or my testing routine.
autocomplete cleanup
The tag autocomplete code has seen many updates and expansions over time, and the code had become a mess of hardcoded hooks and logical duct-tape. This week I went into it and pulled all the text parsing and results caching code out to single separate dedicated objects. Essentially most of the big changes this week are all behind the scenes, and you can check the changelog for more detailed info, but multiple small bugs and odd situations with autocomplete entry are now fixed.
A user recently contributed code that parses system predicates from text input. Now I have this nice new parsing system, I hope to integrate this new feature soon.
If you encounter new bugs or certain complicated wildcard lookups that do not work, please report them–it is now much easier for me to fix this stuff and support new search types.
environment changes
If you patch my source code, please note that the source structure has changed. The single flat directory of py files is now split into some basic subdirectories. This is one more step forward in getting the structure more manageable, and I expect to do more in future cleanup weeks.
full list
- autocomplete cleanup:
- the text you type into tag autocomplete is now parsed in a unified object. all the variants of empty text, invalid text, valid text, namespace text, and wildcard text are all tested and fetched in one simple location with better code
- autocomplete results caching is now a unified object that tracks and filters results in one location. wildcard searches are now never cached by accident, and switching from tag cache to system predicate cache and to non-initialised cache is instant and more reliable
- when an autocomplete, either in a search page or a context that manages tags, has results include multiple sibling variants of the typed text, they are now all elevated to the top of the list. the ideal is at the top, the entered text is next, and any known siblings follow
- the search character 'collapse' that ensures quote marks and hyphens and other odd characters are unified across tags now applies uniformly to all non-complicated-wildcard search tags, with namespace not collapsed and subtag always collapsed
- when entering an explicit wildcard search, both strict and autocomplete versions (whether they end with an asterisk) are now displayed
- the way tag results are filtered is now more accurate for some unusual wildcards
- it is now more difficult to slip cpu-killer search tags (weird asterisk combinations) through
- the quick-broadcast that happens when the user hits enter before any results have started loading now uses the unified object and chooses a safer and more reliable broadcast value. the test whether to do the quick-broadcast is also more reliable, particularly in unusual situations where a recent search was cancelled or delayed. note that for many users, the cache and search tech is fast enough that this very rarely triggers
- searching with a wildcard below the autocomplete threshold can no longer trigger a full search, nor an invalid exact-text search
- namespace count merging is now unified across db tag fetches and media fetches
- include current/pending buttons now filter down to media-based tag autocomplete counts
- namespace tag autocomplete queries will no longer show up some unusual siblings below the 'anything' tag
- deleted a whole bunch of old a/c and caching code
- added comprehensive unit tests for the new parsed autocomplete text object
- added comprehensive unit tests for the new predicate results cache object
- .
- the rest:
- fixed a stupid typo bug in the new domain checking code that was stopping subscriptions with incomplete file queues from starting. I apologise for this
- network error responses 502 (Bad Gateway) and 503 (Service Unavailable) are now treated as a retryable. the 503 is assuming it is not a CF challenge page. if they fail all retries, they are considered a network infrastructure error
- all other misc 5xx http responses are now treated as instant network infrastructure errors and will be logged in the new domain health tracker
- the exit splash screen now opens a bit earlier, so you now shouldn't have any momentary uncertainty where no windows are open
- clients that start minimised to system tray _should_ be better about restoring splitter positions on first show
- the various 'management panels', the panels on the left of main gui pages, now have smaller minimum width where available. the gallery and watcher panels are still the widest, which is a limitation of the current list tech. when it gets better column sizing code and selection memory, this will improve
- fixed an issue loading gifs with some OpenCV versions
- brushed up some running from source help
- deleted the Py2To3 script that attempts to detect a legacy python 2 install
- improved all the gui files' import order
- cleaned up and refactored some subscription code
- added a bunch of type hints to edit panel code
- misc code cleanup
- .
- environment updates:
- did second step of hydrus project structure improvement–now the project is split into subdirectories for core/client/server/misc and some client subdirs. work here will continue
- linux build gets some new libraries, cv is up to 4.2.0
- it isn't important, but hydrus is now built in python rather than directly from command line. my build scripts now include cloudscraper and the new hydrus source code tree in the build as they are, rather than hardcoded copying
next week
I would like to catch up on small jobs and bug fixes. I feel pretty exhausted due to IRL stuff, so I want to take my foot off the throttle a little. I hope you are hanging in there and staying healthy.
Version 393
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v393/Hydrus.Network.393.-.macOS.-.App.dmg
linux
EDIT: This release had a hotfix two hours after initial release to fix the CloudFlare code. If you got it early, please redownload. The links are the same, the build is new.
I had a good week catching up on messages and small jobs. There is also a (hopefully) neat prototype solution for some cloudflare issues.
cloudflare and network
CloudFlare hosts content for many sites online. They have a variety of anti-DDoS tech, normally a variation on a 'is this a web browser?' test, that sometimes stops hydrus from downloading. If you have seen unexplainable 503 errors on a site that works ok in your browser, this may have been it.
This week, I am adding a library, cloudscraper, to my network engine to try to solve this. Now hydrus will attempt to detect CF challenge pages when they are downloaded and pass them off to this new library, which attempts to solve the javascript challenge (the part where a CF site sometimes says 'this process may take five seconds…', just as a browser would, and then copies the solved cookies back into the hydrus network session.
This first version can only solve the simple javascript challenges. It cannot do the more serious captcha tests (which usually appear when a site is under active attack) yet, but this is a possible future expansion. I would appreciate any feedback from users who have had bad CF problems. If a page cannot be solved, or if your entire IP range had been flat-out blocked, hydrus should now attach new CF-appropriate error messages.
For users who run from source, this library is optional. You can get it through pip, 'cloudscraper'.
Additionally, I am rolling in the very first basic version of 'this domain is having trouble' tech to my network engine. Now, if you keep having connection errors or CF issues with a site, hydrus recognises this and slows down access for a bit. The default is that no new network job will start as long as its site has had three serious errors in the past ten minutes, but you can edit this rule, including turning it off completely, under options->connection. Subscriptions will also try to wave off sites having trouble, just as if you had hit bandwidth limits. I expect to extend this system significantly in future, particularly by adding UI to see and manage current domain status.
the rest
The Windows build of hydrus is now on a newer version of python, 3.7 instead of 3.6. This makes for a variety of small background improvements all over the place, and some updated libraries, but may also introduce an odd bug here or there in more rarely used hydrus systems. Let me know if you run into trouble!
I did another round of file search optimisations on searches that do not have tag search predicates, such as bare system:rating or system:inbox queries.
Thanks to some excellent work by a user, there is another new DA parser. This one gets full-size images, video, flash, and even pdf! The only proviso is that it needs to be logged in to DA to get most content, otherwise it 404s. I believe the hydrus DA login script works, but if you have trouble with it, Hydrus Companion is always a good fallback.
The blacklist in tag import options gets a bit of work this week. It now tests unnamespaced rules against namespaced tags, so if you have a rule for 'metroid', it will also block 'series:metroid'. This is a special rule just for the tag import options blacklist (it also tests all siblings of the test tag), and to reflect it, the tag filter/blacklist edit panel now has a second 'test' text to show how the test input applies to a TIO blacklist, as opposed to a regular tag filter.
This is a small thing, but if you are mostly a keyboard user, the mouse now hides on the media viewer without having to be moved!
If you have had crashes trying to open a file or directory picker dialog, try the new BUGFIX checkbox under options->gui. It tries to use a different style of dialog that may be better for you.
full list
- cloudflare and network:
- the hydrus client now has an experimental hook to the cloudscraper module, which is now an optional pip module for source users and included in all built releases. if a CF challenge page is downloaded, hydrus attempts to detect and solve it with cloudscraper and save the CF cookies back to the session before reattempting the request. all feedback on this working/breaking irl would be welcome. current expectation for this prototype is it can pass the basic 'wait five seconds' javascript challenge, but only a handful of the more complicated captcha ones
- if a CF challenge page is not solvable, the respective fail reason for that URL will be labelled appropriately about CloudFlare and have more technical information
- .
- the hydrus network engine now has the capability to remember recent serious network infrastructure errors (no connection, unsolvable cloudflare problem, etc..) on a per domain basis. if many serious errors have happened on a domain, new jobs will now wait until they are clear. this defaults to three or more such errors in the past ten minutes, and is configurable (and disableable) under options->connection. this will be built out to a flexible system in future, with per-domain options+status ui to see what's going on and actions to scrub delays
- basically, if a server or your internet connection goes down, hydrus now throttles down to limit the damage
- subscriptions now test if a domain is ok in order to decide whether they can start or continue file work, just like with bandwidth
- serverside bandwidth alerts (429 or 509) are now classified as network infrastructure errors
- I expect this system will need more tuning
- .
- the hydrus downloader system now recognises when an expected parseable document is actually an importable file. when this is true, the file is imported. this hopefully solves the situation where a site may deliver a post url or a file
- .
- the rest:
- the windows build of hydrus is now in python 3.7.6, up from 3.6. this rolls in a host of small improvements, including to network stability and security (e.g. TLS 1.3), and possibly a couple of new bugs in more unusual hydrus systems
- similarly, all the windows libraries are now their latest versions. opencv is now 4.2
- greatly sped up several file searches that include no tags such as bare system:rating, most system file metadata predicates, or bare system:inbox, when the result size is much smaller than the total number of files in the file domain
- thanks to some excellent work by a user, the Deviant Art downloader gets another pass–it can now get high res versions of images where they are available, and video, and flash, and pdf! the only proviso is that you need to be logged in to DA to get most content, otherwise you get 404. the current hydrus DA login script _seems_ to work ok
- tag import options blacklists now test unnamespaced rules against namespaced tags. so if you blacklist 'metroid', a 'series:metroid' will be caught and the blacklist veto signal sent. this can be escaped with the 'advanced' exception panel, which now permits you to add 'redundant' rules
- the edit tag filter panel now explains the blacklist rules explicitly and has a second 'test' green/red text to display test results for a tag import options blacklist, with the new sibling and namespace check
- added some unit tests to test the new tag import options blacklist namespace rule
- when 'default' tag import options are set, the edit panel now hides the per-service options, rather the the previous disable
- the system tray icon now destroys itself when no longer needed, rather than hiding itself. it should now be more reliable in OSes that do not support system tray icon hide/show. if your OS still doesn't get rid of them, and you get a whole row of them, I recommend just leaving it always on
- the system tray now has a tooltip with the main hydrus title and pause statuses
- the timer that hides the mouse on the media viewer is now fired off when the window first opens (previously it would only initiate on the first mouse move over the window), so users who navigate mostly by keyboard should now see their cursors nicely hide on their own
- added some semi-hacky import/export/duplicate buttons to edit shortcuts. I'll keep working on this, it'd be nice to have import/export for whole shortcut sets
- added a semi-hacky duplicate button to the 'manage http headers' dialog
- the 'clear' recent tag suggestions button is now wrapped in a yes/no dialog
- a new checkbox under options->gui now lets you set it so when new cookies are sent from the API, or cookies are cleared, a popup message summarises the change. the popup dismisses itself after five seconds
- the client api now also returns 'ext' on /get_files/file_metadata calls, just as a simpler alternative if the 'mime' is a pain
- fixed a bug when petitioning tags through the client api, with or without reasons
- fixed an error where subscriptions that somehow held invalid URLs would not be able to predict some bandwidth stuff, which would not allow the edit subs dialog to open
- the string transformation dialog's step subdialog is now ok with example strings that are bytes. even then, this str/bytes dichotomy is an old artifact of python 2 and I will likely clean it up sometime so string transformers (and downloaders) only ever work utf-8 and hashes just work off utf-8 hex
- added a BUGFIX checkbox to options->gui that tells the UI to use Qt file/directory picker dialogs, instead of the native OS one. users who have crashes on file selection are encouraged to try this out
- updated running from source help with cloudscraper, a new pip masterline, and some windows venv info
- the 'import with tags' button on 'import files' dialog gets another rename for new users, this time to 'add tags before the import >>'. it also gets a tooltip
- handled an unusual rare error that could occur when switching out a media player inside a media viewer, perhaps during media viewer shutdown
next week
Next week is a cleanup week. I'd like to go into the tag autocomplete code and clean up how the input text is parsed and how results are cached and filtered. Otherwise, I'll do more wx->Qt cleanup and catch up on some small jobs.
Version 392
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v392/Hydrus.Network.392.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v392.tar.gz
I had a mixed week, but I am happy with my work. Some large database jobs are faster and tag autocomplete results have better sibling support.
tag sibling database cache
I was not able to get as much done here as I had hoped, but I was able to finish the first step and fix some sibling bugs. The client now keeps a cache for fast sibling lookups, which will be a great tool going forward.
The tag autocomplete system now uses this new cache. When you type in the start of a tag, possible matches' siblings are added to the results list, even if they do not strictly match what you typed. These siblings used to be computed on the fly, but now it is looked up using fast pre-generated data. I have also improved the accuracy of the search and added 'pending' siblings in.
As I was working here, I also fixed some sibling presentation and search bugs. If you discovered that a 'series:blah' search was delivering some unexpected 'cosplay:blah' results, this should now be fixed. There is still more to do here, which I hope to naturally fix as I replace more old systems with nice new caches.
There are two next steps here: first to write a 'what file tags should look like with siblings collapsed' cache, which will speed up tag search and file load and also eliminate many of the uncertain counts you get with autocomplete results like 'character:samus aran (500-700)'. Second is to write a full-featured 'which siblings apply to this service?' options system, so you can say 'ignore the ptr's siblings, use mine instead' and other clever combinations, rather than the current 'always apply all to all, or one to one'.
the rest
e621 and sankaku get revised default downloaders, fixed for getting 'rating' and 'genre' tags respectively.
The system tray icon got some bug fixes. It is also more stable under non-Windows, but doesn't seem to hide reliably yet.
The hydrus server and the client's API and booru now try to host their services on IPv6 as well as IPv4. This is new stuff to me, so let me know if you come across any problems!
full list
- db-level tag sibling cache:
- the hydrus client db now maintains a fast cache of current+pending tag-to-ideal-tag sibling relationships. it works for specific services and 'all known tags'. this is a nice tool and the first step in having a proper hard-baked siblings mappings cache
- the new sibling cache can be regenerated under _database->regenerate_. the 'autocomplete cache' entry under that menu is also renamed to the now more appropriate 'tag mappings cache'
- the db repair system can regenerate this new cache if any part is missing on boot
- the lookup that finds tag sibling matches for autocomplete uses this and is now faster, specific to the searched service, more accurate about status, and now includes pending siblings
- wrote a new unified object to manage a collection of tag siblings, it is now in use at the db level
- as I continue to develop this new fast tech, the old 'apply all sibs to all services' option, which was always buggy, may sometimes not apply in it. I will ultimately replace it with a fuller per-service choice system that will work quickly and properly and in the same unified way
- fixed a bug where only one local tag service's siblings would be matched at the ui level when looking at 'all known tags'
- fixed a bug in the file search code where searching for a tag that had an unnamespaced sibling going to it would result in searching for all possible namespaces of that sibling (e.g. searching for 'character:samus aran' when 'samus aran'->'character:samus aran' sibling existed would result in effectively 'anything:samus aran')
- when tag services are deleted, they are now better about cleaning up their siblings and parents quickly
- optimised some tag and hash id->value database cache population routines to improve performance for large queries (e.g. when fetching all the tag parents/siblings on boot). also these caches are now larger, 100k instead of 25k
- all cache regen code now forces an immediate analyze of the new tables to speed up imminent access
- .
- the rest:
- updated the default e621 file page parser to get rating tags again (looks like their markup just changed again)
- updated the default sankaku file page parser to get their recently redefined 'genre' tags
- in edit subscriptions, the 'overwrite tag import options/check options' actions now initialise their dialogs with the current value for the first subscription, rather than the global program default
- in the edit subscription panel, the checker options button is moved down to the file/tag import options
- when not in advanced mode, the edit tag import options panel now has some red-text at the top to reinforce to new users that they should generally use the defaults
- the tag import options blacklist now secondarily checks against all known siblings of the parsed tags, rather than just the 'collapsed' ideal siblings
- subscriptions are now more aggressive about clearing out old urls from their file import caches–instead of clearing the 251st url after it has aged twice the death period, now they use just one DP. also, checkers with static checker timings will use five times that check period as DP if that is smaller. static checkers, or those that never die, will use a flat value of six months as DP if that is smaller
- moved a bunch of the debug 'data actions' to a new 'memory actions' menu
- significantly reduced how often the system tray regenerates its menu, which seems to improve stability
- fixed an issue where guis that were maximised before a minimise were restoring from a system tray icon click to normal view
- double-clicking the system tray when the ui is hidden should no longer do a fast show/hide
- fixed an issue where if the gui was minimised, the main animation timer would not run for other windows (e.g. a separate media viewer)
- improved ui shown/hidden tracking logic for the new system tray icon for different OSes
- fixed the 'refresh_page_of_pages_pages' shortcut action, which had faulty old wx code in it
- fixed a wx->Qt bug where modal popups that cannot be cancelled, and thus pop up a 'sorry, you can't dismiss this' text when you try to close them, were nonetheless still closing afterwards
- the hydrus client and server now attempt to listen their servers on both IPv4 and IPv6, failing gracefully if IPv6 is not available
- the 'is this a localhost request?' check now understands IPv6 localhost (::1 or ::ffff:127.0.0.1)
- may have solved a 100% cpu repaint issue with the a/c dropdown in some qt environments
- added info to installing help about Windows N and clean installs
- misc media viewer wx->Qt code cleanup
- misc code cleanup
- .
- experimental hellzone, be wary ye scabs:
- added an experimental 'sub-gallery url' url content type to the parsing and downloading system. this url is queued into the gallery log even if the primary gallery page found no file/post urls, and is meant for galleries that link to galleries. not yet ready for primetime, but feedback would be appreciated
- added an experimental ui-hang relief mode, activated under _help->debug->data actions->db ui-hang relief mode_, which _should_ stop the ui hanging in unusual long-time ui-synchronous db jobs. it may cause other problems, so it is default off. it also prints begin/end statements to log for additional info. users who experience ui hang due to db job processing time are invited to play with this mode and report back results
next week
Next week is a small jobs week. I fell behind on messages this week, so my main job is to catch up on those and then to clear out my bugs queue.
Version 391
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v391/Hydrus.Network.391.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v391.tar.gz
I had a great week. The client now has 'minimise to system tray' support, you can search by framerate and 'number of frames', and a bunch of bugs are fixed.
system tray
This first version is buggy/crashy under non-Windows, so I have disabled it for non-advanced mode non-Windows users for now. The icon menu doesn't work well for non-Windows, but double-clicking seems ok. I will keep working on this.
The hydrus client can now make a system tray icon for those OSes that support it. You can customise it under the new options->system tray panel. You can set the client to minimise to and/or close to system tray, and you can set the icon to always display if you like. There is also a new 'hide_to_system_tray' action under the 'globals' shortcut set.
Clicking on the system tray icon will raise a 'shown' client to the front or show a hidden client. Double-clicking or middle-clicking will do hide/show. The icon also has a menu to show/hide the client, pause/unpause subscriptions or network traffic, and exit the program.
You can also set the client to boot minimised into the system tray, but there may be layout issues once you do show the client–please let me know if you get any!
frames
You can now search for file framerate or 'number of frames' under an expanded 'system:duration' search predicate. It is difficult with the data hydrus currently knows to get very precise framerate, so system:framerate searches +/-5% of what you set. You can't differentiate between 30fps and 29.97fps yet.
You can now sort by number of frames.
Video framerate is now shown in the 'file info summary' text that displays on the status bar, thumbnail menu, and media viewer.
the rest
There are new fixed default danbooru parsers, and an e621 login script, both thanks to submissions by users.
I went over the tag autocomplete fetching code, cleaning up old areas and improving the timing and caching logic. It should be snappier, less flickery, and the bug where system predicates sometimes do not re-load after entering a search tag should be fixed. I expect to revisit this in three weeks for the next 'cleanup' week to further clean this semi-hellcode.
I also cleaned up some old wx->Qt stuff related to thumbnail sort. I believe I fixed the weird issue where certain pages (those that had just loaded a favourite search seemed to be one) would not accept certain sorts (sort by time imported was common here, I think because that is default secondary sort), and some collect/sort interactions are also faster and more reliable.
The tag suggestion boxes in manage tags now only do 'add' operations when you choose their suggestions (i.e. they won't remove an existing current/pending tag). They also remove those tags from their list when you do so. See if you like the feel of this new workflow. Also, a bug where some of these suggestion boxes would not acknowledge new pending tags (e.g. not using them for subsequent 'related' tags searches) after moving to a next/previous media in a manage tags off the media viewer is fixed.
If you ever got screwy URLs in your client (invalid garbage like 'tdunrotduenoh'), the program should be much saner about filtering and displaying and editing these. The Client API no longer accepts invalid URLs when associating URLs.
full list
- system tray icon:
- hydrus now can now make a system tray icon for those OSes that support it. it can be buggy/crashy under non-Windows, where it gets some warning labels
- under the new options->system tray page, you can set whether to show the system tray all the time, minimise the main gui to system tray, close-button the main gui to system tray, and start the program minimised to the system tray
- right-clicking the icon brings up a menu to show/hide the ui, pause/unpause network traffic or subscriptions, and to exit hydrus
- the main file menu now has an option to minimise to system tray
- double-clicking or middle-clicking the icon will show/hide the whole hydrus ui as long as there are no dialogs open
- clicking it will restore the main gui from minimise or raise it to the front
- on an ui hide, the current preview window will be blanked and media viewers will be paused, so any ongoing noise/cpu from them should stop
- a new 'global' shortcut 'hide_to_system_tray' is now available
- starting the client minimised may have some layout issues on first show–I particularly had to fix splitter layouts–please report any more you discover
- .
- framerate and num frames:
- system:framerate search added to system:duration panel. precise framerate is tricky with current hydrus info, so it searches +/- 5% of a given value
- system:number of frames added to system:duration panel
- sort by number of frames added
- duration/framerate/num frames sort moved to their own 'duration' submenu
- framerate added to generic media metadata summary string (which appears in status bar and media viewer, etc…). precise framerate is tricky with current hydrus info, so it is rounded to the nearest integer
- .
- the rest:
- rolling in new danbooru file page parsers that should fix file downloads, thank you to a user for the submission
- rolling in a e621 login script, thank you to a user for the submission
- gave tag autocomplete results fetch code a pass, cleaning up several instances of incorrect or inefficient timing and caching logic and I believe fixing the issue where system preds would sometimes not be loaded after entering a tag
- improved reliability of autocomplete dropdown hiding on background pages (some edge cases where these could still hang around _should_ be fixed)
- improved 'hide' tests in several parts of the program related to the new system tray icon, which should help some other cases–e.g. weird shutdowns now _shouldn't_ ever leave a bunch of floating popup messages
- fixed a bug where pages set to open with all known files/tags domains, which is not supported, was incorrectly substituting tag domain with a file domain, which is even more not supported
- cleaned up some sort code–I believe this has fixed the odd issue where a 'time imported' sort would not work on some pages (such as one loaded from a favourite search)
- fixed the 'related' tag suggestion box not knowing about new pending tags added in a manage tags dialog open on a media viewer after next/previous media transitions while the dialog is open. also it and the file lookup's lists now clear when a new lookup starts
- the tag suggestion boxes are now add-only and remove what you add as you add them! let me know if this feels nice or not!
- the splash window now has a different 'booting/exiting' window title, if you would like to hook it with a window manager
- went over all the 'prep url for display', 'filter urls', and 'normalise url' requests across the program to deal with invalid url (e.g. garbled text) better
- you can now no longer add invalid urls via the client api associate_url call–you'll get 400 instead
- cleaned some thumbnail selection and rendering code, particularly fixing some edge case 'where that media go?' issues where collect-by calls happen during thumbnail waterfalls and so on
- cleaned up some page file domain setting code and misc page management code
- improved accuracy of rendered image cache memory footprint calculations
next week
Next week is a 'medium size' job week. I want to get started on a database-level tag siblings cache, which will improve siblings speed and accuracy across the program. I do not think I will have time to finish it, but I know I can move all this forward.
Version 390
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v390/Hydrus.Network.390.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v390.tar.gz
I had a good week. I spent most of my time cleaning code, but there are a couple of important bug fixes as well.
highlights
Several users have reported that potential duplicate pairs could sometimes come up again for files that had previous been set not related/false positive. This particularly hit users who had reset their potential dupes and re-searched. I had trouble reproducing this, but thanks to the latest user report I was and I traced down the problem. It should now be fixed. If you have been hit by this, please reset your potential duplicates (hit the 'cog' icon on the duplicates page), and then re-search–the bad pairs should not be re-added again.
Several tag autocomplete bugs are fixed today, all for unusual search types. Searching 'hyphenated-namespace:' now gives the correct 'anything' search. 'anything' namespace searches now work for namespaces that have asterisk wildcards. Autocompletes in manage tag dialogs ('write' a/cs) now match results with UPPERCASE inputs. Several more cpu-killer searches (usually some variant of 'get everything' through a clever wildcard) are now impossible to add.
Also, I have fixed the recently borked layout of the 'cancel search' button, which pops up when a page search takes more than three seconds, and fixed a couple of instances where a search page could get stuck in the 'Loading…' state until it was poked.
full list
- fixed a bug that was causing potential duplicates to be sometimes re-added between media groups that were previously set as false positive/not related. I apologise for the inconvenience this bug has caused. if you were hit by this, please reset your potential duplicate pairs (hit the cog button on the dupes page) and re-search, and the bad pairs should not be re-added again
- fixed an issue where tag autocomplete entry in the form 'namespace-blah:' was replacing the hyphen or other 'collapsable' character to space, which then was not searching correctly for the _anything_ namespace search
- 'namespace:anything' searches now work when the namespace itself has a wildcard
- fixed 'write' autocompletes not matching inputs with UPPERCASE letters
- fixed adding tags that start with a colon (e.g. ":D") in 'write' autocompletes
- it should now impossible to enter some 'kill my cpu' queries into tag autocomplete, such as '[asterisk]:anything', even if accidentally entered through the fast-add system
- the 'cancel search' stop button that appears after a search takes three seconds is back to being neatly embedded beside the tag autocomplete input box
- hitting the cancel search button now clears the non-interactable 'Loading' thumbnail media page (with its misleading 'Loading…' statusbar) and returns you to an empty thumbnail page
- loading a favourite search with non-immediate search no longer loads the 'loading…' page. it also saves that new non-immediate status to the page session more reliably
- reworked my linux build environment (pyinstaller=3.5, virtualenv=16.1) so that you can launch the built exe using a symlink
- rolling out a first version of a requirements.txt, any feedback would be appreciated
- rolling out another version of the derpibooru file page parser that no longer duplicates namespaced tags as unnamespaced, thank you to the user who submitted this
- .
- boring stuff:
- .
- moving old pubsub system to Qt signals:
- all the 'refresh query' calls that do changes to the current file search across the program
- the current file and tag domain update calls for search pages
- the clear/set file focus calls when launching and exiting the media viewer browser or archive/delete filter
- the way thumbnails send the current focused media to the preview media window
- the way widgets with shortcuts-based tooltips are notified to update those tooltips when shortcuts change
- the way a thumbgrid sends the current tags to be displayed in the 'selection tags' list
- the way a thumbgrid adds newly imported files' tags to the 'selection tags' list as they fade in
- the way the 'searching/waiting' search button is flipped on and off by shortcut. btw what should be the correct name + label for this button? should it really be an icon?
- .
- cleanup:
- NOTE: the 'include' folder is renamed this week to 'hydrus'. if you have source patches, please update. as I further disentagle code in future, hydrus will ultimately move to typical nested folder/module structure
- decoupled the shortcuts edit ui code from the controller and db, unified how shortcuts are edited, and eliminated db wait when booting shortcuts editing
- decoupled the shortcuts manager from the controller, cleaned all the code, and moved to a nicer reference with proper typing hints
- refactored the frame and media controls of clientguicanvas into separate files
- renamed the hoverframes file to canvashoverframes and updated its classnames to 'canvas' rather than the old 'fullscreen'
- fixed two wx->Qt typo artifacts in the login script edit ui
- reduced some occasional idle memory bloat of clients that have large subscriptions
- cleaned up how media-based taglists are appended with new media
- removed some old booru object update code
- some misc setmedia/clearmedia cleanup
- misc search code cleanup
- misc typing hints to clear up pylint confusion
- misc tag autocomplete code cleanup
- misc 'global' variable cleanup
- misc gui code refactoring, cleanup and typing
next week
Next week is a 'small jobs' week. I will catch up with little jobs, perhaps 'search by framerate' or adding select/remove for filetypes. I definitely want to add close/minimise to system tray, which keeps being put off
Version 389
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v389/Hydrus.Network.389.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v389.tar.gz
I had a great week. I fixed many small bugs, added some quality of life, and am rolling in updated downloaders for e621 and Deviant Art.
downloaders
Unfortunately, last week's e621 downloader was not getting md5 hashes reliably. This coupled with the URL format change often meant an increased bandwidth load for the subscriptions that were trying to re-find their place. Thankfully, another user has provided a more accurate one that rolls into today's update. It also pulls rating tags. If you paused your e621 subs, please resume them again.
Deviant Art have also been through a layout change recently. The existing file page parser was getting a little shaky, so I wrote a new one this week that uses their API. The default hydrus DA downloader is now wholly API-driven, so it should all be more resistant to any further changes on their frontend. If you would like to search for nsfw DA content, the default DA login still seems to work, but Hydrus Companion is also an excellent option.
For advanced users: Thanks to user contribution, there are two newer default file lookup scripts, an updated iqdb-based danbooru lookup and one for direct danbooru lookup using md5 hash.
For advanced users: The parser String Transformation system now has a 'url percent encoding' option for the encode/decode types. This lets you convert complicated text to a URL-parameter-friendly percent-encoded string, or vice versa, if you need to do something like a clever API URL transformation.
the rest
The way the new mpv.conf file was stored was failing for users with portable installs. This file is now cached in the db directory alongside your other user-configurable data. All users will reset to using the hydrus default mpv conf this week (now just in that new location), so if you have been using a different one, please re-set it just this once after you update.
Hitting Escape on a non-empty tag autocomplete text box will now clear that box. This sounds like a very small thing, but give it a go–since shift+home and ctrl+a are diverted to the list below when you have results, it feels nice to have a quick shortcut to reset. Hitting Escape on an empty tag input box continues to cause the box to first lose focus (hiding any floating dropdown list), and then, in the manage tags dialog, closes the dialog.
Short-duration media (e.g. a gif that only lasts 0.5s) now loop properly in slideshows with a longer slideshow period.
The new 'favourite searches' system now has a new 'empty page' default, which updating users will also receive. It is just a nice way to clear a page, but feel free to delete it if you do not like or need it.
The collect-by dropdown now has checkboxes again in Fusion style! It should be fixed for any other styles that had problems, but let me know if you get anything odd now.
A bug where pages created by the new 'duplicate page' command would accidentally share some data updates with their original until a client restart has been fixed.
full list
- downloaders:
- the e621 file page parser is updated again, thanks to a user's contribution. this one gets md5 and file url more reliably, and also gets rating tag
- added a 'e621 file page (old format)' url class to help match and search for files downloaded with the old format. please be aware there is no good solution to auto-convert old urls to a new format yet, so this connection does not (yet) solve the old/new comparison test
- updated deviant art file post parser to use their json api. this should be more resilient to their current layout changes
- the nijie.info login script appears no longer to function. as with exhentai last week, it has been removed to make it easier to log in with hydrus companion. please use hydrus companion if you would like to log into nijie.info
- updated file lookup scripts for 'iqdb danbooru' and 'danbooru md5' thanks to a user's contribution
- .
- the rest:
- the way the mpv.conf works changes this week. it is now correctly fully portable, stored in the db directory beside the .db files. if this file does not exist, the 'default' as stored under the install_dir/static/mpv-conf folder will auto-populate it. if you have been using a non-default mpv conf, please re-set it one time after update, and you should be good
- the code that loads mpv.conf is now more graceful on 'missing file' errors, which now means when both the db conf and the default conf are missing
- hitting escape on a tag autocomplete input that has text will now clear that text! note that hitting escape on an _empty_ a/c input will still do 'lose focus' and then 'close dialog'
- updated the slideshow logic so that if a media with duration has a shorter duration than the slideshow duration (e.g. a gif that lasts 0.5s on a 10s slideshow), the media will keep looping until the duration is up. media that has duration longer than the slideshow time will continue to play through once completely, delaying slideshow progression and then stopping promptly when it has finished
- the string transformation system now allows 'url percent encoding' under the encode/decode type!
- fixed the 'only add existing tags' filter in the tag import options, which was denying all the tested tags. it seems to have been hit by a typo in the last three months
- the 'favourite searches' defaults now include an 'empty page' entry, which is a convenient way to simply clear a page. all users will also get this on update, feel free to delete if you don't like/need it
- opening a new search page from a tag or an active search predicate ('open a new search page for…' or middle-click) now copies the file service (e.g. looking at trash) from the original page
- opening a new search page in the 'all known files' file domain when the tag domain should be 'all known tags' (a currently unsupported combination) now coerces the tag domain to 'all local tags'
- checkboxes should now appear again on the collect-by dropdown in Fusion (and hopefully any other) style
- fixed an issue where entering 'namespace:*' explicitly would show the much less efficient wildcard search rather than the efficient 'anything' namespace search
- fixed an issue where wildcard search could include multiple asterisks in a row
- fixed an issue with page duplication where the main management object was not being duplicated properly until a session reload, meaning the two pages would sometimes share signals and changes
- an old wx delayed hide/show performance hack is removed, making the floating autocomplete dropdown now update more smoothly to resize or move requests, such as when the main gui window is dragged
- the program base installation directory is now calculated more accurately, both when running from source and the frozen build, and when launched using a symlink
- install dir and db dir are now specified in the help->about window
- the petition page content checkbox list now has a taller minimum height
- improved error text reporting in hydrus service login failure, hydrus service delay reason-setting, and all 'cancelled' errors across the program
- the review services panel now has elided… text. when unusually long errors propagate up to its status texts, it now won't suddenly jump to 2,000 pixels wide. full text appears in tooltips
- code refactoring: the tag autocomplete input now now takes responsibility for the active predicate list above it
- refactored some tag lists and added typing hints to improve how current page predicates are determined
- did some prep work for tag filters supporting wildcards, but it isn't ready yet
- cleaned up some wx->Qt data fetching code
- misc code cleanup
next week
Next week is a 'cleanup' week. I want to focus on more wx->Qt code cleanup, removing more old UI hacks, improving layouts, and smoothing out how some behind-the-scenes updates are sent across the program.
I also want to have a deeper look into a bug in the duplicates system where the 'not related' relationship is not sticking when some duplicate search information is reset. I previously could not reproduce this, but I have had more reports of it in the past couple of weeks, so I expect to write some more automated test code to root it out.
Version 388
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v388.tar.gz
I had a great week. The client can now save and load searches.
favourite searches
Every tag autocomplete input text box that searches for files–the most obvious being the one on normal search pages–now has a star icon button beside it. Click this, and you get a menu to save your current search, manage your saved searches, or load up one that is saved!
Currently, the saved information is the list of search terms (the tags and system predicates), the current file and tag domains (e.g. my files/all known tags), whether the system is 'searching immediately' or waiting, and optionally the current sort and collect.
Each saved search has a name and optionally a folder, for easy grouping. I have started all clients with an example inbox processing search, just as an example.
If you, like me, keep ten or fifteen empty search pages open (for me it is mostly different creator+inbox searches), I hope this system lets you collapse it down, making for a lighter and simpler session. It should also help for more unusual workflows like duplicate filtering and even file maintenance jobs.
There may be a couple of bugs in the system, something like a collect-by not being saved or updated correctly. Let me know how you get on!
the rest
A note from the users managing Hydrus Companion: The Chrome Web Store release of Hydrus Companion is no longer available due to publishing issues. If you have been using it in the past, please install the extension manually as outlined here instead: https://gitgud.io/prkc/hydrus-companion
e621 changed their site format, breaking hydrus's default downloaders. This is now updated, thanks to a user's contribution. The URL format has changed, so unfortunately your subscriptions will take longer (but will not redownload files) and note they have hit their periodic limits on their next cycle, which you can ignore. If you use an e621 login with hydrus, you may also want to check your account-based tag blacklist on the site, as I have heard this has also changed.
Hydrus's default mpv configuration file now has some normalisation that seems to work well.
If your subscriptions are paused, or if all network traffic is paused, this is now noted in the status bar.
full list
- favourite searches:
- hydrus can now save, load, and edit favourite searches. this first system stores searches with a name and an optional folder name, and contains search predicates, file and tag domain, whether the search is live or not, and optionally sort-by and collect-by
- this is program-wide and all accessed through the new 'star' icon menu button beside any 'read' tag autocomplete input on search pages, duplicate pages, export folder ui, and file maintenance selection
- wrote a favourite searches manager
- wrote a dialog to manage favourite searches
- wrote a dialog to edit a single favourite search
- wrote load and save search functionality
- autocomplete dropdowns that have buttons beside them now stretch their floating dropdown windows across the button width also
- cleaned a variety of search code, simplifying objects and responsibility
- cleaned up some collect-by ui code
- refactored sort and collect controls to better location
- refactored search constants
- numerous small search code fixes and cleanup
- renamed clientguipredicates to clientguisearch
- .
- the rest:
- a note from the users managing Hydrus Companion: The Chrome Web Store release of Hydrus Companion is no longer available due to publishing issues. If you have been using it in the past, please install the extension manually as outlined here instead: https://gitgud.io/prkc/hydrus-companion
- the default e621 downloader is updated to their new system, thanks to a user's submission. if you log in to e621 with hydrus or the hydrus companion and discover some tags are now blacklisted, please check your blacklist settings on your account on the site
- an old test e-hentai login script from 2018 that is no longer in the client defaults will be deleted from clients that still have it today. if the user has no other login script for e-hentai, the domain entry will be deleted as well. this removes potential technical barriers for users that wish to use hydrus companion to access e-hentai, which is now the recommended method
- hydrus mpv now has an appropriate stream title, which propagates up to the os-level sound mixer. it was previously the ugly hydrus filename
- improved error handling when mpv is passed an invalid conf
- the default mpv conf now has audio normalisation that seems to work ok
- fixed an issue with the 'delete/move out missing/corrupt file' file maintenance job where record deletes were not processing correctly. it now deletes the file record correctly and also clears that deletion record, to make re-import of the correct file, if found, easier
- all hydrus menu labels are now "middle…elided" when they are greater than 64 characters
- all new hdd, url, and simple download pages should now obey the 'remove files when trashed' rule. pages in existing sessions will not
- updated the user-created CutieDuck darkmode qss file to the latest version, which alters the recent hydrus qss styling colours like green/red button labels
- did a full pass of all service fetching–all file and tag services should now present in lists and tabs in service_type, alphabetical order, e.g. for manage tag siblings, the tabs will always be local_tags, tag_repositories, both in alphabetical order
- fixed an issue where a 'get darker or lighter comparison colour' calculation was not working well for black or very dark colours
- if subscriptions or general network traffic is paused, the bandwidth section of the main gui statusbar now says it
- the status bar now tooltips each section
- clarified some labels on the edit url class panel
- moved all delayed focus-shifting code to a more stable system
- cleaned up how the global icon cache is initialised and referenced
- updated the hydrus project gitignore to hide all db, log, server, recovery, and media files that could be under the db directory
- updated the endchan links in the help to have a .org secondary link
- more general code refactoring
next week
Next week is a 'small jobs' week, so I will be back to catching up on small things and general mpv/shortcuts/cleanup work. Also Deviant Art are perhaps going through some more layout changes of their own, so I will be looking at that.
As the virus hits, I expect to keep working on hydrus as normal. I am supplied, low social contact, and healthy. I hope you are as well. If we all end up locked down for a month, I hope hydrus can offer a distraction. I will post if I have to stop for a couple of weeks.
Version 387
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v387/Hydrus.Network.387.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v387.tar.gz
I had a great week mostly fixing things and adding and improving small features.
all misc this week
The 'sort files by' dropdown on all pages is now a button. It launches a menu that groups the different sort types, cutting the long list down into easier to navigate groups. Mouse wheel still works on it!
Also, 'sort by framerate' is added. It just does a simple num_frames / duration calculation for now. Fps isn't surfaced in the UI atm, so I expect in the near future to add it to normal file labels and also to add a system search predicate for it.
The options->sort/collect panel finally got its overhaul. Managing namespace sorts is much more sane, support for namespaces with hyphens (like 'creator-id:') is added, and you can edit the order in which they appear in the sort by menu.
Some logic behind tag autocomplete lookup is improved this week. The way special characters like braces and paretheses are handled is better (for instance if you want to search for '[intensifies]'), and now hyphens and underscores are included in these special rules. Typing in 'blue_eyes', 'blue-eyes', 'blue eyes', or 'eyes' will match all of 'blue_eyes'. 'blue-eyes', and 'blue eyes'! Although the results may still be separate without tag siblings, you should never have to worry about searching an underscore version of a tag again. It will take up to five minutes to update your client to reflect the new rules. This is a simple change to a complicated system, so let me know if it fails anywhere, particularly in namespace or wildcard searches!
Right-clicking on a page tab now shows a 'duplicate page' menu item. It simply makes a complete copy of the page (or page of pages) right next door!
Numerical ratings (the ones with multiple 'stars') can now be set by dragging the mouse. You can click on 2/5 and drag up to 4/5 if you change your mind.
The derpibooru downloader gets an update thanks to a user's submission. The 'no filter' search should work again. Also the new tvch.moe imageboard is added to the supported watchers (thankfully, it was compatible with an existing parser, so this was a quick job).
full list
- the sort-files-by dropdown is now a button that launches a nested menu. it still supports mouse wheel events. it should now be quicker to find what you want!
- added 'sort by framerate' to regular file sort. it works for file search at the db level as well, when mixed with system:limit
- under options->sort/collect, the namespace sort-by ui has finally had its makeover. it now has add/edit/delete buttons and up/down buttons for reordering how the entries will appear. it also deals with bad input better. furthermore, namespaces that have hyphens (like 'creator-id') are now supported in namespace sort (and hence collect-by dropdowns!)!
- numerical (multi-star) ratings can now be set by dragging the mouse across the line of stars
- added 'duplicate page' to the page tab right-click menu! it just makes a copy of the page or page of pages right beside it
- system:everything will now always show up in non-query-page autocomplete dropdowns (such as in the file maintenance dialog)
- wrote a maintenance routine to repopulate and correct the tag text search cache. it is possible to trigger this (though it is typically pointless) from the database->maintain menu
- updated the characters that are ignored in autocomplete tag text search rules, which help skip over unusual characters and assist word-break discovery for searching for tags like '[intensifies]'. as well as the previous brackets, braces, paretheses, quotes, and double-quotes, now slash, backslash, hyphens, and underscores(!) are ignored. searching for 'bbb' will now match a tag 'aaa-bbb', and searching for 'blue_eyes', 'blue-eyes', 'blue eyes', or 'eyes' will match all of 'blue_eyes'. 'blue-eyes', and 'blue eyes'!
- to effect the above change, the client will take a few seconds to a minute to update
- the above tag text search rules now collapse contiguous unusual characters, or combinations of whitespace and characters, better
- namespace and simple wildcard search inputs no longer have the tag text search rules applied to them, meaning you can now search for these unusual characters more specifically when desired
- updated the derpibooru gallery search objects to use their api, thanks to a user's submission. this re-enables the 'no filter' mode
- added watcher support for tvch.moe, which works with an existing 4chan-style parser
- the 'add the ptr' help item now warns the user about the ptr's modern drive storage requirements (4GB download+files, 25GB db). the help files are also updated
- I believe I fixed the sometimes crazy fast media drag-move that could happen in archive/delete and duplicate filters
- fixed an old uncaught wx->qt issue with the simple downloader where editing the formulae would throw an error
- fixed a bug in the 'move highlighted thumbnail' code in the rare case where the currently focused thumbnail can not be found
- text input dialogs are now mostly wider
- refactored some ui code, cleaning up core objects and import hierarchy
- did some controller/gui refactoring, pushing on untangling things
- cleaned up a bunch of no-longer-used import statements
- misc ui code cleanup
- slight rewording of database menu
- prepped shortcuts system to ignore a window-activating click (for the media viewer filters), but can't turn it on yet as media viewer clicks are not yet fully plugged in
next week
Next week is a medium-size job week. I would like to get 'favourite searches' working, so you can save a particular page's search and then quickly load it up later wherever you like.
I would like to add some default ratings services to the client as well, since they are easy for new users to miss.
Version 386
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v386/Hydrus.Network.386.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v386.tar.gz
I had a great week. I mostly cleaned code, moving old wx definitions and calculations up to Qt format, fixing bugs and colours along the way. There is not much significantly new this week, but I am happy to have cleared out some behind-the-scenes mess.
gif and mpv.conf
Some gifs have metadata that says 'play this once' or 'play this five times' rather than looping infinitely. Hydrus now parses this information, and if you tell it to under options->media, it will obey it.
MPV is extremely configurable. It uses a file called 'mpv.conf' to load its settings. Hydrus has a very bare conf file at the moment, but I expect to expand it as we experiment more with settings that are useful for hydrus. Now you can set the path of the mpv.conf file you want to use, again under options->media. I have added three test mpv.conf files to the hydrus static directory–one for high quality, and two for different kinds of audio normalisation–if you are interested in this subject, or if you have your own mpv.conf files you would like to experiment with, please give them a go and let me know what you think. Once we know what works well for different situations, I will make some more well-rounded profiles for users to choose from.
misc code cleanup notes
You can now edit how fast the mouse cursor autohides on the media viewer, including turning autohide off completely. Yet again, under options->media. The default is 700ms.
The window positioning and sizing system has had several calculation improvements, and it will also now be better at rescuing off-screen windows (for instance if you have a multi-monitor setup that changes between client boots), all with better error handling. Furthermore, some windows like the splash screen–which finally has fixed colours–will now appear on the monitor your mouse is on. If you have had problems with window positioning or sizing, particularly if you have an unusual desktop or multi-monitor setup, let me know if this improves or worsens your situation.
If you are interested in editing your own hydrus QSS stylesheet, please check the install_dir/static/qss directory again this week. I have moved many of the previously hardcoded colours in the client (like green/red dialog button labels) to a new 'default_hydrus.qss' file. This file is loaded by hydrus, and is prepended to any custom stylesheet. You can add these hydrus-specific qss class definitions to your own qss to overwrite them. I expect to continue migrating hardcoded colours to this file in future.
full list
- gifs and mpv:
- the client now parses gifs for loop count metadata (some gifs say they should only be played x times through, usually 1). options->media now has a checkbox to control whether this value should be obeyed. both the native viewer and the mpv viewer should follow this. default value is still to loop indefinitely
- if gifs are set to play with the mpv player, those without duration will now still be loaded in the native image viewer. the media viewing options ui now notes this
- the mpv.conf file used in the mpv window can now be changed under options->media. it _should_ update the conf for all open mpv players on options dialog ok. added to the hydrus static mpv-conf directory are three new 'test' mpv confs for high quality and two audio normalisation tests. all test feedback and recommended conf info is welcome
- .
- ui cleanup and improvements:
- the media viewer mouse autohide time is now customisable under options->media, including disabling it completely. it defaults to 700ms
- improved the timing and reliability of the media viewer mouse autohide code
- the mouse should now never autohide while a dialog is open
- improved the bad colours of the splash screen. it should now be all one colour now, no ugly stand-out white square or other hardcoded colours. hydev also deployed his unparalleled gimp skills to get a white fade around the transparent-background hydrus icon, so it should look correct in darkmodes as well
- created a default_hydrus.qss file in the qss folder in order to handle formerly hardcoded colours using hydrus-specific classnames and properties. as well as being loaded by default, this qss file is prepended to any custom stylesheet, so any custom stylesheet that includes its own versions of the hydrus-specific entries will override the defaults. this qss will get more work in future
- added on/off buttons to hydrus default qss and converted existing object to use class and properties to obey this
- added a variety of valid/invalid/warning text colours to hydrus default qss and converted existing text objects to use classnames to obey this
- added accept/cancel buttons to hydrus default qss and converted all green/red buttons across the program to use classnames to obey this
- the migrate database dialog now has an outright 'remove location' button to reduce confusion and speed up removal of high weight locations
- if a location does not exist on the migrate database dialog, it will now stop throwing multiple error popups every time the list slightly changes, and will complain if file rebalancing is attempted, and will provide different 'remove' yes/no messages if that missing location currently has files or not
- slight ui touch-ups to the migrate database dialog
- if a window that remembers its position attempts to re-position to a location not on a current display, the windowing system now attempts to rescue it to the primary display, with appropriate popup messages given and errors caught more gracefully
- extended these off-screen rescue calls to windows that pull their position from their parent. e.g. if you open the options dialog while the main gui is half over the left side of your screen, it should rescue to the primary display
- windows that position off the center of their parent now calculate that reliably on the parent window, not just the parent widget, which never really worked as intended
- windows that have no position memory and no parent to pull center/topleft position from will now appear center/topleft of the monitor your mouse is on
- the splash screen now appears centered on the monitor your mouse is on
- cleaned up and improved a bunch of window/screen coodinate code, moving 'space on screen' calculations to 'space on screen minus taskbar' and similar
- unified a 'dialog is open' check across the program
- cleaned up the old wx->Qt size, coordinate, and colour conversion code
- cleaned up some old wx->Qt calculation code
- improved 'light' and 'grey' colour detection code to now work in HSV
- improved colour changing code to now work in HSV
- improved some internal single-shot scheduled job code
next week
Next week is a 'small jobs' week. I want to fix bugs, improve some ui, maybe add some new shortcut actions and other small features.
Version 385
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v385/Hydrus.Network.385.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v385.tar.gz
I had an ok week. There are more improvements to mpv and the shortcuts system, and a few misc small jobs done as well.
mpv
Some more general improvements this week:
MPV windows now work in slideshows, and all audios/videos are smoother to move to the next file when slideshows are running.
When you scrub the seekbar below a native or mpv video player, the video should now more reliably pause itself while you drag. Also, more kinds of one-frame music videos (where it is just an image webm with a song in the background) should now appear with their seekbars now, and the seekbars for all these unusual videos and audio files should work better.
A zoom bug where video could navigate down a few pixels on zoom in/out cycles is fixed, and the blank audio mpv window now has some hardcoded zoom capability and should scale itself to fit in the small preview window, always letting you interact with its seek bar.
I attempted to get mpv working for macOS but ran into some technical walls, I think because I am limited to 10.12 Sierra. I do not think there are excellent solutions for this, at least for my macOS expertise, so mpv support on macOS is unfortunately on hold until I learn more.
shortcuts
The media viewer gets more shortcut work this week. Until now, hitting enter, return, or escape on any viewer, and middle or double left-click on the normal browser, was hardcoded to attempt to close that viewer. This is now all plugged into the shortcuts system and is editable. It comes under the 'close_media_viewer' command for all the 'media_viewer' sets. Mouse integration is not complete, but we are almost there, and most actions should work, if you want to map, say, alt+left click to something.
Last week's double click support added an interesting bug to existing workflows: whenever you intended to click twice fast–like when using the archive/delete filter quickly–the second click was detected as a double-click instead, and often nothing would happen. This is now fixed. Now, if there is no double-click command mapped, including on any parent window, that second click will be attempted instead as a second single click. So, you can now click through the archive/delete filter fast again.
This new single/double click logic (when any parent window does have a double-click command mapped, the single click substitution does not occur and that double-click does count and is passed up) adds a neat new capability to mpv windows, which, because they scale up so well and usually almost fill the screen, leave little whitespace to click 'outside' the window to do anything but pause/unpause on a left-click. Now, a double-click can be neatly passed up to a parent if there is a mapping. For instance: a double-click on an mpv player in the regular browser media viewer will now pause and then close the window (rather than doing a quick pause/unpause), because 'media_viewer_browser' on the larger window has double-click mapped to close the window. I have also made it default that double left-click in the archive/delete filter is now mapped to 'keep', so in a similar way, double clicking an mpv window in the archive/delete filter will now do 'keep and move on'! If this is a bit complicated to read, don't worry–have a play with it and you'll get the feel. I hope this makes filtering videos a bit nicer now we have mpv.
the rest
Danbooru apparently changed how their tags work behind the scenes, so this week adds new parsers for danbooru that should have everyone working again. Thank you to a user for submitting these.
I fixed some searches, usually ones without inclusive tags, where files in the trash could appear even though the domain was 'my files'.
A popup message now appears while a session is loading, just as a little indicator for slow-loading sessions until I can get better loading progress ui integrated here.
The splash screen looks less ugly, but it is still bad. I will keep working on this to get the greys/whites/blacks and layout margins looking better for all stylesheets.
full list
- mpv:
- the mpv window is now plugged into the slideshow system, so when an mpv window has played its media once, a media viewer browser currently slideshow-ing will now correctly know to move to the next media
- slideshows of videos should progress to the next file faster and more smoothly, with no more half-second blit of the start of the movie before the next file loads
- pausing video/audio is no longer cause for the slideshow to move on–now only the 'has played once through' check will naturally trigger it. you can now more reliably seek/scrub a video during slideshow
- the mpv window and native animation player is now better about pausing while a seek drag (scrub) is ongoing
- various music videos that have 1 frame should now show a seekbar correctly for the mpv window, with correct timecode based seeking
- fixed a bug in the video/audio resizing code that meant zoom in/out cycles would move a video player down a few pixels off center
- the blank audio mpv window now has basic hardcoded zoom support and will scale down to a too-small viewer window, so you can still access the seek bar
- fixed some 'has duration' calculations for audio/video that has duration but no frame count
- .
- shortcuts:
- if a shortcut handler on an individual window does not have a double-click mapping for a command–and furthermore if none of its parent windows that are fully plugged in have one either–it will attempt to map the single-click version of the event as a backup. so now if you have an archive/delete filter, you can click fast again and the double-clicks will be interpreted as single-clicks (unless you map double-click to mean something else on the media_window or one of the media_viewer parents)
- the media viewers across the program are now fully plugged into the shortcuts system for key presses, and half-plugged in for mouse clicks
- 'close_media_viewer' is added to all 'media_viewer' shortcut set types. enter/return/escape are now defaults for 'media_viewer' (applying to all), and middle-click/double left-click are now defaults for 'media_viewer_browser'. this is no longer hardcoded. if you are a madlad, you can now unmap all 'close_media_viewer' commands
- double left-click is now assigned to 'keep' in the 'archive_delete_filter' shortcut set. due to the new click/double click rules above, this means that by default double clicking a video/audio in the archive/delete filter will now mean 'keep and move on' on that fast second click!
- edit shortcut set ui now sorts its command list on load
- significant shortcuts refactoring
- general shortcuts code and debug code cleanup and improvement
- .
- local file import pages and most downloaders will now report more file import steps to their status labels. most will blit by too fast to see, but if it hangs for a bit, you will now see the step it is caught up on. I imagine in most cases, this will be metadata generation for large videos
- fixed a variety of searches that could return files not filtered to the current file domain (e.g. files in trash while in my files, or not in ipfs while in ipfs) when the search did not include an inclusive tag
- updated the default danbooru file page parsers to get their new tag format, thank you to a user for submitting these
- a popup message now appears while sessions are loading. it auto-dismisses once the load is complete
- the edit media view options dialog (under options->media, launched from the filetype list) is now better at disabling non-applicable widgets based on filetype
- fixed an issue where clicking from the autocomplete dropdown floating window to the same control's text input could result in a single flicker-frame where the dropdown is hidden
- tightened the size of the splash screen white background. figuring out appropriate colour from the current stylesheet remains elusive lmao
- cleaned up and wx->Qt converted a variety of event handling code
- updated some 'mime' labels across the program to the new 'filetype' wording
next week
Next week is a cleanup week. I will catch up on some general background code improvement. Some users are getting crashes with the new mpv window swapping system, so I will also look at that.
Version 384
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v384/Hydrus.Network.384.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v384.tar.gz
I had a great week updating the shortcuts system.
shortcuts
The 'new' shortcuts system has been in limbo for some time. I like it, but I never really 'finished' it, and there were still many places across the program that had hardcoded shortcuts. This week moves it forward, mostly for mouse clicks and the new mpv window. As a reminder, you can customise the system under file->shortcuts. There are multiple shortcut 'sets' that apply in different parts of the UI.
First of all, the shortcut edit UI has been rearranged so it is less of a mess. Instead of having all possible commands on the same window, you now select which one you want from a dropdown, and unnecessary widgets will be hidden. Also, the main dialog now lets you restore a shortcut set to default values.
Although mouse support remains limited, the windows that take mouse clicks can now catch more. Double-clicks are now recognised, and if you prefer the feel of a click up instead of down on certain actions, you can now differentiate between click 'press' and 'release'.
Next, the actual 'media windows' across the program, the image, animation, or mpv windows, are now plugged in to the shortcuts system. For now, this mostly means clicks. Left-clicking to pause/play is no longer hardcoded, and I have removed the default 'double-click -> open in an external program' (although you can add it back in as ctrl+double left-click or whatever you like) as this command is less important with mpv's audio and fast high-res scaling. There are two new shortcut sets–'media_viewer_media_window' and 'preview_media_window'–that have actions for pause, pause/play, open_externally, and close/launch_media_viewer. By default, middle-click or double left-click on the preview media window now launches the media viewer, just like on a thumbnail.
And the 'media_viewer_browser' set now has 'show_menu' as an action. It defaults to 'release right-click', but if you would prefer your menus on a click-press, or ctrl+release right-click or something, you can now change it. I expect to do this for the other important menus across the program as well. Also, most menus can now be opened with the context menu key on your keyboard, if you have it, so you might want to even remap right-click to something else.
The new 'global' shortcut set now has actions to exit and restart the application. They should be caught by the main gui window or any media viewer and work just like the actions on the file menu.
Most of the work this week was cleaning up behind the scenes code. I feel great about now extending it to more windows (such as thumbnails, to customise selection click shortcuts) and commands, and eventually planning advanced features like multi-actions so you can build pseudo-filters where a left-click is 'apply this tag, archive, and move to next file'. For now, I hope to slip in little bits of work here and there.
EDIT: A user let me know that the new double-click detection makes the archive/delete filter not work if you click too fast (it now detects the second click as different than a single-click and hence doesn’t map the ‘keep and move on’ etc.. action). I will fix this for next week, but in the meantime, if you are a fast clicker, please add additional shortcut mappings to ‘archive_delete_filter’ for double-click, and you should be working again.
the rest
I cleaned up some of the borked media-switching code I introduced to get mpv working. Transitioning from one media to another should have fewer instances of single-frame stretch. I am going to keep working here, because in some situations there is a single-frame of blank, which can be a bit of a flash.
The tag autocomplete dropdown window should be better at choosing when to hide and show.
I fixed a bug related to the advanced 'clear deleted tag record' command, which is usually fired by the tag migration window. The action was not properly clearing the cache that thumbnails use to fetch their tags, so while the deleted tag was deleted from the master database table, it did not appear so in UI. If you were hit by this, please run database->regenerate->autocomplete caches one time to resync yourself.
If you are a user who uses the parsing system, the string transformation panel has had a significant rework in the past two weeks. It hides/shows unneeded controls just like this week's shortcuts panel, and there is now a live preview of the current step's example and example-transformed text in the individual transformation rule edit dialog, which updates as you type, including showing regex errors! I expect in the near future to add the string transformation system to the filename tag panel soon, swapping out the jank old regex systems there for the nice full-featured converter.
full list
- shortcuts:
- the shortcut system now supports mouse double-clicks–left, right, or middle
- the shortcuts system now differentiates between press or release single mouse clicks–although complete support for release mouse events may be a bit patchy, as full mouse integration is ongoing
- the shortcut edit ui is now simpler–the command type is selected by a list, and the individual command sub-panels hide and show as appropriate–no more stupid 'set command' buttons
- the shortcut edit ui now has a 'restore defaults' button that will restore an individual set back to default settings
- two new shortcut sets are added–'media_viewer_media_window' and 'preview_media_window'. they control pause, pause/play, open_externally, and close/launch_media_viewer respectively. they work on the static image viewer, the native animation widget, and the new mpv player, and they support mouse clicks. the old pause/play (formerly left-click) and open_externally (double left-click) commands are no longer hardcoded
- by default, the preview window's media window now launches the media viewer on a middle- or double-left-click
- 'media_viewer_browser' shortcut set now has 'release right-click' bound to 'show_menu', a new command, which is no longer hardcoded
- most menus across the program can now be opened with the keyboard context menu key
- the 'global' shortcut set now has 'exit_application', 'exit_application_force_maintenance', and 'restart_application' commands
- fixed the rating increment/decrement command option not hiding in non-'media' shortcut sets
- fixed some issues loading edit ui for shortcuts with rating actions
- significant refactoring and some cleaning of shortcut code
- .
- the rest:
- mpv windows should not longer get a single frame of previous-window-stretch when flicking between one mpv media to another with a different aspect ratio on the same media canvas. when a video is caught in a frame of loading, it should now flicker a frame of black
- switching from one static image or native animation to another of the same type _should_ be less likely to do a single frame of stretch when transitioning. when an image or animation transition is caught on a new frame, it _should_ now flicker a frame the same colour as the media canvas background
- the string transformation edit panel's individual transformation rule edit panel has had some more work: much like with shortcuts, the controls now hide and show based on transformation type, the controls' text labels now change based on transformation type, and the example text now updates on any widget change. the manual 'update example' button is removed
- fixed a typo that caused an error when establishing the correct mouse cursor to use over the volume control when hydrus was using PyQt5 (rather than PySide2)
- in order to reduce accidental micro-drags that cause mpv load-pause issues, starting a thumbnail drag now takes more pixels and requires a smoother drag to start, let's see how it goes
- improved the show/hide logic of the floating autocomplete dropdown window. it should now also reliably detect when window focus goes from the dropdown itself to another window
- fixed a bug where clearing the deletion record of a deleted tag would not remove the record from the fast cache that populates thumbnail tags (making it seem on most file loads that the tag still existed). if you were hit by this previously, please hit _database->regen->a/c cache_ one time to resync the cache
- relatedly, thumbnails should now correctly live-update their deleted tags on deletion record clearance updates
- if mpv is not available, opening the about window will now make a popup with the actual import error trace
- significant refactoring of various ui code
next week
Next week is a 'small jobs' week. I have plenty to be getting on with, so I will just try to mix it up with a variety of fixing and improving work. I'd like to push on some more mpv issues like slideshow and custom .conf support in particular.
Version 383
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v383/Hydrus.Network.383.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v383.tar.gz
I had a good week. Unfortunately it was a little changelog-light again, but hydrus now has a nice volume control, and mpv should be working for the Linux build.
mpv volume
You can now set volume and mute properly across the program. Any mpv player that has audio will now have a small speaker icon button next to the seekbar. This controls 'global mute', which silences the whole client without exceptions. This control is also on the top hover window of the media viewer.
When you put your mouse over the speaker icon, a popup appears with a second speaker button and a volume slider. The second button controls mute just for the preview window or the media viewer. By default, the volume sliders are also respective to preview vs media viewer, so you can have a quiet preview vs a loud media viewer if you like. (if you wish, you can change this under options->audio to have them both use the same single global volume setting)
You can also change the mutes and volume from the media canvas right-click menus.
Furthermore, there is a new shortcut set, 'global', which now has actions to set, unset, or flip the global mute. At the moment, it works on the main gui window and any media viewer. By default, I have set ctrl+g to flip global mute on and off. Please feel free to set whatever works for you under file->shortcuts->global.
I expect to add more options and shortcuts for altering volume.
linux
I have updated my Linux dev machine from Ubuntu 16.04 to 18.04, and with it, I have added mpv support for the Linux build.
This has caused some changes to my environment and the build. My build packager has included more library files than it needed, so the release has bloated. I have taken out many .so files that I am fairly certain are not needed, but it is still about 200MB. I will test this more and hope to cut down more in future, as well as improving my PyInstaller configuration to slim this down more naturally.
It is possible that some (perhaps older) flavours of Linux that were happy with the 16.04 build will not be happy with the newer 18.04 libraries. If you use the Linux build, I would be interested in knowing:
1) Does 383 boot for you?
2) Does MPV load correctly and play audio?
Early tests seem to be good, but if you do use an older version, please be prepared to roll back.
full list
- mpv:
- updated the prototype volume/mute controls on the top media viewer hover window to be a proper 'speaker' icon button for mute with a volume slider that pops up or down on mouse-over
- the new volume control is on the hover window and any media that has audio
- the right-click menu of the preview viewer and media viewer now have volume submenus to set mute/volume
- the client now has multiple volumes and mutes:
- for mute, there is a global mute which overrides everything, and the preview and media viewers have their own mutes that just apply there.
- under options->audio, you can choose whether preview windows have their own separate volume value, default is yes they do
- there is a new shortcut set called 'global', which applies on the main gui and the media viewer both, and which currently has actions to alter global mute. by default, ctrl+g flips global mute
- after reports of unusual rendering bugs for some users, the default mpv.conf is now more barebones. more work will happen here
- .
- linux:
- the linux release is now built on Ubuntu 18.04 (was 16.04). unfortunately, my build packager bundled in a variety of surplus libraries, so the archive has bloated somewhat–I have removed some that I am confident are not needed, but I may have made a mistake, and there are likely more that can be taken away
- the linux release now comes with mpv support
- please let me know if you have any errors running this build or loading mpv. early tests seem good though!
- .
- the rest:
- the launch/exit splash screen now uses a cleaner Qt-compatible layout system. It resizes and obeys stylesheets better, colouring text and background according to current style
- removed the 'has duration' text label option from 'audio and duration' options panel as it is no longer used, and renamed the panel back to just 'audio'
- the string transformation edit panel's individual transformation edit panel now shows that transformation step's example string and the transformed string, which is updated by button. this edit panel will get some more love soon, including dynamic hide/show of applicable controls and live updates of the example transformation as you type
- misc ui layout improvements
- misc ui improvements
next week
I would like to put some time into the shortcut system. The shortcut edit UI could do with some slimming down, and I want to integrate more mouse actions, particularly now we have MPV. I would love to have the currently hardcoded clicks like video pause/play and 'open externally' completely configurable so users can set up the archive/delete etc… workflows that work best for them.
Version 382
EDIT: Fixed Links
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v382/Hydrus.Network.382.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v382.tar.gz
I had a good but light week. MPV should be working better, and some of the recent UI animation flicker/jitter should be gone.
mpv
Introducing MPV added a weird bug–after the first time an mpv window was opened, the client would then in some cases fail to animate thumbnail fade or process certain intra-program events unless another event, such as a mouse move over the program, restarted the event loop. Some media viewer transitions have also had some single-frame flicker. I have banged my head at this problem for a couple of weeks now, thankfully improving some behind-the-scenes code along the way, but I could not find the full solution. As it happens, rolling Qt back to a slightly older version (5.14.0 to 5.13.0, for users running from source) seems to have completely fixed it. It could simply be a 'native window' bug in the new version of Qt, but at the moment I still suspect my mostly borked wx-code-trying-to-talk-to-Qt-code. Until I can clean that code up, I am happy to have this easier solution.
I fixed a recent issue with the preview viewer not clearing out media correctly on page changes (which was particularly significant now we have audio) and also fixed an mpv crash for Linux/macOS users who happened to already have MPV available through their OS. Also, MPV is now configured to have a 'lighter touch' on filesystems, so if you have a client with a laggy file storage system (e.g. a very large client stored over a network share) and have experienced extreme latency when trying to open MPV, this situation should be much better. MPV should be fairly stable now, so my big remaining tasks are to get nicer volume controls and proper build support for Linux and macOS.
MPV as a program relies on '.conf' files for most of its configuration. Hydrus now follows this format. If you have your own conf or otherwise would like to edit this, please check out the mpv.conf under install_dir/static/mpv-conf. If you have settings you think should be in the default for all users, let me know. I'll likely let you optionally override where hydrus looks for its conf (where it can't be accidentally overwritten on update) in future.
the rest
There are now a couple of user-created darkmode stylesheets in a default install, so the stylesheet section of options->style now has some entries for everyone. Feel free to check them out–there are still some colours in the client that are hardcoded like green/red button text and anything under options->colours, but I plan to move everything over to Qt in time.
When you zoom out from a zoom that makes the media larger than the media viewer canvas to a zoom that fits inside it, the media will now re-center. See if you like this–I may make it optional.
An issue with the 'the client is already running' dialog not appearing is fixed.
full list
- mpv:
- rolled Qt back from 5.14.0 to 5.13.0 on the releases, which seems to have fixed our 'event queue sometimes halts until mouse move' issue that occurs after initial mpv load. some other ui and media viewer resize jank seems to be cleared up. I dunno what happened with 5.14, and I don't suspect it as the problem nearly as much as my currently borked Qt event processing code, but rolling back seems the easiest solution for now
- fixed an issue that was crashing non-windows that were able to load mpv
- mpv now loads an mpv.conf from install_dir/static/mpv-conf. please feel free to swap in another conf or edit that one as you like. I would be interested in feedback
- default mpv conf is now set to specifically enable some hardware acceleration to improve playback for some users, and to never load sidecar files like subtitles as this was introducing incredibly large load lag for users with large/high latency file storage
- fixed a new issue where preview windows were not unloading media (particularly significant for mpv with audio) on page change and client shutdown
- fixed an issue with global volume propagation to multiple open mpv widgets
- .
- the rest:
- added two dark qss stylesheets from the user-creation github repo to the default install
- when zooming out from a zoom that makes the media bigger than the media viewer canvas to a zoom where it fits, the media will now recenter. see if you like this, maybe it should be an option
- to help forestall unnamespaced filename tag spam in various new-user scenarios, the 'filename' checkbox-and-namespace widget on the filename tagging options panel now initialises with 'filename' as the namespace
- fixed a recent window sizing issue with the 'the client is already running' dialog not appearing
- file export paths that include subdirectories that could possibly have empty text, like "[creator]/[page]", will no longer error when this is so (e.g. if a file in this case has no creator tags)–they will eliminate the subdirectory entirely, becoming "[page]". this should work for all platforms and for any nested subdirectory
- fixed an issue with some fractional dataspeeds below 1KB/s displaying with many significant figures
- improved some custom event handling definition code
- reworked hydrus's internal object publisher/subscriber messaging system to be more Qt-happy
- if the file import tagger is given a neighbouring .txt file to pull tags from that does not decode to utf-8 nicely, it now catches and reports the error more gracefully
- reworded a bit of the installing help and first-start popup to emphasise that hydrus does not auto-update
- added links to https://github.com/Zweibach/text/blob/master/Hydrus/PTR.md , a new guide for the PTR, to the help
- removed the old 'hardcoded shortcuts' help entry, since it is increasingly irrelevent
next week
I took a bit of time off this week and rejiggered my future schedule to reduce some work stress. I feel great. Next week I would like to make some nice mpv volume controls and clear out some miscellaneous small jobs.
Version 381
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v381/Hydrus.Network.381.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v381.tar.gz
I had a good week with a couple of challenges. MPV is now ready for all windows users and is turned on by default.
MPV
Thank you to the advanced users who tested and gave feedback on MPV. I have eliminated the crashes, tightened up the jank, and am now rolling it out to all Windows users by default for video, audio, and gif/apng. All media view settings under options->media will be reset this week.
MPV is a good free media viewer. One of the core benefits of moving hydrus to Qt was being able to plug it into our media player, enabling hardware-accelerated video playback and audio. It looks just like the native player, with the seek bar beneath, but it works much faster, able to play 1080p or 4k videos at 60fps at full or unusual zooms. And of course, it makes noise!
This is early days. I have only just started plugging into MPV, so many features are basic or not yet available. The global volume and mute controls are currently some very ugly controls in the top media hover panel. Slideshows will not move forward on an MPV window (hydrus doesn't know when an MPV player has 'played once through' yet), and some processes like the archive/delete filter will need some extra workflow options now that more users will be playing videos at high res (left-click on the player pauses the video, so to set 'keep' on archive/delete, you'll have to click on some whitespace, of which there is so much less when the video is so big). Please let me know what your top priorities for improvement are, and of course, if you encounter errors or crashes, let me know. I'll keep working.
Even though it was a lot of difficult work, I am overall really pleased with how this has gone. The only big remaining bug that I need to nail down is an unusual thing where after multiple mpv viewings in a preview window, that page will stutter some query/thumbnail loading unless the mouse is moving. This is a slightly frustrating bug, but the benefits of MPV are enough that I am happy to live with it. I will also get it going for Linux and macOS, which I did not have time for this week.
the rest
Assuming that users will want to set/unset MPV and other view/zoom options for filetypes in the coming weeks, I have reworked how all of that works under options->media. By default, you now set view and zoom options for 'all video files' and 'all images', and then if you have specific options for just webms or pngs, you can set those specific options to override the group default. Every user will be reset to the new defaults on update. Please have a play with this this week.
Similarly, I have reworked the UI for system:filetype. The growing list of individual filetypes are now hidden from view when not needed, and the 'group' types have tristate checkboxes for better review. It takes up less space and just feels better.
Also, I have pulled gifs and apngs out of the 'image' group and created a specific 'animation' group for them. This doesn't change much, but it makes it easier to search for or manage settings for static images vs little animations.
I fixed a bunch of the weird layouts that were accidentally introduced last week. Please report any more you find–I am still fighting to convert old wx layout code to Qt's system, so this could happen again in a place I do not notice.
full list
- mpv:
- mpv is now available and the default for all windows users
- I believed I have eliminated the final reported mpv crash
- mpv load and unload delays are greatly reduced. initial load still takes about half a second, but subsequent loads are now as quick as native renderers
- mpv seems to work well for gif and apng
- added a very simple global volume slider and audio mute checkbox to the media viewer top hover window. this was a quick patch–much better controls and shortcuts will come in future
- mpv windows now properly re-show the cursor on mouse movement
- unified mpv mouse press/release handling with native animation–click down now does pause/play and starts a drag event
- unfortunately, in some cases embedding mpv requires overriding local OS number rendering (e.g. 1,234 vs 1.234). hydrus number rendering is now coerced to the english style with commas until we can figure out a better solution–sorry!
- cleared up an issue where simple clicks on page tabs would trigger micro-page drags that were immediately cancelled. this situation was exacerbated when the page being left had an active mpv window. the flicker of page drag cursor is now gone, and some weird situations where static clicks during busy time could move a tab should be fixed
- eliminated the recent issue in the media viewer where transitioning from one media type to another through navigation, particularly mpv->other, would flicker a single frame of the last 'other' media shown(!)
- fixed a bug where repeated mpv views in the preview viewer could disable client file drag and drop
- the bug where thumbnails may not waterfall in unless the mouse is moving after some mpv videos are loaded for a page is relieved but not completely fixed
- if the preview window is collapsed and hidden, media will no longer ever load into it
- fixed an edge-case bug where the mpv window would not like being told to show nothing when it was already showing nothing
- wrapped mpv load errors in a basic graceful catch
- fixed an issue some users had with loading mpv's dll
- .
- file types:
- a new file metatype, 'animation', is added, for gif and apng. these are no longer considered 'image' for a variety of purposes
- the filetype selection panel, which is used in system:filetype and import folder UI, has had an overhaul–it now has tristate 'mime group' checkboxes to represent a half-filled group and expand/collapse buttons to hide the tall filetype lists. individual filetype lists will start hidden unless their default value is a partially filled group
- the media view options have a similar overhaul: they are now collapsed to general filetypes by default. you set view and zoom options for the generalised 'video' type under options->media, and if you want to set specific options for webm or anything else, you can add/delete those types to override the general default
- the new default options for a fresh client are just for these general types. if mpv is available, video, animations, and audio now start with mpv as the default viewer. video and animation zoom is now flexible (not fixed to 50%, 100%, 200%) and will fill the media canvas
- all media view options will be reset to this simple default on update! if you have specific zoom or display preferences, please reset them after the update–but you might like to play with mpv a bit first, as it renders at large and smooth zooms very well
- .
- the rest:
- the new thumbnail right-click file selection routine will now only focus and scroll to the first member of the selection if no other members of the new selection are already in view
- fixed some caching code and sped up the new select/remove menu count generation (which can lag for very large pages) by two to six times
- sped up file filter counting code by about ten percent
- fixed weird layout on: migrate database panel, duplicates page (left and right), edit shortcuts, edit import folder, and the filename tagging panel
- fixed an issue where the media viewer's hover windows might flicker into view for one frame when the mouse moved over the center of the media viewer for the first time
- fixed a media viewer shutdown issue that would sometimes lead to the first file in the list being opened in the shutting-down viewer for an instant or highlighted as the new thumb focus
- the file maintenance system that queues up missing/broken files' urls for redownload will no longer re-select the download page on every new url
- fixed an issue where a downloader's tag blacklist was not being applied on the child files of certain kinds of multiple-file post (such as with pixiv)
- deleting a very long tag should no longer create a very wide confirmation dialog in the manage tags dialog
- fixed some 'the panel grew a bit, but the parent window didn't grow quite enough and now it has scrollbars for two pixels of extra content' sizing issues
- fixed some dialog sizing calculations when the parent window was borderless fullscreen
- maybe fixed a rare event processing bug
- improved quality of some misc data comparison code across the program
- did some significant backend event/pubsub code cleanup, mostly related to getting mpv working a bit cleaner
- improved thumbnail rendering time
- improved smoothness of thumbnail fade animations (at least for when they are working right, ha ha!)
- misc fixes
next week
Unfortunately, I believe that I burned out over the past four to eight weeks. I have been pushing too hard, trying and failing to keep up with my promises, and along with some IRL stuff it nuked my schedule and energy and mood. It hit a breaking point this week, and I realised I was working non-sustainably. I will fix this situation in the coming weeks by altering my schedule. I expect to scale back on overall work hours and hydrus changelog work specifically, focusing instead on keeping myself healthy first so I can face other work (like keeping up with messages and maintaining a productive workspace) and not go nuts. I will also try to promise less when it comes to timeframes so I do not feel bound to stay up late working. I apologise if you have been waiting on me for something–I lost where I was.
I would like to do some more mpv work next week, and do some code cleaning. I will also be taking a bit of time off, so it will be a light week. Thanks everyone!
Version 380
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v380/Hydrus.Network.380.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v380.tar.gz
I had a couple of difficult weeks, with illness and other IRL problems getting in the way, but I got some hopefully pretty neat work done. A new fast video and audio player is available for advanced users to test, and there are a bunch of fixes and ui improvements as well.
A user just notified me that the Duplicates page has crazy layout! I apologise. Nothing is broken, it is just sizing wrong, and part of a longer fight I am having to convert my old wx layout code to Qt. I know exactly what happened here, and I will have it fixed for 381. If you discover more UI like this (the system predicate panels have a bit of it), please let me know.
mpv
This is just for advanced users this week. It is a basic prototype that is not ready for real use. I will improve a bit before turning it on for everyone, hopefully next week.
MPV is a good free video player. One of the core benefits of moving hydrus to Qt was the potential of being able to neatly embed it into the media viewer. I am happy to report that these past two weeks have been successful, and it looks like hydrus is finally getting hardware-accelerated video playback and full native audio support!
If you are using the windows build in advanced mode, please check out the big ugly filetypes list under options->media. Under all the video and audio filetypes, you can now set 'show using mpv' as a show action. Please forgive how large this list has grown–before I turn mpv on for everyone, I am going to rework this list so you have the option to just say 'for all video: xxxxxx', rather than always having every single filetype.
Once you have it set, it should load video almost exactly the same as my native renderer, with the scanbar below for navigation. Audio files will show as a (for-now fixed-size) black box or album art. Be warned: there are no volume/mute controls yet! I have set it at 70% volume for now. MPV has an optional and configurable on-screen display, but for technical reasons I had to stick with my custom controls.
I know what features I need to add, so the feedback I am most interested in this week is in where mpv breaks. If it doesn't work for you, or if it fails to load a file, or it has bad performance, or it sizes wrong, or you get a crash after closing the media viewer, anything like that, please let me know. Performance in my tests has been excellent–rendering 4k 60fps no problem as long as you have any sort of branded GPU. I have had some audio jitter on some unusually encoded videos and some crashes (which I think I have fixed).
One annoying thing is mpv requires a hefty 57MB dll, making the already bloated hydrus download even bigger. One user suggested perhaps putting out two releases–a 'core', and an 'update', which wouldn't have the big stuff. I am still thinking about it–while I like the overall idea, the hydrus build is held together with duct tape, and having a small non-functional release will increase complexity and inevitably cause ugly syncing difficulties whenever the big stuff does need to be updated.
Users running from source will need libmpv and python-mpv to enable this. I haven't pinned down the most convenient and functional way to do this for Linux and macOS yet, so we'll have to figure this out over the coming weeks and get it into the help. Any knowledgeable feedback here would be great.
For the near future of mpv, I would like to: figure out the technical side of including mpv in the Linux and macOS builds; write volume/mute UI and shortcuts; add global volume/mute management; test mpv with gifs and apngs; explore custom shaders and configuration files; and generally polish the whole thing.
quality of life
The tag right-click menu has had a significant overhaul. More commonly used actions are moved up, less common down. Tags with siblings now provide all siblings under the 'copy' menu! System search predicates have better copy and search-action support (like 'exclude' system:archive, which will actually add 'system:inbox').
Right-clicking on the thumbnail grid's 'whitespace' no longer does a 'deselect all' action, so you can now easily right-click in an empty area and go remove->selected!
Across the program, all filetypes are now referred to with human language. Instead of 'image/jpg', it is now just 'jpeg'. Instead of 'application/x-7z-compressed', it is now just '7z'. Please let me know where it looks bad, and if you would like to edit these labels.
The 'new page chooser' dialog now dismisses itself if you click off it! So, if you accidentally middle-click in some page tab whitespace and it annoyingly pops up, just left-click, and it will go away. Also, this dialog now catches enter/return key, and will hit the 'first' button (if you imagine the nine possible button locations as a number pad, starting at top-left). Hitting enter twice will typically open a new 'my files' search page.
A user surprised me recently by asking where to find the 'pause video' shortcut–I hadn't realised there wasn't one! Now this is fixed: 'pause_media' and 'pause_play_media' are added to the 'media_viewer' shortcut set, and 'pause_play_slideshow' is added to 'media_viewer_browser' set. Previously, the slideshow pause/play was hardcoded to space bar–this is now gone, so if you used this, you'll want to set it up yourself.
the rest
I have added simple support for RealVideo, RealAudio, and TrueAudio files. These are some formats, so if you come across some variable bitrate rmvb file or something that doesn't work, please send it in and I will have a look. MPV couldn't handle a couple of my test .rm files' audio tracks, but the .ttas seemed great.
Ever since the CloudFlare-8chan break, I have been telling myself to figure out some new downloader objects for the bunkers everyone scattered to. This job kept being put off, long enough that 8kun came back(!), but I managed to fit in some time in this morning and got 8kun and vch.moe support added for today's release. 8kun was obviously easy to convert from the old 8chan parser, and vch's engine provides an API very similar to 4chan that I was able to duplicate and modify for. I had a proper look at julay, smuglo.li, and endchan, but these use their own API formats (or no API at all), so I will have to write new parsers, which will take longer. I really would like to get them done for 381–please remind me if it looks like this job has slipped away again.
full list
- basic mpv support is added. it comes with the windows build this week, and is a prototype meant for initial testing. the library is optional. users who run from source will want 'python-mpv' added via pip and libmpv available on their PATH, more details in running_from_source help
- took an qt-mpv example kindly provided by a user, updated it to work with the hydrus environment, and integrated it into the client as a new choosable view type under audio/video filetypes under options->media for advanced users
- reworked how the 'start paused' and 'start with embed button' media viewer options work under options->media. these are now separate checkboxes, not combined with the underlying 'show action'. existing embed/paused show actions should be converted automatically to the correct new values
- unfortunately, due to some python/qt/libmpv wrapper mouse interaction issues, mpv's 'on screen controller' overlay is not available
- for now, left click pause/plays the mpv window, just like the native mpv window.
- preview/next frame shortcuts should work for the mpv window when playing video
- no volume/mute controls yet, these will come in the coming weeks, including global mute settings
- updated media show and sizing code to account for mpv widgets
- reworked my animation scanbar to talk to mpv, and for my mpv window to talk back to it
- improved the animation scanbar to be more flexible when frame position and num_frames are not available, both in displaying info and calculating scanbar seek clicks
- mpv api version added to help->about
- .
- new downloader objects:
- thanks to a user, updated the 'pixiv artist page' url class to a new object that covers more situations. the defunct 'pixiv artist gallery page' url class is removed
- added 8kun and vch.moe download support. I got started on julay, smug, and endchan, but they were a little more tricky and I couldn't finish them in time–fingers crossed, next week
- .
- menu quality of life:
- a right-click on thumbnail whitespace will now not send a 'deselect all' event! feel free to right-click in empty space to do an easy remove->selected
- remorked the tag menu layout to move less frequently used actions down:
- - moved the discard/require/permit/exclude search predicate actions down
- - moved 'open in a new page' below select and copy
- - moved copy above select
- and some misc menu layout improvement on this menu
- fixed some labelling with the discard/require/permit/exclude verbs on negated tags
- right-clicking on system search predicates now shows the 'copy' menu correctly
- system predicates that offer easy inverse versions (like inbox/archive) should now offer the 'exclude' verb
- when right-clicking on a single tag that has siblings, its siblings and those siblings' subtags will now be listed in the copy menu!
- copying 'all' tags from a list menu, with or without counts, will now always copy them in the list order
- across the program, all menu 'labels' (menu text items that do not have a submenu and have no associated action, like 'imported 3 years 7 months ago') will now copy their text to the clipboard. let's see how it goes
- .
- other ui quality of life:
- across the program's UI, filetypes are now referred to with simpler terms rather than technical mimetypes. instead of 'image/jpg', it is now typically just 'jpeg'
- the 'remove selected' buttons on the gallery and watcher pages are now smaller trash icon buttons
- the new page chooser will now auto-dismiss if it loses focus–so if you accidentally launch it with a middle-/double-click somewhere, just click again and it'll go away
- hitting enter or return on the new page chooser now picks the 'first' button, scanning from the top-left. hitting enter twice now typically opens a new 'my files' search page
- added pause_media and pause_play_media shortcuts to the media_viewer shortcut set. new clients will start with space keypress performing pause_play_media
- added pause_play_slideshow shortcut to the media_viewer_browser shortcut set. this shortcut is no longer hardcoded by space keypress
- the six default shortcut sets now have a small description text on their edit panels
- the options->media edit panels now enable/disable widgets better based on current media/preview action
- added a checkbox to _options->gui pages_ to set whether middle-clicking a tag in the media viewer or a child tag manager to open a tag search page will switch to the main gui. default is false
- mr bones now reports total files, total filesize, and average filesize
- mr bones now loads your fate asynchronously
- .
- the rest:
- added tentative and simple realvideo (.rm) and realaudio (.ra) support–seems to work ok, but some weirder variable bit rate formats may not, and I have collapsed the various different extensions just down to .rm or .ra
- added trueaudio (.tta) audio support
- fixed a bug from the recent search optimisations where a bare inbox search would not cross-reference with the file domain (so some trash could show up in a simple inbox/'my files' query)
- fixed an issue with searching for known urls by url class where the class was for a third-or-higher-level domain and was not set to match subdomains (this hit 4chan file urls for a few users)
- fixed the issue with 'open externally' button panel not clearing their backgrounds properly
- fixed some of the new unusual stretchy layouts in the options dialog
- removed overhead from subscriptions' 'separate' operation, which should stop super CPU hang when trying to split a subscription with hundreds of thousands of urls
- fixed an issue where the advanced file delete dialog would not show the simple 'permanent delete' option when launched from the media viewer's right-click menu
- fixed the select/remove actions for local/remote
- fixed 'set_media_focus' from manage tags to correctly activate the underlying media viewer as well as set focus
- stopped the 'file lookup script' status control from resizing so wide when it fetches a url
- fixed a rare mouse wheel event handling bug in the media viewer
- reduced db overhead of the 'loading x/y' results generation routine. this _may_ help some users who had very slow media result loading
- cleaned up how the server reports a bootup-action error such as 'cannot shut down server since it is not running'–this is now a simple statement to console, not a full error with trace
- improved client shutdown when a system session shutdown call arrives at the same time as a user shutdown request–the core shutdown routine should now only occur once
- fixed an issue with thumbnail presentation on collections that have their contents deleted during the thumbnail generation call
- misc wx->Qt layout conversion improvements
- updated the github readme to reflect some new links and so on
- misc code cleanup
next week
I pushed it a bit hard this week, so I am going to try to take it easy. Hopefully I will be able to get some volume/mute controls going for mpv and then launch it for all users, including on the Linux and macOS builds. Assuming there are no catastrophic problems this week, I will be overwriting user settings to move all 'show as normal' video and audio options to 'show with mpv'. Then these new imageboard parsers and some more Qt cleanup work.
Hentai Project
Due to the recent take down of ExHentai, is pretty alarming that the same might happen to similar sites. So I propose to anons using Hydrus to save the doujins they like within the platform and tag them along so whenever Hydrus reaches an stage where sharing is easy, maybe you'll have not only a decent alternative to Sad Panda, but a lasting one.
Numerous ideas have been thrown around about it, but here is what the BO have confirmed:
1) Hydrus is not currently great at handling paged media. It can do it, but the workflow is a little awkward. I plan to improve this in future.
2) IPFS support is prototype, and mostly for advanced users who are already comfortable with the program. I'm working on it now.
3) Hydrus is not for everyone. If you have 10,000+ files and want to manage them better than files and folders, you may like it. If you want a clean and professional experience with beautiful UI, you may not.
So it's nothing but a project now. The main reason I do the thread is to keep anons up to support it and use Hydrus for managing whatever manga they downloaded to be used later when the project get to the point it can be used and to post resources and the such.
As so, here are a couple.
An index with tags and names:
https://mega.nz/#!dBxmUADB!4piJItp7ja7_9WbTmoLGo3nMLOP2Fr0AH_ju9W4-PLY
Relevant boards:
Maybe one day, a true alternative to ExHentai can be done by us, for us.
Version 379
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v379/Hydrus.Network.379.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v379.tar.gz
Happy New Year! Although I have been ill, I had a great week, mostly working on a variety of small jobs. Search is faster, there's some new UI, and m4a files are now supported.
search
As hoped, I have completed and extended the search optimisations from v378. Searches for tags, namespaces, wildcards, or known urls, particularly if they are mixed with other search predicates, should now be faster and less prone to spikes in complicated situations. These speed improvements are most significant on large clients with hundreds of thousands or millions of files.
Also, like how system:inbox and system:archive 'cancel' each other out, a few more kinds of search predicate will remove mutually exclusive or redundant predicates already in the search list. system:limit predicates will remove other system:limits, system:audio/no audio will nullify each other, and–I may change this–any search predicate will replace system:everything. I have a better system for how this replacement works, and in the coming weeks I expect to extend it to do proper range-checking, so a system:filesize<256KB will remove a system:filesize<1MB or system:filesize<16KB or system:filesize>512KB, but not a system:filesize>128KB.
downloaders
I have started on some quality of life for the downloader UI. Several of the clunky buttons beneath the page lists are now smaller icons, you can now 'retry ignored' files from a button or a list right-click, any file import status button lets you right-click->show all/new in a new page, and the file import status list now lets you double-click/enter a selection to show that selection in a new page.
I have rolled in a fixed derpibooru downloader into the update. It seems to all work again.
With the pixiv login script confirmed completely broken with no easy hydrus fix in sight, if you have an 'active' record with the old, now-defunct default pixiv login script, this week's update will deactivate it and provide you with a note and a recommendation to use the Hydrus Companion web browser addon in order to login.
the rest
m4a files are now supported and recognised as audio-only files. These were often recognised as mp4s before–essentially, they are just mp4s with no video stream. I have made the choice for now to recognise them as audio-only even if they have a single frame 'jpeg' video stream. I hope to add support to hydrus for 'audio+picture' files soon so I can display album art better than inside a janked single-frame video.
The 'remove' and 'select' menus on the thumbnail right-click have been improved and harmonised. Both now lay out nicely, with file service options (like 'my files' vs 'trash' when there is a mix), and both provide file counts for all options. Support for selecting and removing from collected media is also improved.
full list
- downloaders:
- the right-click menus from gallery and watcher page lists now provide a 'remove' option
- gallery and watchers now provide buttons and menu actions for 'retry ignored'
- activating a file import status list (double-clicking or hitting enter on a selection of rows) now opens the selection in a new page
- file import status buttons now have show new/all files on their right-click menus
- on gallery and watcher pages, the highlight, clear highlight, pause files, and pause search/check buttons are now smaller bitmap buttons
- as the old default pixiv login script is completely broken, any client with this active will have it deactivated and receive an update popup explaining the situation and suggesting to use Hydrus Companion for login instead
- updated the derpibooru downloader
- .
- search:
- when search predicates are added to the active search list, they are now better able to remove existing mutually exclusive/redundant predicates:
- - system:limit, hash, and similar to predicates now remove other instances of their type
- - system:has audio now removes system:no audio and vice versa
- - any search predicate will remove system:everything (see how you feel about this)
- improved 378's db optimisation to do tag searches in large file domains faster
- namespace search predicates ('character:anything' etc…) now take advantage of the same set of temporary file domain optimisations that tag predicates do, so mixing them with other search predicates will radically improve their speed
- wildcard search predicates, which have been notoriously slow in some cases, now take full advantage of the new tag search optimisations and are radically faster when mixed with other search predicates
- simple tag, namespace, or wildcard searches that are mixed with a very large system:inbox predicate are now much faster
- a variety of searches that include simple system predicates are now faster
- integer tag searches also now use the new tag search optimisation tech, and are radically faster when mixed with other search predicates
- system:known url queries now use the same temporary file domain search optimisation, and a web-domain search optimisation. this particularly improves domain and url class searches
- fixed an issue with the new system:limit sorting where sort types with non-comprehensive data (like media views/viewtime, where files may not yet have records) were not delivering the 'missing' file results
- improved the limit/sort_by logic to only do sort when absolutely needed
- fixed the system:limit panel label to talk about the new sorted clipping
- refactored tag searching code
- refactored namespace searching code
- refactored wildcard searching code and its related subfunctions
- cleaned all mappings searching code further
- .
- the rest:
- m4a files (and m4b) are now supported and recognised as separate audio-only mp4 files. files with a single jpeg frame for their video stream (such as an album cover) should also be recognised as audio only m4a for hydrus purposes for now. better single-frame audio support, including functional thumbnails and display, is planned for the future. please send in any m4a or m4b files that detect incorrectly
- the remove thumbnail menu has been moved to a new, cleaner file filtering system. it now presents remove options for different file services and local/remote when available (most of the time, this will be 'my files'/'trash' appearing when there is a mix), including with counts for all options
- the select thumbnail menu is also moved to this same file filtering system. it has a neater menu, with counts for each entry. also, when there is no current focus, or it is to be deselected, the first file to be selected is now focused and scrolled to
- for thumbnail icon display and internal calculations, collections now _merge_ the locations of their members, rather than intersecting. if a collection includes any trash, or any ipfs members, it will have the appropriate icon. this also fixes some selection-by-file-service logic for collections
- import folders, export folders, and subscriptions now explicitly only start after the first session has been loaded (so as well as freeing up some boot CPU competition, a quick import folder will now not miss publishing a file or two to a long-loading session)
- the subscription manager now only waits 15s before starting first work (previously, the buffer was 60 seconds)
- rearranged migrate tags panel so action comes before destination and added another help text line to clarify how it works. the 'go' confirmation dialog now summarises tag filtering as well
- tag filter buttons now have a prefix on their labels and tooltips to better explain what they are doing
- the duplicate filter right-center hover window should now shorten its height appropriately when the pairs change
- fixed a couple of bugs that could appear when shutting down the duplicate filter
- hackily 'fixed' an issue with duplicates processing that could cause too many 'commit and continue?' dialogs to open. a better fix here will come with a pending rewrite
- dejanked a little of how migrate tags frame is launched from the manage tags dialog
- updated the backup help a little and added a note about backing up to the first-start popup
- improved shutdown time for a variety of situations and added a couple more text notifications to shutdown splash
- cleaned up some exit code
- removed the old 'service info fatten' maintenance job, which is not really needed any more
- misc code cleanup
- updated to Qt 5.14 on Windows and Linux builds, OpenCV 4.1.2 on all builds
next week
Next week is a 'medium size job' week. Now I am more comfortable with Qt, I would love to see if I can get an MPV window embedded into hydrus so we finally have legit video+audio support. I can't promise I can get anything but a rough prototype ready for 380 for all platforms, and there is a small chance it just won't work at all, but I'll give it a go.
Hydrus had a busy 2019. Starting with the jump to python 3, and then the duplicate storage and filter overhaul, the Client API, OR search, proper audio detection, the file maintenance system, multiple local tag services, tag migration, asynchronous repository processing, fast tag autocomplete, and all the smaller improvements to downloaders and UI workflow and latency and backend scheduling and optimisations for our growing databases, and then most recently with the huge Qt conversion. The wider community also had some bumps, but we survived. Now we are in 2020, I am feeling good and looking forward to another productive year. There are a couple of thousand things I still want to do, so I will keep on pushing and try to have fun along the way. I hope you have a great year too!
Version 378
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v378/Hydrus.Network.378.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v378.tar.gz
I had a great, simple week. Searches are less likely to be very slow, and system:limit searches now sort.
all misc this week
I identified a database access routine that was sometimes not taking an optimal route. Normally it was fine, but with certain sizes or types of query, it could take a very long time to complete. This mostly affected multi-predicate searches that included certain tags or system:duration and system:known urls, but the routine was used in about 60 different places across the program, including tag and duplicate files processing. I have rewritten this access routine to work in a more 'flat' way that will ensure it is not so 'spiky'.
Also in searching, I managed to push all the 'simple' file sorts down to file searches that have 'system:limit'. If you search with system:limit=256 and are sorting by 'largest files first', you will now see the 256 largest files in the search! Previously, it would give a random sample. All the simple sorts are supported: import time, filesize, duration, width, height, resolution ratio, media views, media viewtime, num pixels, approx bitrate, and modified time. If you want something fun, do a search for just 'system:limit=64' (and maybe system:filetype) and try some different sorts with F5–you can now see the oldest, smallest, longest, widest, whateverest files in your collection much easier.
There are also some fixes: if you had sessions not appearing in the 'pages' menu, they should be back; if you have had trouble with ipfs directory downloads, I think I have the file-selection UI working again; 'remove files when trashed' should work more reliably in downloader pages; and several tag and selection lists should size themselves a bit better.
full list
- if a search has system:limit, the current sort is now sent down to the database. if the sort is simple, results are now sorted before system:limit is applied, meaning you will now get the largest/longest/whateverest sample of the search! supported sorts are: import time, filesize, duration, width, height, resolution ratio, media views, media viewtime, num pixels, approx bitrate, and modified time. this does not apply to searches in the 'all known files' file domain.
- after identifying a sometimes-unoptimal db access routine, wrote a new more reliable one and replaced the 60-odd places it is used in both client and server. a variety of functions will now have less 'spiky' job time, including certain combinations of regular tag and system search predicates. some jobs will have slightly higher average job time, some will be much faster in all common situations
- added additional database analysis to some complicated duplicate file system jobs that adds some overhead but should reduce extreme spikes in job time for very large databases
- converted some legacy db code to new access methods
- fixed a bug in the new menu generation code that was not showing sessions in the 'pages' menu if there were no backups for these sessions (i.e. they have only been saved once, or are old enough to have been last saved before the backup system was added)
- fixed the 'click window close button should back out, not choose the red no button' bug in the yes/no confirmation dialogs for analyze, vacuum, clear orphan, and gallery log button url import
- fixed some checkbox select and data retrieval logic in the checkbox tree control and completely cleared out the buggy ipfs directory download workflow. I apologise for the delay
- fixed some inelegant multihash->urls resolution in the ipfs service code that would often mean a large folder would lock the client while parsing was proceeding
- when the multihash->urls resolution is going on, the popup now exposes the underlying network control. cancelling the whole job mid-parse/download is now also quicker and prettier
- when a 'downloader multiple urls' popup is working, it will publish its ongoing presented files to a files button as it works, rather than just once the job is finished
- improved some unusual taglist height calculations that were turning up
- improved how taglists set their minimum height–the 'selection tags' list should now always have at least 15 rows, even when bunched up in a tall gallery panel
- if the system clock is rewound, new objects that are saved in the backup system (atm, gui sessions) will now detect that existing backups are from the future and increase their save time to ensure they count as the newest object
- short version: 'remove files from view when trashed' now works on downloader thumbs that are loaded in from a session. long version: downloader thumb pages now force 'my files' file domain for now (previously it was 'all local files')
- the downloader/thread watcher right-click menus for 'show all downloaders xxx files' now has a new 'all files and trash' entry. this will show absolutely everything still in your db, for quick access to accidental deletes
- the 'select a downloader' list dialog _should_ size itself better, with no double scrollbars, when there are many many downloaders and/or very long-named downloaders. if this layout works, I'll replicated it in other areas
- if an unrenderable key enters a shortcut, the shortcut will now display an 'unknown key: blah' statement instead of throwing an error. this affected both the manage shortcuts dialog and the media viewer(!)
- SIGTERM is now caught in non-windows systems and will initiate a fast forced shutdown
- unified and played with some border styles around the program
- added a user-written guide to updating to the 'getting started - installing' help page
- misc small code cleanup
next week
I am going to take a few days off for the holiday and make the next release in two weeks, for New Year's Day. I expect to do some small jobs, push more on the database optimisation, continue improving the UI layout code, and perhaps put some time into some space-clearing database maintenance.
𝕸𝖊𝖗𝖗𝖞 𝕮𝖍𝖗𝖎𝖘𝖙𝖒𝖆𝖘!
Version 377
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v377/Hydrus.Network.377.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v377.tar.gz
I had a good week. I mostly caught up with my smaller jobs queue, just pushing on work that had piled up.
Qt
Thank you for the continuing Qt bug reports. I fixed a variety of typo-broken buttons this week, mostly buried in less-used UI, and if you have set a browser launch path override in the options, hyperlinks across the program should obey that again. I also believe I fixed the annoying issue where media viewer hover windows that needed to shrink (because of switching to a new media that had fewer info lines or known urls) would linger too tall for one frame. EDIT: I'm still having slight hover window resize flicker in my IRL client when I keep my mouse over the top-right hover, I'll give it another go next week.
Furthermore, all non-menubar menus across the program now open on right-click release instead of press. This sounds trivial, but there was a mix of down/up click handling here, now unified, and some things like thumbnail and tag selection are handled more naturally in the initial click down. This also fixes an issue where users who had menu styles with no significant border were sometimes accidentally activating the first menu item on a slow click (the click down would show the menu, the click up would activate it).
Last week's support for styles and stylesheets for Qt seems to have gone well. There is more to do here, but this week brings better error handling for unusual situations, better 'we already have this set, no need to waste CPU re-applying it' detection, and fewer hacky colour overrides, so custom stylesheets should be able to colour regular panels without lots of unintended grey or white boxes.
the rest
The shortcut system should now be able to support any text key. This includes extended characters like ø or æ. If you need to apply a modifier to get these keys (e.g. ctrl+alt+e), these modifiers are likely to show up spuriously in the shortcut's label (e.g. ctrl+alt+æ), but everything should nonetheless line up in the program correctly. Let me know if you discover any new bugs here!
There are also some new shortcut commands this week, for the 'media viewer' set. They are 'pan_top/bottom/left/right_edge' and 'pan_horizonal/vertical_center'. They move the whole image around to edges, or center it again, and should be helpful if you often look at tall or wide comics at 100% zoom. There will be more to come here in coming weeks.
The work on making the UI smoother is also continuing. Session saving and a variety of 'media refresh' events that occur on file import or maintenance are now significantly faster and do the bulk of that work in the background. Users with very large sessions should experience less juddery behaviour during both idle and heavy work periods.
full list
- qt:
- all non-menubar menus across the program now launch on click release. some previously launched on click press. a variety of related click event behaviour is cleaned up, particularly with thumbnail/tag selection on the click down. this also fixes some users' menus immediately activating the first entry on slow clicks in some ui styles
- I think I fixed the annoying single-frame delayed size-down resize on media viewer hover frames when changing media!
- the vast majority of old wx panel background colour hacks are removed, so custom stylesheets should now cover much more of the UI
- improved the new custom style and stylesheet setting, resetting, and error handling code, particularly for not re-applying the same style or stylesheet twice, and for handling un-re-settable styles (seems to be defaults initialised by third-party OS-wide Qt style) gracefully
- fixed hyperlinks not using the custom web browser launch path as set in the options
- fixed the 'migrate entire db' and 'set thumb location' buttons in the migrate database dialog
- fixed a typo bug when launching the url selection tree after adding an ipfs directory to download
- fixed two typo bugs when editing regex favourites and simple downloader formulae
- fixed an issue where custom shortcut sets could not be deleted
- fixed a typo in the edit account type panel
- fixed sorting the login listctrl when there are session logins mixed with non-session logins
- removed some old media viewer hover window display/raise hacks
- retired the 'always show hover windows' debug mode
- the media viewer will no longer perform any drag calculations on anything but left-click drag
- misc Qt code refactoring/cleanup
- .
- url searching:
- the database now stores 'known url' domain information more efficiently. it will take a few moments/minutes to reshape the db when updating
- system:known url's exact url search now runs extremely fast. this will only affect new predicates of this type, not those in existing sessions
- system:known url's domain search now runs much faster and matches subdomains of the given domain. this will only affect new predicates of this type, not those in existing sessions
- system:known url's url class search now runs much faster. this will only affect new predicates of this type, not those in existing sessions
- when entering a regex system:known url predicate, the dialog will now not OK (throwing up an error dialog) if the regex is invalid
- .
- the rest:
- the shortcut system now allows all text characters. if it has text, it should work, but it is the wild west in terms of modifier labelling. anything unusual on your keyboard like ctrl+alt+e to make æ will _display_ as ctrl+alt+æ, but the same key combination will match up in the program all correct
- added shortcut actions 'pan_top_edge', 'pan_bottom_edge', 'pan_left_edge', 'pan_right_edge' to the media viewer shortcut set that will move the current image so the respective edge is aligned with the larger canvas's
- added shortcut actions 'pan_horizontal_center' and 'pan_vertical_center' to do as above but center on that axis
- session save now hangs the UI significantly less, whether triggered by user command or auto-saving 'last session'
- saving of last/exit sessions on client close is a little faster
- the call to refresh thumbnail file info (and redraw if needed) when a file is imported or has metadata-regenererating file maintenance done will now only call for files that are actually loaded, run faster per file, run faster when the client has large collections in its session, and not hang the ui thread when waiting for the new media info to arrive
- like regular popups, modal popups (like those created when big vacuum/analyze jobs jump in) will now only appear if the main gui or an on-parent child has OS focus
- the main gui/on-parent child OS focus test now includes misc child windows like the autocomplete results hover window
- network jobs that fail for one reason or another will now be more reliably cleaned up, and their connections returned to the connection pool. this may fix the 'too many open file handles' errors some users were seeing after long term unreliable network traffic
- fixed an issue where some thumbnails that were trashed or physically deleted were being removed from 'all known files' and file repository views when it was not appropriate
- connection and downloader retry time options now have a wider min/max range when in advanced mode, with an accompanying warning label for the connection panel
- checker options times now have a wider min/max range when in advanced mode, with an accompanying warning label
- cleaned up some shutdown reporting text
- misc debug improvements
next week
The holidays are coming up. As 379 would normally be due on Christmas Day, I will postpone that to New Year, which means next week's 378, on the 18th, will be the last release of the year. I am due for a 'cleanup' week, which will also make for a nice 'safe' release before the break. So I will clean up code, do some more Qt neatening, and otherwise catch up on smaller jobs like I did this week.
I never strictly marked down the day when I started working on hydrus, but December 14th, 2011, the date of the first non-experimental beta, has emerged as a decent birthday. This week marks eight years. It has been a lot of work and a lot of fun. There are a couple thousand things I would still like to do, so if I can, I would like to keep going as I have into the 2020s. I deeply appreciate all the feedback, help, and support over the years. Thank you!
If you would like to further support my work and are in a position to do so, my simple no-reward Patreon is here: https://www.patreon.com/hydrus_dev
Release Tomorrow!
I had a good week. I focused on catching up on smaller work. More bugs are fixed, there's some neat new shortcuts to pan media around, the UI should be smoother when importing files and saving sessions, and the system:known urls predicates now run much faster.
The release should be as tomorrow.
Version 376
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v376/Hydrus.Network.376.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v376.tar.gz
I had an ok to good week. Subscriptions run better and there are more UI fixes and basic theming for Qt.
subscriptions
Subscriptions have always been resource heavy. As last year's downloader overhaul extended their capabilities, we saw increased CPU lag with many subscription operations. I had hoped to completely fix this this week, even permitting subs with >250,000 items cached, but the surgery needed to figure this out was more complicated than I expected. While I otherwise prepped for this next step, I also tightened up subscription timing code and wrote a new scheduler that reduces overall subscription resource needs.
Subscriptions will now try to load when they are due, and only then. So, if a query in your sub is due to check in 45 minutes, or if it has file to download but has run out bandwidth for 45 minutes, it should now run, typically, in 45-60 minutes. They will also respond better to cancelling and pause events, and if only 30 of 100 queries are due for a check in this run, or have outstanding files to download, you will now see x/30 on the popup progress text and gauge, just as better indication of how much work remains.
Previously, subs checked every four hours. It meant a bunch of wasted CPU for subs that were not due, late checks for subs that were due just moments after the check time, and also a delayed 'clumping up' of work. Today's change is subtle but significant, so I am interested in feedback from all users of subs–if you notice anything odd, bad, or great, please let me know. Ideally, subs will be quieter for you, running more often when you are not looking.
Although the data overhaul was too much work to do in one week, I now have a good plan and am in better position to get it done. I really want subs to load and save their metadata in milliseconds so the manage subs dialog can launch equally quickly and the client api can start talking to them. The year is fading, so I am not sure when this will get done, but hopefully this week's scheduling will at least relieve some of the work involved.
Qt
Hydrus now has very simple support for Qt styles. Please check the new options->style page. All users are encouraged to try out the 'style' option, which should have one or more flavours for your OS and also 'Fusion'. These will change how buttons and other widgets are drawn across the program.
Advanced users may wish to play with 'stylesheet' support, which uses css/qss files to change colours and layouts using portable files. This is still prototype, just rough importing of stylesheets from a barebones folder in the install dir, and is likely to be replaced with something more user-friendly and technically sensible in the coming weeks. If you play with this, I am interested in areas where hydrus's old colour system conflicts foolishly with a qss stylesheet (such as some panel backgrounds and the green/red yes/no buttons on many dialogs), and also recommendations for stylesheets you would like bundled as future hydrus defaults.
The macOS build can now do page tab drag and drop again! Media drag and drops between tabs should also work. Everything should line up right and work like the Windows and Linux builds, but please let me know if it does not. A tip here, if you still have trouble with a centered page tab bar, is just to set the Fusion style as above, which will make hydrus look more like a standard PC application. You may prefer the macOS look, but if not, you now have the option.
Beyond a bunch of general bugfixes, I also improved the stability of some review bandwidth panels and some bad 'should I close/exit?' dialogs that were either not respecting 'no' answers or causing unusual behaviour as 'yes' closes were going on.
the rest
Pixiv tag search and twitter username lookup get updated default downloaders this week. Pixiv should be fixed, and twitter should work more efficiently, only getting 'media' tweets.
Client shutdown should now be faster and snappier for most users.
File export filename phrases now support a new '{file_id}' term that substitutes a unique permanent numerical identifier. If you need to create guaranteed-unique filenames, this is easier than using the '{hash}' term.
full list
- subscriptions:
- wrote a new subscription manager to better look after subscription scheduling
- rather than checking every four hours or after manage subs dialog close, subscriptions now record an indication of when they are next due for work, whether that is the estimated next check time or when bandwidth is free on remaining file downloads, and launch in a fifteen-minute window around that time. delays due to previous errors or user cancels are also taken into account. this reduces background cpu and i/o greatly for clients with large subs
- if a sub is paused, or all its queries are paused, it will now never be reloaded after first load until a change via the manage subs dialog
- furthermore, if a single sub takes a very long time to work, the whole sublist can re-cycle if they come up due for more work before it is finished
- if a sub query is DEAD but still has outstanding files to download, it will no longer automatically pause
- subs now clean up more tidily if they are running on a program exit
- the subscription popup now shows check/file progress based on the number of queries that appear to have pending work. instead of 'query 300/450' with 420 that aren't due, you'll get 'query 12/30'. if a query becomes due during a round of checking, another round of checking will run
- if a subscription fails to load from the db, the error is handled better and no more subs will run in that boot
- improved subscription startup checking logic, tightening up various paused/dead/cansync tests
- improved subscription interrupt checking logic, tightening checks on global network pause and various shutdown scenarios
- cleaned up some more subscription code in prep for data storage breakup
- .
- qt:
- added experimental Qt style settings to the new options->style page! all users should now be able to set Fusion style, and perhaps some alternate OS styles. advanced users are invited to play around with QSS stylesheets (although be warned that some of hydrus's custom colour system overrides QSS, so more work is needed here), which will be extended and made user-friendly in coming weeks
- fixed tab position calculations for all tab/media drag and drops for tab bars that are centered or otherwise positioned far off top-left alignment
- fixed tab drag and drop event object handling for macOS. tab and media DnD is now enabled for macOS
- the popup toaster can now unhide if an on-top-of-parent non-modal frame (like review services) is focused (so hitting 'process now' should show you the work)
- fixed a variety of old hacky wx close-window veto tests. the 'close client?' confirmation dialog will now reliably veto a close requent on 'no'/cancel, dialog close events that are vetoed (such as closing the manage tags dialog with pending tags) will now veto more than just the first time, and several bad media viewer archive/dupe filtering cancel and end-of-window events should now work more cleanly and correctly. users who had crashes at the end of filtering may find they are stable again
- as a quick patch against some multiline notes and statuses, list controls now force single-line text in all cells
- list controls now tooltip all cells
- fixed the shutdown splash not updating after the daemons shut down (lmao)
- 'modal' message dialogs, which are created by blocking maintenance tasks such as vacuum, will no longer raise the program to the foreground on creation
- should have fixed the taglist vertical positioning jank that could occur in the row after a tag with a tall emoji unicode character (and also sometimes kanji/hangul)
- fixed a typo bug that was throwing an error for the upnp port widget in the local client server management panel when 'allow non local connections' was checked
- improved stability of bandwidth review panel bandwidth rules refresh
- improved stability of review services bandwidth rules refresh
- improved some dialog cleanup code
- reverted a bad environment-setting change put in last week that was causing some running-from-source users trouble
- misc qt code cleanup
- .
- the rest:
- updated the default pixiv tag search downloader to one submitted by a user. it now uses their api
- updated the default twitter username lookup to a downloader submitted by a user. it fetches just the media tweet feed, making it more efficient. also added (but not linked by default) is a new tweet parser that can fetch most videos using a third-party site, advanced users may wish to play with this
- added a {file_id} term for file export phrases that substitutes a unique and permanent numerical file identifier
- fixed an issue where idle maintenance jobs could sometimes sneak in a few milliseconds of work during certain long shut down pauses, such as while waiting for a 'should I do shutdown work?' dialog to return. program shutdown should be snappier for many users as forced startup delays in these calls will no longer trigger
- added a date 'encode' string transformation rule, which takes an integer timestamp and converts it to a pretty date string. the date rules are now renamed to the clearer 'datestring to timestamp' and vice versa
- fixed page parser edit panel's 'test parse' when string transformations perform pre-parsing conversion. the handling and passing of test data for all the panels here is cleaned up throughout
- system:limit predicate edit panel now has a small label describing its sampling behaviour
- updated the various 8chan links in the client and help to 8kun, let me know if I missed any, and added Endchan bunker link to help menu
- improved some misc status text handling across the program
- refactored cache and manager code into different, simpler files
- updated sqlite on windows build to 3.30.1
next week
It is already December! For me, and I know for many of you, 2019 has just disappeared. Christmas is exactly three weeks away, so I am afraid some of my hopes for significant tag overhaul like tag siblings or processing filters may have to be delayed a little. I have a bit more Qt work to do, but also plenty of regular small jobs and bugfixes, so for next week I will take a simple bit of misc work, nothing huge, just to catch up on some things. I also have messages to catch up on.
If you were an 8chan user and missed the news, the /hydrus/ board is back on 8kun.top. The site is still unstable, and reception from the community is mixed, but against all odds they are back up and improving. I will keep posting release posts on Endchan and 8kun simultaneously for the time being, so please use whichever you feel better about, or just hit me on discord/tumblr/email if you prefer.
Version 375
windows
macOS
app: https://github.com/hydrusnetwork/hydrus/releases/download/v375/Hydrus.Network.375.-.macOS.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v375.tar.gz
I had a great week. There are a bunch more Qt fixes, and a few other things as well.
Qt
I have fixed a bunch more bugs in the Qt code. We are getting to the end now–this is mostly smaller stuff like an unusual dialog button not working, but I have fixed another important memory leak that was causing some backend not to be deleted correctly when a media viewer closed on a video. This should radically reduce memory use for some heavily used clients.
Some windows that were large, or could expand to be, like the options dialog on some pages, were sizing off the edge of the screen. This should be fixed now, and a variety of child-window initial size calculations should be a bit more accurate–Qt manages window size a little differently than wx, and the additional buffer represented by the window title frame and border was not being taken into account.
The menubar menus should work a little snappier this week. Things like the rapid 'pending (xxx)' menu updates when a downloader is importing files with tags. The whole UI should get a little latency benefit from this during these high-traffic times.
I also gave the layout and scaling another go on clients that use UI scaling (when you tell your OS to display UI at >100% on a monitor) on high dpi displays. The experiment from last week did not go how I wanted, with pixelly scaled-up thumbnails, so I have reworked that and fixed the taglist, which was crunching tags together on these displays.
Release Tomorrow!
I had a great week. I have fixed some more Qt problems, including some bad window positioning, thin popup message widths, a video memory leak, and another round of improvements for high-dpi displays. Also are some improvements to the file maintenance system for recovering (including auto-redownloading) missing files, some better network retry timing code, and new downloaders for some broken sites.
The release should be as normal tomorrow. I will be making a release post here as well as Endchan.
Version 362
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v362/Hydrus.Network.362.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v362.tar.gz
I had a mixed week. The duplicates overhaul work is finished.
duplicates work finished
The duplicates storage overhaul is done! Everything is now on the new storage system, the duplicate filter has had some quality of life attention, and now there is some updated help:
https://hydrusnetwork.github.io/hydrus/help/duplicates.html
If you have used hydrus for a bit but haven't checked out the duplicate system yet, this is a good time.
I added some final new things this week: the duplicates filter now highlights common resolutions, like 720p and 1080p, and there is a new 'comparison statement' for common ratios like 4:3 or 16:9. Also, the thumbnail right-click file relationships menu now provides a choice to clear out potential relationships and, if in advanced mode, perform some en masse remove/reset actions on multiple thumbnails at once.
I am now free to move on to another large job. Audio support was very popular at the last vote, so I will spend a couple of weeks trying to get some simple 'has audio' metadata going, but then I am going to address some growing issues related to tag repositories, easier 'I want these tags' management, namespace siblings, and multiple local tag services.
deepdanbooru plugin
If you are an experienced user and interested in testing out some neural net tagging, check this out:
https://gitgud.io/koto/hydrus-dd/
This project by a hydrus user lets you generate tags using the DeepDanbooru model and get them into hydrus in a variety of ways. If you give it a go, let me know how it goes and what I can do to make it work better on the hydrus end.
full list
- duplicates work finished:
- updated the duplicates help text and screenshots to reflect the new system
- duplicate files search tree rebalancing is now done automatically on the normal idle maintenance routine, and its over-technical UI is removed from the duplicates page
- the duplicate filter's resolution comparison statement now specifies 480p, 720p, 1080p, and 4k resolutions and highlights resolutions with odd (i.e. non-even) numbers
- if the files are of different resolution, a new 'ratio' comparison statement will now show if either have a nice ratio, with current list 1:1, 4:3, 5:4, 16:9, 21:9, 2.35:1
- added a 'stop filtering' button to the duplicate hover frame
- made the ill-fitting 'X' button on top hover frame a stop button and cleaned up some misc related ui layout
- added a 'remove this file's potential pairs' command to the thumbnail file relationships menu
- if in advanced mode, multiple thumbnail selection right-click menus' file relationships submenus will now offer mass remove/reset commands for the whole selection. available commands are: 'reset search', 'remove potentials', 'dissolve dupe groups', 'dissolve alt groups', 'remove false positives'
- .
- the rest:
- added link to https://gitgud.io/koto/hydrus-dd/ , a neat neural net tagging library that uses the DeepDanbooru model and has several ways of talking to hydrus, to the client api help
- cleaned up a little of the ipfs file download code, mostly improving error/cancel states
- rewrote some ancient file repository file download code, which ipfs was also using when commanded to download via a remote thumbnail middle-click. this code and its related popup is now cleaner, cancellable, and session-based rather than saving download records to the db (which caused a couple of edge-case annoyances for certain clients). I think it will need a bit more work, but it is much saner than it was previously
- if you do not have the manage tags dialog set to add parents when you add tags, the autocomplete input will no longer expand parents in its results list
- fixed an issue displaying the 'select a downloader' list when two GUGs have the same name
- hitting apply on the manage parsers or url classes dialogs will now automatically do a 'try to link' action as under manage url class links
- fixed (I think!) how the server services start, which was broken for some users in 361. furthermore, errors during initial service creation will now cancel the boot with a nice message, and the 'running … ctrl+c' message will appear strictly after the services have started ok the first time, and services will shut down completely before the db is asked to stop
- improved how the program recognises shutdowns right after boot errors, which should speed up clean shutdowns after certain bad server starts
- the server will use an existing server.crt and server.key pair if they exist on db creation, and complain nicely if only one is present
- the 'ensure file out of the similar files system' file maintenance job result will now automatically remove from/dissolve the file's duplicate group, if any, and clear out outstanding potential pairs
- a system language path translation error that was occuring in some unusual filesystems when checking for free disk space before big jobs is now handled better
- like repository processing, there is now a 1 hour hard limit on any individual import folder run
- fixed an issue where if a gallery url fetch produced faulty urls, it could sometimes invalidate the whole page with an error rather than just the bad file url items
- subscriptions will now stop a gallery-page-results-urls-add action early if that one page produces 100 previously seen before urls in a row. this _should_ fix the issue users were seeing with pixiv artist subs resyncing with much older urls that had previously been compacted out of the sub's cache
- until we can get better asynch ui feedback for admin-level repository commands (lke fetching/setting account types), they now override bandwidth rules and only try the connection once for quicker responses
- misc code cleanup
next week
I was unable to get to the jobs I wanted to this week, so I think I'll go for a repeat: updating the system:hash and system:similar_to predicates to take multiple files and extending the Client API to do cookie import for easier login. And I'll play around with some audio stuff.
Release Tomorrow!
I had an ok week. I did some of the last duplicates work, including matching zooms for files with different resolution ratios and adding a neat pixel-for-pixel duplicate detector, polished some of the logic for the new tag autocomplete search, and cleaned and fixed a bunch of code.
The release should be as normal tomorrow.
Version 361
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v361/Hydrus.Network.361.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v361.tar.gz
I had an ok week. Some final duplicates work is done, and there is some polishing to the new tag autocomplete.
duplicates
The duplicates filter can now detect if static images with the same resolution are pixel-for-pixel duplicates! If they are, it gives one of the standard 'comparison' statements on the right hover window. Furthermore, if one file is a png and the other not, this statement will colour green/red and bias heavily to the non-png, since the png is likely a bloated 'clipboard' duplicate that you don't want. Pixel summary data is not cached long-term, so this routine takes a bit of extra CPU. It only kicks in if both files are images with the same res, but nonetheless please let me know if this makes the duplicate filter too laggy for you. I expect the new 'pixel hash' data will be cached at the db level in future to auto-resolve png/not-png dupes like this.
Also, the duplicate filter will now match the two files' zooms even if their resolution ratio differs! Zoom is now locked so the two files' widths are matched, along with the files' top-left corners. Two files with resolution 1920x1080 and 1912x1080 will now line up pretty good even if you zoom and pan.
And I have written a new system predicate for whether a file is the best file of its duplicate group (also called the 'king'). It provides an easy way to find only the best files, or only those that are not the best. It is bundled with the old 'num duplicate relationships' system predicate under the new 'system:file relationships' predicate.
I also fixed an issue with the 'custom action' button that wasn't letting custom actions go through unless some file was deleted–the final deletion question dialog now has a 'delete neither' choice, which is the default.
And if you want to feel some despair, Mr Bones now reports potential, duplicate, and alternate counts.
The duplicates storage overhaul is pretty much done. There is plenty more I could do, but I have now finished the main db focus of the work. Beyond some final UI stuff, there is only some decent new help to write. I would like to have that done next week, so I can draw a line under this job. It was more work than I expected, but I am overall really happy with it.
tag autocomplete
Last week's autocomplete changes seem to have overall gone pretty well. However, being able to search so fast has revealed some old 'limiters' I had in place to stop certain super laggy searches going ahead. And some old wildcard logic was flawed. Now we have more power with this control, I have been able to clean it up a bit.
First off, entering something like 'character:*' should now work everywhere (although it will likely lag a whole bunch once the final results come in). Also, searches with an explicit namespace, like 'character:ara', will now match 'character:samus aran', just as the simpler 'ara' does. Wildcard searches, like 'char*:sam' or 'c*r:*mus' should be a bit more sensible overall, finding more possible results and matching complicated queries more reliably.
And 'media' autocomplete fetches, which happens on a search page when you start typing with thumbnails already loaded, should also be much faster now (they were lagging last week with high numbers of thumbs). I improved the media search efficiency and added similar 'cancel' tech as I did to the db search last week, so it should now be fairly smooth and fast, even up to 10,000 files in view.
the rest
Just a note: system:size is now system:filesize.
Also, if you are a Linux user, or I have otherwise previously suggested you turn on the options->media->BUGFIX: Load images with PIL option, please check that option again this week and try turning it off. There was a time when PIL was more reliable than OpenCV–the other image library I use–but now things seem to be the other way around, and OpenCV is significantly faster too.
A user reported to me recently that he had an external hard drive he had hydrus installed to die due to overheating. It looks like it was related to heavy hydrus import folder work. This is the first time I have heard of something like this, but it still concerns me a lot. If you are running from a drive that can get similarly very hot, I strongly recommend you ensure you are not running any very heavy, hour-long-plus import or repository processing jobs on it. In the meantime, I will write some pause/throttle options for all the big routines to help users reduce load according to their situations.
full list
- duplicates:
- the duplicate filter now compares the pixel content of static image pairs of the same resolution–if they have the exact same pixels, a comparison statement is added, and if one file is a png and the other not (i.e. the png is likely a useless clipboard copy), the statement notes this and a strong duplicate score is applied
- added 'system:is/is not best file of its group' to search for file kings
- renamed 'system:num duplicate relationships' to 'system:num file relationships'
- wrapped the two file relationship system predicates into one 'system:file relationships' stub predicate that opens to a dialog with two pred panels
- added a 'add potential pairs' command to the thumbnail right-click file relationships menu, which will force-queue files for the duplicates filter
- the duplicate filter now ensures the two medias' zoom is locked so they have the same width through a transition. furthermore, their current dragged top-left position is pinned in the same location. this ensures files that have slightly different resolution ratios (especially when they are just a couple of pixels off) still remain reasonably comparable when switching back and forth
- reworked and simplified how position/drag delta is handled in the media canvas to support the above
- fixed the 'custom action' button on the duplicate filter, which had no 'delete neither' choice and whose 'forget it' button cancelled the whole custom operation, making it impossible to custom action without deleting something. I have added a 'delete neither' green-text button to the front, as the default action
- mr bones now reports on your potential, duplicate, and alternates numbers
- .
- tag autocomplete:
- greatly sped up tag autocomplete search when fetching from a current media view (i.e. from thumbnails in the search page)–it had some CPU-inefficient testing/counting that mattered at high media/tag counts
- greatly improved cancelability of tag autocomplete search when pulling from a current media view–this was resulting in high lag when typing fast with multi-thousand results
- fixed the gui-level tag matching test to match namespaced search inputs with offset subtags (e.g. 'character:aran' now matches 'character:samus aran'), both for wildcard and specific namespaces
- when typing an explicit wildcard tag search that does not end in a *, you will now be presented with two wildcard options–one with the implicit * suffix, one without
- fixed 'write' tag autocomplete inputs (like in manage tags) being able to search for chunky 'namespace:*' explicit wildcard searches
- .
- the rest:
- fixed the ipfs nocopy path translation control saving rows for client file paths outside of the main install path for non-Windows, where it was forgetting on save
- renamed 'system:size' to 'system:filesize'
- sped up some system:inbox searches
- disabled a PIL 'load truncated images' backup mode, which on the current version can seemingly lead to infinite load hangs
- file report mode now prints info when it deletes/recycles a path, including stack traces
- fixed a long-running and silent 'port already running' bug related to setting services on the server that was stopping successful service-set-restart from the client in many situations. 'port is already running' checks that conflict with other processes will now give an immediate error to the client without saving any changes
- the server now prints to the log as it stops/starts/has started its services
- improved how the server can report certain 500 errors
- the 'critical service tag/file reference' repository processing error has been improved: rather than reset the whole repository, it now pauses the repo and resets processing status for just the repo's 'definition' update files (without deleting any existing entries, so they should ultimately reprocess super fast) and also schedules a complete integrity and metadata check for all updated files
- keyboard interrupts from the console should now trigger a clean exit request for the client
- polite and forced shutdown requests when logging off should now trigger a fast exit (i.e. no yes/no dialog, no shutdown maintenance, but otherwise session saved and so on) for the client. this fast exit is noted in the log
- moved the tag and rating service listctrls in duplicate merge options panel to the new listctrl object
- moved the manage regex favourites listctrl to the new object
- updated a bunch of yes/no dialogs to the new panel system
- deleted some old unused dialog code and related unit tests
- fixed up deletion-and-reimport file location handling for lingering media objects, which were not correctly forgetting combined local file deletion record on the reimport
- improved shutdown error handling during repo processing
- deleted the mishimmie default downloader
next week
The duplicates help, is the top thing. I need to draw some diagrams, take some new screenshots, and brush up the existing text to better explain the new system. After that, I will catch up on small jobs. I'd love to have multiple system:hash search added (for searching a bunch of md5s, say), maybe some subscription thumbnail publishing cleanup, and perhaps some Client API work, where I'd like to have web browser cookie import for easy login.
Once the duplicate work is done, I expect to do a little work on audio support, likely basic 'has audio' metadata for files, and then crash on some client/tag-repository & PTR overhaul.
Version 360
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v360/Hydrus.Network.360.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v360.tar.gz
I had a great week. There are a bunch of little things and an important speed overhaul to tag autocomplete.
tag autocomplete
I have never really liked the tag autocomplete workflow. It once blocked the UI completely, and some unusual timing options were needed to make it even useable, but it still often lagged out or just responded with gigantic lists in judders. After chipping away at the problem, this week I am finally updating it to what I really wanted.
So, the main change is that autocomplete results are now fetched as soon as you type. It responds very quickly and overall, I think, feels great, particularly once you have put four or five characters in. You get what you want as soon as you type, and can hit enter right away. The db 'job' to fetch results is now completely divorced from the UI and is able to cancel much faster when you type a new character, so the artificial fetch-start latency requirements of the old system are no longer needed.
There are now just two options for tag autocomplete, under options->speed and memory: whether tag autocomplete should fetch as you type (default on), and a character threshold to switch from 'exact match' autocomplete searches to 'full' ones. This 'exact match' defaults to 2 characters, and means if you type 'sa', you will only get results for 'sa' rather than 'sa*', which is generally a pretty giant laggy list, but if you type 'sam', you'll really be searching 'sam*' and get all matching autocomplete tags. You can reduce or increase this 'exact match' threshold as you prefer, or turn it off completely and get full autocomplete for any input.
Also, I have added some quality of life features. If results do not appear within 200ms, you'll now get a 'loading results…' label in the dropdown for feedback, and any 'static' results, such as exactly what you typed for a manage tags dialog input, or the special 'namespace:*anything*' for a search input, will appear immediately so you can select them without having to wait. Also, entering 'character:sa' will now trigger the same smaller 'exact matches' test as for the unnamespaced 'sa', rather than searching for the whole giant list of tags beginning 'character:sa*' as happened previously.
Furthermore, a query like 'char' no longer matcher 'character:samus aran'. This was a neat idea, but it proved too unwieldy IRL and is now better served by wildcard queries such as 'character:*' or 'char*:*'.
Overall, I am very pleased with the change. Please give it a go, and you'll see the difference immediately. However, tag autocomplete is a complicated system with different workflows, and I have made some big changes, so if you are, say, the sort of user who types fast and just hits enter, and you find the new results are a bit 'flickery' or something for you, let me know and I'll see if I can smooth it out.
more file maintenance
I brushed up the new file maintenance UI under database->maintenance->review scheduled file maintenance to have a nicer list and have added three new jobs:
One fixes file permissions on Linux and OS X. For a while, due to an oversight, file imports have been getting 600 permissions on Linux and OS X. I have fixed this now to be 644 (so, for instance, a nocopy ipfs instance or a network file share running on another user can access the files) for all new files, and this new file maintenance job will retroactively try to fix the permissions of existing files.
The other two are for the duplicates system. One explicitly regenerates similar files metadata, and the other checks if a file is improperly in or out of the similar files searching system, and fixes it if it is in the incorrect place (scheduling metadata regen as appropriate). Both are mostly for debug purposes but will get real use when I eventually add videos to the duplicates system, which will be a giant CPU job we'll want to spread out with the nice new non-blocking file maintenance pipeline. The old maintenance button on the duplicates page that kept eligible files 'up to date' is now gone as a result, and any outstanding jobs there (although most users shouldn't have any by now) should be migrated to the new file maintenance system on update.
full list
- tag autocomplete:
- after various tag autocomplete async work, fetch timings get a complete overhaul this week. the intention is for a/c jobs to appear as fast as possible, with good ui feedback, without interrupting ui while they work. feedback on how this works IRL would be appreciated
- there are now just two autocomplete options under options->speed and memory:
- - whether autocomplete results are ever fetched automatically, defaults to true
- - the max number of characters in the input that will cause just exact results vs. full autocomplete results, defaults to 2, can be None
- namespaces are no longer searched from an unnamespaced query ('char' no longer matches 'character:samus aran'). this proved too slow for real use, and remains better available with explicit namespace searches such as 'character:' or 'char*:*'
- the 'exact results' character limit now also applies to subtags of namespace searches! so, entering 'character:a' will deliver the same short exact match results as just 'a'–no more gigantic lists when you put in a simple namespace
- improved tag results caching to deal with the new non-namespace matching on subtag input
- tag autocomplete dropdowns will now display a non-selectable 'loading results…' label when results take more than 200ms to load.
- tag autocomplete dropdowns will now also display 'static' tags, such as 'namespace:*anything*' for 'read' inputs and the exact entered text and possible siblings/parents for 'write' inputs, during loading. so, typing 'character:' just to get the special 'character:*anything*' predicate is now simple and does not need a whole load wait to enter!
- cleaned up some tag listbox code to handle parent selection and navigation better along with the new label type
- greatly improved autocomplete search logic in the critical text search portion, collapsing it into one cleverer and more easily cancellable query rather than two or three simpler ones with potentially gigantic lists thrown back and forth
- improved speed of autocomplete cancel for certain large lists with many siblings
- .
- file maintenance:
- the new file maintenance ui now shows scheduled jobs in a listctrl, and only shows jobs that have outstanding work. you can clear/do work on multiple selected jobs
- the file manager should now try to guarantee at least 644 permission on file imports (previously, it was only trying to add 600, which lead to problems with nocopy ipfs running on another user etc…)
- added a file maintenance job to check and fix file permissions
- added a file maintenance job to regenerate similar files metadata
- added a file maintenance job to check if a file should be in the similar files system–if it should and isn't, it is queued to get its metadata data regenerated, and if it is and shouldn't be, it is removed
- the previous bulky similar files metadata regen job from the duplicates page is now removed, and any outstanding scheduled regen will be transferred to the new file maintenance manager on update
- .
- client api:
- added POST /manage_pages/focus_page, which makes the given page the current page in the main gui
- added help and unit tests for this new call
- client api is now version 9
- .
- the rest:
- fixed an issue recording media viewtimes when no max viewtime is set
- fixed the new missingdirectory errors not printing the missing path
- fixed an issue with some human-started repository actions waiting silently on bandwidth when it was not intended (e.g. account refresh)
- export folders now raise proper errors and pause themselves if their path is not set, does not exist on the file system, or is not a directory (previously, they silently stopped work without error)
- cleaned up some misc import folder code, and put in additional protections to the delete/move code to ensure folders cannot be so actioned if they somehow end up in the path import queue
- when unpinning a file or directory from ipfs, the clientside service now first checks that the current daemon considers it pinned (previously, this 500 errored when the object was not pinned due to a reinitialised daemon etc…)
- fixed an issue with the new ipfs path translation control, which was forgetting values when the clientside path was outside of the default db structure
- media objects that transition from trashed to physically deleted but remain in view will now correctly be aware of their complete previously-deleted status (rather than being simply remote, as they were before until a client restart)
- improved some of the recent duplicates db update code to pre-optimise the new tables on update (some users were getting slow behaviour due to mis-scheduled analysis maintenance)
- extended the new panel system to deal with custom button panels and moved the duplicate filter 'commit and continue?' dialog to the new panel system
- moved the archive/delete and duplicate filter 'commit and finish' dialog to the new panel system
- wrote a new question panel for the typical yes/no dialog used across the program and started a cleanup job to migrate all 140-odd instances of this over
- fixed an issue where a program instance that quit due to a user deciding to leave an already running instance in place would clear the original instance's 'running' file in its shutdown, meaning subsequent runs would charge ahead and hit 'database is locked' problems on db init!
- wrote a new 'similar files metadata generation report mode' to provide debug info on this cpu/gpu intensive routine
- added 'why use sqlite?' entry to the help faq, with a link to prkc's excellent document about the subject, https://gitgud.io/prkc/hydrus-why-sqlite/blob/master/README.md
- also added prkc's excellent Linux package requirements information to the 'running from source' help page
- fixed some old py 2.7 references in running from source help and an old link in ipfs help
- moved the 'file viewing statistics' menu down on the database menu
- fixed some dialog Escape key event handling
- fixed some ui ancestory testing code
- improved some misc similar files system code
next week
Next week is a cleanup week. I'd like to just do some boring rewrites and ui code updates, and I'll see if I can hack away at the last duplicates overhaul jobs.
How do you tag? Metholodogy discussion thread
To quit spamming the Q&A thread (sorry) I agree this should be made into its own thread.
I'd like this to be a place to ask questions on "how should I tag this" for those that don't use the PTR and think you may be able to be more intelligent with how you tag. Also a place to discuss how you tag things and, most important, WHY that way. For example, what sort of issue were you looking to solve or what did it make easier by doing it that way?
T
o start off the discussion - I'm going to write a small essay on my thoughts on tags/tagging. It is my belief that tagging should be, above all else, two things
1) Simple - There are few scenarios where you must think if something should be tagged or not. Edge cases and oddities should be minimized at every opportunity. The number of tags should be kept small enough to maintain the entire list in memory. This is important for consistency. Speaking of which, the second item is…
2) Consistent - Tags should be well defined and used consistently. If a tag is often forgotten to be applied then it is a bad tag and should be either removed from the system or renamed/redefined as to be applied more consistently. In order to keep consistent tags it is also a requirement that all tags are able to be kept in memory to be applied properly. If you have so many tags that you are incapable of remembering them all then you will likely be tagging inconsistently. It is okay to add new tags to cover areas.
A tag that is used infrequently because the number of items in the gallery are too few is acceptable but is discouraged if the term is rarely searched for (either as an inclusive or exclusive term). Remember: tags are meant to help you search for images. This is important when it comes to naming tags.
I use a collection of 288 unnamespaced tags and 11 namespaces. The tags are mostly descriptors of clothes, physical traits, and objects. Namespaces are used for more meta information like character: artist: series: or whether a piece of art is SFW or pornographic.
Some of the more heavily opinionated choices I've made in regards to my tags include:
The tag "dog" is for actual animals and anthropomorphic characters are tagged using a namespace of "species:". There are no "doggirl", "catgirl", "mousegirl" tags, but "species:dog", "species:cat" instead. This rule was made because of Kemono Friends, Strike Witches. Because the rull is for all anthropomorphic characters, I end up with humorous scenarios like Upotte! and Girls Frontline using species:gun and Azur Lane using species:submarine, species:cruiser, etc. Along these lines, "cat ears" is not a tag that is used. "animal_ears" is the tag for all sort of animal ears and to specify which kind of animal ears, include a species: tag in the search.
The series:original does not exist. Any artwork that lacks a series: tag is assumed to be an original work, otherwise it would have a series: tag saying which series it is from.
Character names never include the series in parenthesis. This is what the series tag is for. To search for a specific character who shares the name of a character from another series - you include the series: tag in the search.
The series is always the name of the first season - even if a character comes from season two which has a different name. This is primarily because of Fate/Grand Order but also series like THE iDOLM@STER. This was done for simplification purposes - as remembering which season/spinoff of a show a character is from leads to mistakes and inconsistent tagging.
Underscores are always used for spaces. This is done so that wildcard searches function properly.
My namespace for SFW/NSFW is "Lewdness" and has 3 levels: SFW, NSFW, Pornographic. NSFW is defined as any visible nudity so very lewd art is still "SFW" as long as it does not contain nudity.
The cutoffs between "white", "silver", and "grey" are defined with a hexadecimal color code and the darkest color of the hair is sampled to determine the cutoff.
Release Tomorrow!
I had a great week. As well as fixing bugs and catching up on small jobs, I put some time into the new file maintenance manager and significantly reworked and improved the search timings of the tag autocomplete system. Tag results now come in as fast as they can–it feels completely different.
The release should be as normal tomorrow.
Version 359
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v359/Hydrus.Network.359.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v359.tar.gz
I had an excellent and very full week.
file maintenance
The new file maintenance system now has some nice UI under database->maintenance->review scheduled file maintenance. It has two pages. It is a little advanced, but if you have some hydrus experience, please feel free to have a play around with it.
The first page reviews what is currently scheduled to run. It shows job counts for each type and lets you manually start work on that job type or cancel them.
The second lets you schedule new jobs. It uses the standard file search interface, so you can queue up a thumbnail regen for all webms with a certain tag, say, or recheck file metadata on all pngs (just in case they were truly apngs). There is also an advanced button to run talks on repository update files.
Furthermore, the new file maintenance manager now handles file integrity checks. The old 'file integrity check' entries under the database menu, which would check all files in your system in one go, are gone, now handled on the nicer system that doesn't interrupt work if you force it and will otherwise chip away at any large job in smaller pieces in idle time. Missing files or files will incorrect data will now automatically have any known URLs exported to .txt files, and incorrect files will also be exported, all to an appropriate directory beneath your main db dir.
I would like to add saved file searches within the next month or so, at which point I will attach some recommended jobs here, such as detecting apngs and correcting some webms that were once detected as mkvs.
duplicates
The thumbnail right-click menu now lets you reset or undo file relationships! If a file has duplicates, you can remove it from that group, and if it has alternates, you can pull it from that. You can also completely 'dissolve' the groups and clear out false positive relationships. It gives you little yes/no explanatory dialogs for each action, if you aren't quite sure what you mean to do.
I expect to add commands to apply these sorts of operations to entire selections in the next few weeks.
The duplicates storage overhaul is coming to a close. I have a few more cleanup jobs like this, some filter interaction improvements, and some proper help to write. We are getting there!
ipfs nocopy
I have added an advanced and experimental 'nocopy' IPFS pin mode to the client. This permits IPFS to share files straight from your hydrus file store, without having to make a copy. If you are not familiar with this, I recommend you not try. I would like feedback on this from the users who are enthusiastic about it so I can iterate on it and make it easier for regular users in future.
Essentially, the IPFS review/manage services panels now have a bit of nicer UI to talk to the daemon. You can check that 'nocopy' is turned on and enable it under manage services. Unfortunately, nocopy will only work on any file that is beneath the parent of the main ipfs conf directory, which is typically your USERDIR, so if your hydrus client_files folder is above that, or on another drive entirely, you will need to remap the locations with some symlinks(!) before it will work. The manage services panel has some help on it, but please feel free to ask me if you need some more.
I tested this, and it appears to work, but I am not a big IPFS user, so I would appreciate anyone who is giving this a proper shake-down and letting me know what you think. I'd love to add some easier plug-n-play IPFS sharing ability to the client, but we'll need to get over some hurdles first.
I expect to write a new IPFS downloader or similar that will pull multihash data from public http gateways so users can get into this more without having to set up a daemon.
deviant art
Deviant Art seem to be rolling out a complete site redesign that breaks the existing login script and gallery & file page parsers. Much like a recent Pixiv rollout, not all users are being affected at the same time–it looks like users who remain logged in from a previous session are not seeing the update as quickly.
I have written completely new objects for DA for this update. There is a new login script and parsers, and now there is a new downloader just for tags. Unfortunately, I think you need to be logged in–or otherwise have some particular cookies established from previous activity–to access the new download searches, or else you will get 404. If you are logged in and get this 404, try resetting your login and then logging back in. Dragging file page URLs straight on to hydrus seems to work fine, it is just this gallery search step that needs a 'new' session set up before it works.
I am afraid I have also had some CloudFlare 500 server error results with the new search. I suspect DA hasn't completely rolled out their new tech yet or something weirder is happening. These gallery 500s can pause the downloader hydrus-side annoyingly, so I will write some ui next week to let you try again quicker. In any case, the new downloaders I am putting out today are an improvement on 'it doesn't work at all', but they may not be everything we need. If you are a keen DA user, please let me know how you get on.
the rest
I fixed the 'file lookup script' GET problem. I apologise for the inconvenience.
I am also rolling out an improvement to the shimmie parser this week that pulls source time and md5 hash.
full list
- ipfs nocopy:
- wrote a new panel to better show ipfs daemon status and added it to the review and manage ipfs service panels
- added nocopy config review and enable status and buttons to this new panel
- added an EXPERIMENTAL 'use nocopy' checkbox to the ipfs manage services panel
- added accompanying WEWLAD path translation ui to enable nocopy when your hydrus media storage paths are inaccessible to the ipfs daemon for nocopy purposes. a help button explains this more–it currently needs some symlinking, so non-advanced users should stay away
- if everything is set up, ipfs nocopy seems to work! I am not totally happy about the setup required here, so feedback from advanced ipfs-fluent users would be appreciated and we can iterate on this
- improved stability of ipfs daemon/version checking code
- .
- file maintenance:
- wrote some proper file maintenance ui under database->maintain->review scheduled file maintenance!
- for existing work, the new file maintenance ui shows how much work is scheduled for each job type and lets you cancel that work or run it manually
- for new work, the new file maintenance ui lets you queue up work of any type for files you select with the standard tag autocomplete search interface! you can schedule all pngs to be rescanned in case they are truly apngs, or regen thumbs for all files imported before a certain date, or whatever you wish. you can also queue up repository update files
- the file maintenance manager can now deal with repository update files when it does a complete file metadata regen
- the file maintenance manager now takes responsibility for checking file presence and file integrity. the old 'check file integrity' options under database->maintenance, which did all files in one go, are now gone
- file integrity checks will now always export broken files and missing/broken files' known urls to .txt files to your db_dir/missing_and_invalid_files. appropriate popups and log data will be sent as well. also, the known urls will be both exported on a per-file .txt basis and appended to one unified .txt
- if a file now fails to parse on a metadata reparse, it is now automatically checked for file data integrity
- if a repository encounters a missing, invalid, or incorrect filetype update during update processing, it now schedules all updates in the repo to be appropriately rescanned by the file maintenance manager
- if the storage subdirectory directory does not exist on a client file path request or thumbnail-add attempt, a special error will now be raised with instructions to reconnect the location or shut the client down immediately
- cleaned up some ffmpeg mime-detection logspam
- .
- duplicates:
- added several single-file thumbnail right-click dissolve/reset duplicate actions:
- - reset search status
- - remove from duplicate group (if in one and not the king)
- - dissolve duplicate group (if in a group)
- - remove from alternate group (if in one)
- - dissolve alternate group (if in one)
- - clear false-positive relations (if it has some)
- added some new code to deal with dissolution and member extraction at the db level
- when a member is extracted from alternate group, its constituent files are now requeued for potential search
- multi-selection duplicate right-click actions are now available to non-advanced-mode users
- wrote some unit tests for the new dissolve/reset actions
- cleaned up some misc duplicates code
- .
- the rest:
- fixed a recent bug in the file lookup script GET call–I apologise for the mistake
- the main gui page tab menu now lets you sort page tabs by the number of files they have
- deviant art seem to be rolling out a new page format. this week hydrus introduces completely new deviant art downloader objects that, fingers crossed, will update any existing users smoothly and also provide new tag search functionality. users who are still logged in may still be getting the old page format. if this is you, and this update does not work (although I _think_ it should, even so), please try clearing your existing login and logging in again
- new deviant art login script, artist + tag GUGs, gallery url classes, file and gallery parsers
- updated the shimmie file page parser to pull source time and md5
- improved the 'process now' advanced button to only focus on actual specific outstanding processing. previously, it was also checking for new metadata when due, which, when the server was not available, could seemingly idle for a time before actually processing updates due to the new delaying connection retry code
- wrote a new 'file import report mode' mode to help->debug->report modes
- fixed a progress display issue with the janitorial petitions processing page
- improved accuracy of sibling and parent petition counts, and properly capped them at 1000
- mapping petitions are now grouped by namespace, and will come in more manageable chunks
- fixed the server launch-and-init test debug code
- misc string-to-string control improvements to support the new ipfs edit ui
- removed the old 'continual tag archive sync' legacy code from tag services, which has been semi/non-functional for a long time
- cleaned up the annoying separator hanging on the end of certain tag right-click menus
- cleared out the 'Exception ignored in' spam that is often printed after the log closes
next week
I pushed it a bit hard this week, so I will take it easier as I catch up on a variety of smaller jobs. I also need to catch up on my messages.
Version 358
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v358/Hydrus.Network.358.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v358.tar.gz
I had a great week doing duplicates work and fixing bugs.
duplicates
I split this big duplicates storage overhaul into three jobs, and this week marks the third and final job done. Like alternates and duplicates information, potential pairs are now stored in a unified and more efficient way.
On the front end, you may notice your potential pairs queue shorten again on week. It will also shrink faster as you process in the duplicate filter, which will present more 'useful' duplicate pairs first and apply your decisions more intelligently at the db level.
All the code is simpler except for one key area. If you notice certain the 'show some random potentials', duplicate filter or potential counts load time take way too long, please let me know about your situation. It is possible I will have to revisit this complicated 'join', although in my tests it is performing well.
Also, I have written the new record to stop alternate pairs coming up in the duplicate filter repeatedly, as some users have experienced. These relationships are now more concrete, and this concreteness is plugged into duplicate merge operations and so on. You may see one more round of alternates appearing, and then they will be saved properly.
Now everything is stored on the new system, there are two main jobs remaining: re-adding various administrative commands like remove/dissolve to properly undo relationships, and adding some options and improvements to the duplicate filter workflow.
the rest
Pixiv changed their format recently, so hydrus's default parser broke. This should be automatically fixed this week. Thanks to a user who sent in this fix.
The issue where mouse scroll events were not being caught when a media viewer did not have focus is also fixed.
The 'watcher' page now reports file and check status in the 'status' column! I missed this somehow when I added it for the gallery downloader. This makes it just a little easier to see what a list of threads is currently doing.
I may have fixed the problem where exiting manage tags from a media viewer sometimes falls focus back to the main gui. Please let me know if you still get this (and if so, if you know a way you can reliably repeat this behaviour).
I improved some of the network engine's 'this connection ended early' checks. This may have fixed some issues users had downloading images and page data from some unreliable servers, but if it does not, please send me any incomplete jpegs and the URLs they came from so I can check further on my end. Also, the whole system is more strict about response lengths now, so if you discover false-positive network failures here, please report them.
Also some server issues related to last week's client api authentication improvements (such as file repository file upload sometimes breaking) should be fixed.
new client api library
If you would like to work on the Client API using Node.js, check out the new module a user wrote here:
https://github.com/cravxx/hydrus.js
This is now in the help along with the rest of the API here:
https://hydrusnetwork.github.io/hydrus/help/client_api.html
full list
- duplicates:
- the final large data storage overhaul work of the duplicates work big job is done–potential duplicate information is now stored more sensibly and efficiently. potential pair information is now stored between duplicate file groups, rather than files themselves. when duplicate file groups are merged, or alternate or false positive relationships set, potentials are merged and culled appropriately
- your existing potential data will be updated. the current potential pairs queue size will shrink as duplicate potential relationships are merged
- the duplicate filter now presents file kings as comparison files when possible, increasing pair difference and decision value
- potential pair information is now stored with the 'distance' between the two files as found by the similar-files search system. the duplicate filter will serve files with closer distance first, which increases decision value by front-loading likely duplicates instead of alts. distance values for existing potential pair info is estimated on update, so if you have done search distance 2 or greater and would like to fill in this data accurately to get closer potentials first, you might like to reset your potential duplicates under the cog icon (bear in mind this reset will schedule a decent whack of CPU for your idle maintenance time)
- setting alternate relationship on a pair is now fixed more concretely, ensuring that in various search expansions or resets that the same pair will not come up again. this solves some related problems users have had trying to 'fix' larger alternate groups in place–you may see your alternates compared one last time, but that should be the final go. these fixed relationships are merged as intra-alternate group members merge due to duplicate-setting events
- a variety of potential duplicates code has been streamlined based on the new duplicate group relationship
- improved how a second-best king representative of a group is selected in various file relationship fetching jobs when the true king is not permitted by search domain
- one critical part of the new potential duplicates system is more complicated. if you experience much slower searches or count retrievals IRL, please let me know your details
- expanded duplicates unit tests to test potential counts for all tested situations
- fixed a bug where alternate group merging would not cull now-invalid false-positive potential pairs
- the rest:
- updated the default pixiv parser to work with their new format–thank you to a user for providing this fix
- fixed the issue where mouse scroll events were not being processed by the main viewer canvas when it did not have focus
- file page parsers that produce multiple urls through subsidiary page parsers now correctly pass down associated urls and tags to their child file import items
- updated to wx 4.0.6 on all built platforms–looks like a bunch of bug fixes, so fingers-crossed this improves some stability and jank
- updated the recent server access-key-arg-parsing routine to check access from the header before parsing args, which fixes an issue with testing decompression bomb permission on file POST requests on the file repository. generally improved code here to deal more gracefully with failures
- the repositories now max out at 1000 count when fetching pending petition counts (speeding up access when there are large queues)
- the repositories now fetch petitions much faster when there are large queues
- frames and dialogs will be slightly more aggressive about ensuring their parents now get focus back when they are closed (rather than the top level main gui, which sometimes happens due to window manager weirdness)
- rewrote a bad old legacy method of refocusing the manage tags panel that kicks in when the 'open manage tags' action is processed by the media viewer canvas but the panel is already open
- hitting 'refresh account' on a paused service now gives a better immediate message rather than failing after delay on a confusing 'bad login' error
- improved login errors' text to specify the exact problem raised by the login manager
- fixed a problem in the duplicates page when a status update is called before the initial db status fetch is complete
- the manage tag siblings panel now detects if the pair you wish to add connects to a loop already in the database (which is a rare but possible case). previously it would hang indefinitely! it now cancels the add, communicates the tags in the loop, and recommends you break it manually
- added a link to https://github.com/cravxx/hydrus.js , a node.js module that plugs into the client api, to the help
- a variety of user-started network jobs such as refreshing account and testing a server connection under manage services now only attempt connection once (to fail faster as the user waits)
- the 'test address' job under manage services is now asynchronous and will not hang the ui while it waits for a response
- fixed some unstable thread-to-wx code under the 'test access key' job under manage services
- improved some file handling to ensure open files are closed more promptly in certain circumstances
- fixed some unstable thread-to-wx communication in the ipfs review services panel
- improved the accuracy of the network engine's 'incomplete download' test and bandwidth reporting to work with exact byte counts when available, regardless of content encoding. downloads that provide too few bytes in ways that were previously not caught will be reattempted according to the normal connection reattempt rules. these network fixes may solve some broken jpegs and json some users have seen from unreliable servers
- fixed watcher entries in the watcher page list not reporting their file and check download status as they work (as the gallery downloader does)
- the client api will now deliver cleaner 400 errors when a given url argument is empty or otherwise fails to normalise (previously it was giving 500s)
- misc cleanup
next week
I had hoped to do some IPFS work this week, but I ran out of time to do it properly. This is now the main job for next week. Otherwise, I will do some of this final duplicates work and some misc small jobs.
Release Tomorrow!
I had an excellent and very full week. I have extended the new file maintenance system and written some UI to manage it, added some undo/reset functions for the new duplicate system, fixed a variety of bugs, and figured out an advanced-users-only extention to the IPFS service to permit the experimental 'nocopy' pin mode,
The release may be late tomorrow, as there is more than usual to test.
Version 357
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v357/Hydrus.Network.357.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v357.tar.gz
I had a great week fixing bugs and cleaning up other small jobs.
shortcut and hover window fixes
The recent menubar open/close detection did not work well on OS X and some versions of Linux. It misfiring broke some shortcut processing and hover window display. The good news is that as I went into that code more closely this week, I discovered several other menu, shortcut, and hover window problems that were causing long-term issues. All clients should have less janky hover window placement and behaviour, but particularly on non-Windows.
Specifically, the OS X shortcuts-in-media-viewer issue is fixed. Also, on all platforms, hover windows should size and place better, with less flicker, and handle mouse and keyboard interaction more sensibly. A small memory-leak that was affecting OS X and some Linux versions is also cleared up, which is likely to improve some increasing menu-lag and other jank. I know there is more work to do here for certain Linux window managers, so continued feedback would be appreciated here.
Semi-relatedly, the advanced 'custom' shortcut sets are now hidden from non-advanced-mode users under file->shortcuts.
the rest
The popup toaster (on the bottom-right, where messages appear) now has a collapse/expand button! So, if your subscriptions suddenly appear and get in the way of your browsing, you can now finally hide them. As a result, the little 'dismiss all' summary bar now always appears when you have some messages up.
The duplicate filter's new jpeg quality comparison now has customisable 'weight' scores under options->duplicates. Also, it seems I messed up the default values here, swapping high vs very high values. I have fixed the defaults and set them to reset this week so all users get decent values. If you prefer these values different, please check them again after update.
Several places across the program that have 'write' tag autocomplete inputs, such as manage tag siblings and parents, now have paste buttons beside the tag input box. They let you enter tags straight from the clipboard, including the standard newline-separated format for entering multiple tags at once. It is now much easier to set up twenty tags to have a certain parent.
Downloaders now report their 'added' time without seconds, so '12 minutes' instead of '12 minutes and 23 seconds'. The continue text updates here were making large fresh download lists pretty flickery, so this should calm things down a bit.
When bandwidth is blocked, rather than just saying 'yes', the network->review bandwidth usage window now reports how long until bandwidth is free!
full list
- client api:
- the client api can now receive the access key through a GET or POST parameter rather than the header
- the client api now supports GET /session_key, which provides a temporary key that gives the same access as its permanent access key with the Hydrus-Client-API-Session-Key name through header or GET/POST param. it expires after 24 hours of inactivity or if the client is restarted
- the GET /manage_pages/get_pages call now returns the unique 'page_key' identifier that will be useful in future page management when multiple pages share a name
- the POST /add_urls/add_url command now takes 'destination_page_key' to exactly specify which page you would like a URL to end up on. if the page is not found, or it is the incorrect type, the standard page selection/creation rules will apply
- cleaned up some serverside request processing code
- cleaned up some misc client api permission checking code
- updated client unit tests to check the new changes
- updated client api help to reflect the new changes
- cleaned up some GET and POST parameter parsing
- client api version is now 8
- .
- shortcut and hover window fixes:
- moved the canvas shortcut processing code more towards the new shortcut system
- the OS X shortcut-in-media-viewer issue, which was being boshed in a similar way to the main gui last week, should now be fixed
- when the hover windows have focus, they now pass shortcuts up to the canvas parent more reliably
- removed a legacy menu highlight-tracking system that was malfunctioning and generally throwing a slow-memory-leaking wrench in several places, particularly some non-Windows situations
- the 'menubar is open' test code is now only active for Windows. the other platforms have mixed reliability with menubar open/close events
- some related OS X hover-window flickering and hiding-under the main page problems (having problems due to thinking menus were open) are also fixed
- some hover window flicker on certain focus changes due to clicking focus windows should be fixed
- hover windows now try to size themselves a little better on init, which reduces some initial flicker or false-positive single-frame display on some systems
- extended the hover report mode to report some 'ideal' pos/size info as well
- under file->shortcuts, custom shortcuts are now hidden for non-advanced-mode users
- .
- the rest:
- the popup message toaster now always shows its 'dismiss all' summary bar whenever any messages are being displayed. the summary bar now also has a ▼/▲ button to collapse/expand its messages!
- added duplicate comparison score options (under options->duplicates) for the new jpeg quality estimator
- fixed the default duplicate comparison score values, which appeared to be reversed for higher vs much higher–they will be reset to these new defaults on update, so recheck them if you prefer different
- in manage tag siblings and parents, the filename tagging dialog, and some misc options panels, tag autocomplete input controls now have 'paste' buttons to make entering many results much easier
- to reduce update flicker, the downloader and watcher pages do not list seconds in their 'added' column ('12 minutes 24 seconds ago' is now '12 minutes ago')
- improved clipboard access cleanup on in-clipboard errors, which was sometimes leading to error popups or clipboard lockup
- rather than the simple 'yes', the review bandwidth usage dialog now puts the waiting estimate (like '12 minutes 50 seconds') in the 'blocked?' column
- improved external program launch code for non-Windows to remove hydrus-specific LD_LIBRARY_PATH completely when no OS default exists to restore it to. this should fix ffmpeg connection for certain installs
- fixed a rare bug when initial media results of a page failed to load due to a subset of unexpectedly unfetchable file records
- gave the rare 'ui freezup on dialog close' event yet another pass. closing via escape key should now be immune to this
- 'remote' files that were once in the client but since deleted now have the 'trashed' icon and will state so on their right-click info summary lines
- fixed various instances where selection-from-list dialogs were failing to sort their list based on underlying data object uncomparibility. an example of this was when selecting which queries to pull from a separating subscription
- on the edit parser panels, fetching test data successfully via the quick button or the manual URL entry will now set that URL in the example parsing context
- on the edit parser panels, the subsidiary page parser's separation formula now launches with the correct example data (the original data from the parent dialog, rather than the post-separated data) on which to test separation. this should nest correctly for multiple subsidiary page parsers
- to reduce server load spikes, clientside petition processing now approves very large mapping and file petitions (such as a petition to delete one tag from 50k files) as a sequence of smaller chunks
next week
I want to do two things: moving 'potential' duplicate information to a more efficient data structure and improving some IPFS connection tech, particularly seeing if we can figure out 'nocopy' so IPFS can share without having to copy files to its own data store. These will both be fairly big jobs, so we'll see how it goes.
Release Tomorrow!
I had a great week working on the duplicates overhaul and fixing bugs. Potential duplicate pairs are not stored in a sensible way, with the duplicate filter now serving up more useful comparisons and clearing out the pending potential queue faster. Also, the issue with the media viewer not capturing mouse scrolls while not focused is fixed, as is the default pixiv parser, file repository file upload, and several more specific issues.
The release should be as normal tomorrow.
Version 356
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v356.tar.gz
I had an excellent and busy week catching up after E3. Duplicate better/worse/same relationships are now stored more logically and efficiently, and a variety of bugs are fixed. Maintenance is also easier to control.
duplicates
The second phase of the big duplicates rewrite is now done! Rather than the old system of trying to store better/worse relationships for every possible member of a duplicate group, which is very complicated, requires many human decisions, and is not very useful in practise, this new system instead lumps all duplicate files into a single group with a single 'king' representing the best file of that group.
The groups are transitive, so putting a file into a group with a better/worse/same command–or even merging a group into another–is now just a single action. Rather than comparing a file with every member of a group, it is now just merged as a simple member or the new king. The overall workflow of dealing with large duplicate groups is now much much faster. Furthermore, alternate relationships are stored between groups of duplicates, not individual files, so again the number of choices you need to make is automatically reduced as you process your normal duplicate filter queue.
Searching and managing duplicate file groups is also easier. When searching for system:num duplicate relationships, you simply search for how many 'duplicates' a file has, rather than how many better/worse/same. Right-clicking on a thumbnail to open up the 'file relationships' menu–which is now available to all users–now presents the same simple 'duplicates' count-and-show-in-a-new-page entry and will say whether the file is the best file of its group. If the file is not the best of its group, you can view that king or set the current one as king with one click.
Future updates will show king/not king status in the thumbnail UI. I will also write a new system search predicate to filter for king/not king.
Your existing better/worse/same data will be converted to the new system on update, with appropriate kings chosen. If in recent weeks you noticed some files that seemed to be both alternates and duplicates, this will now auto-resolve as the duplicates are merged together.
The new jpeg quality estimator, which will sometimes present 'high quality vs low quality'-style statements in the duplicate filter, now appears for all users. It isn't perfect, but it works ok.
The last phase of the rewrite will be to update potential pairs. It will reduce the potential pair queue count even further and improve the pair-elimination efficiency of the decisions presented in the filter (essentially by forcing king vs king comparisons). I want to take a week to do some smaller duplicates work (like commands to undo/dissolve existing relationships intelligently), so with luck I expect to have that done for 358. I am overall very pleased with the work here so far, although there is still a good amount to do, and some proper new help to write up.
the rest
I fixed the popup spam that the new clipboard watcher was falsely making on many non-matching URLs. It was some test code that was misfiring–even though the URLs were not being added since they were not matched, a higher part of the system thought they were not being added due to a problem with pages. I am sorry for the confusion–it is now fixed.
Also the unusual OS X shortcuts issue, where shortcuts were not working on some OS X installs until a menu was opened! This was due to a weird OS X specific problem in the recent main gui menubar detection code. If you are on OS X and notice your menubar stops updating, please let me know.
I put a bit of time into the maintenance code this week. Most maintenance jobs, like repository processing and thumbnail regeneration, now use the same 'clock' to figure out if they should be working. They now know if they are running in idle time or shutdown time or forced by the user, and will cancel out faster depending on the situation as needed. The other upshot of this is that the file menu now has a 'exit and force shutdown maintenance' option, and the shutdown splash finally has a 'stop shutdown maintenance' button! I regret that this took so long to come in–but now it is, and it works well!
You can now also force file maintenance from database->maintenance->file maintenance. It tells you if no work needs to be done.
full list
- duplicates:
- moved better/worse/same quality duplicates relationships to the new 'king' group-based model. rather than tracking every relationship, duplicates are now stored in groups with a single 'best' file
- as a result, duplicate relationships are now transitive! saying that one king is duplicate to another will merge groups. the 'better' king is the new king, and 'same quality' kings choose one of the kings pseudorandomly. advanced exceptions: saying that a king is better than a basic member of another group or saying that two members are same quality is still valid but will simply 'poach' the non-king member from the other group in order to ensure the wrong king doesn't end up on top in the eventual merge. saying KingA is same quality as MemberB will merge the groups with KingB as the new king (since it is presumably same/better quality to all members of A)
- the thumbnail right-click 'duplicates' entry is now renamed to 'file relationships' and is no longer advanced mode only. the 'find similar files' entry is folded into this
- the thumbnail 'file relationships' menu now shows a simple 'duplicates' count rather than the old messy better/worse/equal. it will show all the members of a duplicates group when clicked. the menu also notes if the focused file is the best file of its group. if it is not, you will get the option to show the best file or make the focused file the best
- as a result, it is now much simpler to view a group of duplicates and overrule a 'best quality' member as needed
- added the 'media' shortcut 'duplicate_media_set_focused_king' to shortcut setting a 'best quality' file
- the system:num duplicate relationships now has the simpler 'duplicates' entry, to search on size of the entire group. searching for kings/not kings will come soon
- due to the new duplicate transitivity rules, potential pairs are now eliminated at a much faster rate!
- setting duplicate relationships will overrule false positive or alternate relationships already in place
- manually setting alternate relationships to more than two thumbnails at once will now set each file as alternate to every other file in the selection, completely eliminating potential pairs within the group. if you try to do this to large groups of files you will get a longer yes/no confirmation message just to make sure you aren't overwriting some potential dupes by accident
- all existing better/worse/same relationships will be converted to the new group storage in this update, with appropriate kings determined. potential pair queue counts will be reduce accordingly, and the temporary alternate/duplicate confusion from the alternates update will be auto-resolved by merging truly duplicate 'alternates' together
- fleshed out the duplicate test code significantly to handle the new dupe groups and their interactions with the recent false positive and alternates changes
- refactored some db test code into separate client/server/duplicates files and cleaned up dupe tests readability
- potential pairs are now the only component of the new system still on the old pairs system. the duplicate filter will still serve up some inefficient (i.e. non-king) comparisons
- .
- the rest:
- fixed the issue where many clipboard-watcher-caught URLs that did not match were producing false-positive 'could not generate new page for that URL' error popups
- the clipboard text-fetcher now tests against incompatible clipboard types (like a screenshot) better, and all instances of text fetching now report errors more gracefully and with more information
- fixed the unusual OS X issue where many shortcuts were not being processed after client boot until the top menubar was opened and closed. a variety of other blocking-while-menubar-is-open issues that were false-positive misfiring are now fixed as well, please let me know if you still have trouble here
- the file menu now has an 'exit and force shutdown maintenance' option to force-run outstanding maintenance jobs
- when shutdown maintenance work is going on, the shutdown splash screen now has a 'stop shutdown maintenance' button!
- cleaned up some file maintenance manager maintenance locking and shutdown cancel logic
- moved all the idle-mode maintenance checks to a new system that explicitly defines idle/shutdown/forced maintenance work and tests those states in a unified manner, checking idle mode and the new splash cancel button status and so on more reliably. a lot of maintenance should cancel out quicker when appropriate
- misc shutdown logic cleanup
- added a 'file maintenance' option to the database->maintenance menu that forces the new file maintenance manager to run its queue. it'll make a little popup as it works, or a note that no work is due
- the 'regenerate' thumbnail menu is also available to all users
- jpeg quality estimates are now available for all users in the duplicate filter. they only display when the two jpegs' quality have different labels
- the jpeg quality estimator now handles some unusual jpegs that load with empty quantization table arrays
- the duplicate filter now handles bad jpeg quality estimations gracefully
- cleaned up some ffmpeg communication code
- the ffmpeg debug text that spawns on a help->about call that fails to discover ffmpeg version information now prints stderr output as well. if you have been hit by this, please give it another go and let me know what you get
- the same ffmpeg 'no response' error on file parse now popups and prints some debug info and returns a better error
- dialogs and windows on the new panel system now support a new pre-close tidying system
- the manage tags dialog and window will now cancel any pending large tag autocomplete queries on close
- regular gui pages now support a new pre-close tidying system
- search pages will now cancel any pending search results loading or tag autocomplete queries on close
- improved reliability of the popup message manager chasing the main gui when it is sent to another screen by a keyboard shortcut (such as shift+win+arrow on Windows). it should work now if the mouse cursor is in either window. please let me know if this causes trouble for virtual display navigation
- the network engine now waits significantly longer–60s–on connection errors before trying again, and with every failed attempt will wait n times longer again. when in this waiting state, a manual user cancel command cancels it out faster
- I believe I have fixed/improved a situation where media viewer hover windows would sometimes disappear immediately after appearing on some Linux window managers
- improved hover window report mode to state more focus info in case the above is insufficient
- to better link the two requests and consume bandwidth under strict rules more precisely, the override bandwidth rule that kicks in when a file page has a single file is now 3 seconds instead of 30
- updated options->connection page to specify that 'socks4a'/'socks5h' is needed to force remote dns resolution
- sped up tag parents initialisation
- repositories now group tag sibling and parent petitions by the parent/better tag's namespace
- removed some old network 'death time' code that is no longer useful and was interfering with heavy petition processing
- the log now flushes itself to disk every 60s rather than 300s
- misc fixes and cleanup
next week
This week was a crush, so I focused on the duplicates work and bug fixes and cleanup. I have a variety of small 'new stuff' jobs to catch up on next week. Some more shortcuts to add, some client api work, some quality of life like pasting tags into siblings/parents. And some smaller auxiliary duplicates work.
I enjoyed taking a week off, but I am glad to be back to work.
Release Tomorrow!
I had a great week fixing bugs and catching up on other small jobs. Some OS X and Linux UI issues (including shortcuts and bad hover windows in the media viewer) are fixed or improved, some downloader-page flicker is reduced, the client api is moved forward a little, and the popup system now has a neat collapse/expand button to hide subscriptions and so on while they work.
The release should be as normal tomorrow.
Version 355
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v355/Hydrus.Network.355.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v355.tar.gz
I had a good week catching up on small jobs.
duplicates
I added a 'trash file' button to the duplicate filter hover window, tucked up in a new row with the cog button so they fit on thinner windows. If both potential files are worth deleting, hitting this is an easy way to dismiss them immediately. I also fixed some over-aggressive potential pairs removal from last week's alternates update. Users with many alternates may have lost some valid intra-alternate-group potential pairs, which I expect to add back in once this whole overhaul is complete.
The duplicate action buttons (and several others in the media viewer) are now aware of the shortcuts that can do their actions, and they will display these shortcuts in their tooltips! I am quite pleased with how this turned out, and as I slowly work on unifying program shortcuts to the new system, I hope to have this in more places.
Users in advanced mode get a new jpeg quality estimator comparison on the duplicate filter panel–when two jpegs appear to have fairly different quality, you will get a new 'A vs B' statement about it. This estimator is a bit hacked together, not totally precise, so I would be interested in your overall feedback. I expect to revisit it and add some options before I enable it for all users. Advanced mode users also get duplicate merge options back for 'alternates'.
the rest
I wrote a simple clipboard watcher for URL imports! Check it under network->downloaders. You can watch for 'watchable' URLs like imageboard threads and other 'recognised' URLs like booru pages. It checks for new text every second, adds URLs to pages just like a drag-and-drop event (i.e. to currently selected page if it is valid, otherwise the left-most valid page, otherwise making a new page), and can handle newline-separated lists of URLs. It is off by default.
Note that the client isn't happy about making new pages while it is minimised (their layout messes up), so if you clipboard-watch a bunch of URLs and it needs to make a watcher page for them, I have it set now so they will queue up until it is restored. If you leave your client minimised and want to send it URLs, I recommend you have a page set up for clipboard-watched (or Client API-added) URLs to land on until I can improve this.
The file viewing statistics system gets a pass this week–preview and media views now have minimum view times before they are counted, and they also cap to maximum values (so leaving a preview view up won't accidentally add a heap of time like 1 day 3 hours to your total viewtime. It is also now off by default for the duplicates filter. Check the new settings under options->file viewing statistics. If you got hit by some bad numbers before due to some accidental fast or prolonged views, check the new command under database->file viewing statistics->cull file viewing…, which will attempt to compensate for any bad values based on the new min/max caps. A yes/no dialog explains what it does.
There are several new shortcuts to help navigate the manage tags dialog, all under the 'main_gui' shortcut set: Four are called 'show_and_focus_manage_tags_XXX' and let you move keyboard focus to the various 'suggested' tag columns (recent/favourites/file lookup/related). Combined with 'set_search_focus', default Ctrl+S, which jumps you back to the tag autocomplete input, you should now be able to enter suggested tags completely without the mouse. Furthermore the new 'focus_media_window' will move focus back to the media viewer canvas if the manage tags window is launched from one (say for archiving or rating by shortcut without having to close manage tags).
You can now quick-enter 'namespace:' in a search page's autocomplete tag input to get the special (and efficient) 'namespace:*anything*' search predicate.
I fixed the popup-error problems that could come up when fading 'collected' thumbs! I apologise for the inconvenience here and have added some weekly pre-build tests to catch simple problems with collected thumbs in future.
full list
- duplicates:
- retuned the new alternate and false positive setting code to be less aggressive about removing potential pairs. users with alternate groups who updated to 354 may have lost some intra-alternate potential pairs, which I expect to fill back in once the potential pairs system is updated
- the duplicate filter center-right hover window now has a trash button, which is moved to its own row with the cog icon where they will fit on thinner windows better
- users who are in advanced mode now have access to duplicate merge options for 'alternates' again–but be careful with this, it is advanced. if you had merge options for alternates set up before, your old options _should_ return, but make sure to check it
- if you are in advanced mode, the duplicate filter can now show an experimental jpeg quality comparison statement. if this works out, I will add custom scores and improve this otherwise based on feedback , so comments would be appreciated
- .
- client api:
- the client api has a new 'manage pages' permission!
- /manage_pages/get_pages now fetches a structure of the current page session! there's some help for it
- in future, I expect to add a 'select page' command and get/remove/highlight URL(s) for downloader pages, and reveal the unique page identifier for better targeting here and for add_url commands
- client api is now version 7
- .
- the rest:
- fixed a stupid issue related to drawing collection thumbnails that was causing a lot of errors. I apologise for the inconvenience and have added pre-build tests to catch these simple mistakes in future
- added a clipboard url watcher! there are two options now under network->downloaders–one for watcher urls, and one for all other _recognised_ urls (i.e. post and gallery urls, and file urls that have matching url classes). it checks every second, sends urls to pages according to the same rules as a drag and drop event, and will handle newline-separated lists of urls
- when the client is minimised and needs to create a new downloader page because of an 'add url' command through this new clipboard watcher or the client api, it will now queue up the url and its page creation for when the client is next restored (page creation while minimized causes layout bugs). it returns an appropriate result text to the client api in this case
- added four 'show_and_focus_manage_tags_XXX' shortcuts to the 'main_gui' shortcut set that let you select and focus the four possible suggested tags panels on the manage tags dialog. these are slightly special focus events that do some 'first tag selection' work as well and should let you make the whole process keyboard-only
- added a 'focus_media_viewer' shortcut to the 'main_gui' shortcut set that focuses the media viewer from a media viewer's manage tags window
- file viewing statistics gets an update: it now has its own options page where you can set min/max for preview and media viewing times. the global turn on/off control is moved from database menu to this page, as is a new control for enabling it on the duplicate filter (where you will be flicking back and forth and perhaps do not want lots of views recorded). duplicate filter file stats now default to disabled
- added a new command to database->file viewing statistics that will cull your current stats based on the new min/max values to compensate for previous accidental '3day 4hour' preview view that snuck in. a yes/no dialog explains how it works before it goes
- wrote a 'mixin' class to let my buttons automatically note current shortcut mapping information to the bottom of button tooltips
- the duplicate commands and a variety of other media viewer buttons that work in the new shortcut system now report their current shortcuts in their tooltips!
- quick-entering 'namespace:' in a search page's tag autocomplete input will now automatically swap in the special 'namespace:*anything*' predicate
- cleaned up code around setting a custom temp_dir through the "temp_dir" command, and made it test the given path is indeed a directory and writable-to. if not, the program dumps out with an error popup
- the client and server now similarly test that the directory db is a dir and writable-to
- rearranged some critical boot error handling code and made the client's boot error handling throw up some ui
- fixed the client's repairdb function when needing to regen autocomplete caches
- autocomplete regen now publishes status updates to the splash screen
- fixed the client's repairdb to wait correctly for the ui in case the user wants to bail before repair
- the client's repair db now repairs the new local_tags_cache table if missing
- fixed an issue with showing the manage logins panel when the domain had some unusual cookie expiry numbers
- to reduce initial load gui-event crunch (which affects some systems' layout reliability), initial session load now happens after a 0.5s delay
- cleaned up some image load error handling
- when FFMPEG version information cannot be understood in help->about, a popup note appears and debug information is printed to the log
- the advanced content update panel now puts up a 'working…' dialog while it processed a job
- in the client
- all 4XX and 5XX network exceptions' error texts are now prepended with the actual status code
- extended the hover window debug report mode
- improved some misc temp and permission testing code
- improved some misc testing code
next week
It is E3 next week, so I am going to take my shitposting vacation week as usual. 356 will therefore be on the 19th. I would like to convert same/worse/better duplicate relationships to the new duplicates system, as I recently did for alternates and false positives. Otherwise I will do a bit of code cleaning.
Release Tomorrow!
I had an excellent and busy week catching up after E3. As hoped, I completed the second phase of the duplicates overhaul–all better/worse/same quality duplicate information is now stored more logically and efficiently. Duplicates processing is faster and duplicate groups are easier to manage. I also fixed a heap of bugs, including the clipboard watcher popup spam and then recent OS X shortcuts issue, and improved some maintenance code. The exit splash screen now has a 'stop doing maintenance' button!
The release should be as normal tomorrow, maybe a little late as there is a bit more to test.
Hydrus dev, I'm pretty sure I invented the proof to the big bang. If I get popular, I'll shout you out.
That's not why I made this thread, though. I'm so anxious I feel like vomiting. I never said this out loud anywhere, but besides porn I have nothing else in my life besides Hydrus. I use Hydrus extremely casually, anyway. But Hydrus helps sort porn, so… I was gonna shout it out.
I was gonna shout out my two favorite artists, too. I didn't have any plans to shout out anything else… So I imagine it'd be a huge influx of traffic.
I can't handle this at all. I never asked for this at all. No one will believe me until it happens anyway.
I have an appointment to show it to someone next week. It's fucked. I wish I could just be in protective custody and sleep for 20 hours and it'll be better then maybe.
Version 354
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v354/Hydrus.Network.354.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v354.tar.gz
I had a great week. The first duplicates storage update is done, and I got some neat misc fixes in as well.
false positives and alternates
The first version of the duplicates system did not store 'false positive' and 'alternates' relationships very efficiently. Furthermore, it was not until we really used it in real scenarios that we found the way we wanted to logically apply these states was also not being served well. This changes this week!
So, 'false positive' (up until recently called 'not dupes') and 'alternates' (which are 'these files are related but not duplicates', and sitting in a holding pattern for a future big job that will allow us to process them better) are now managed in a more intelligent storage system. On update, your existing relationships will be auto-converted.
This system uses significantly less space, particularly for large groups of alts like many game cg collections, and applies relationships transitively by its very structure. Alternates are now completely transitive, so if you have a A-alt-B set (meaning one group of duplicate files A has an alternate relationship to another group of duplicate files B) and then apply A-alt-C, the relationship B-alt-C will also apply without you having to do anything. False positive relationships are more tricky, but they are stored significantly more efficiently and also apply on an 'alternates' level, so if you have A-alt-B and add A-fp-M, B-fp-M will be automatically inferred. It may sound odd at first to think that something that is false positive to A could be nothing but false positive to B, but consider: if our M were same/better/worse to B, it would be in B and hence transitively alternate to A, which it cannot be as we already determined it was not related to A.
Mistakenly previously allowable states, such as a false positive relationship within an alternates group, will be corrected on the update. Alternates will take precedence, and any subsequently invalid false positives will be considered mistakes and discarded. If you know there are some problems here, there is unfortunately no easy way at the moment to cancel or undo an alternate or false positive relationship, but once the whole duplicates system is moved over I will be able to write a suite of logically correct and more reliable reset/break/dissolve commands for all the various states in which files can be related with each other. These 'reset/set none' tools never worked well in the old system.
The particularly good news about these changes is it cuts down on filtering time. Many related 'potential' duplicates can be auto-resolved from a single alternate or false positive decision, and if you have set many alts or false positives previously, you will see your pending potentials queue shrink considerably after update, and as you continue to process.
With this more logically consistent design, alternate and false positive counts and results are more sensible and consistent when searched for or opened from the advanced mode thumbnail right-click menu. All the internal operations are cleaner, and I feel much better about working on an alternates workflow in future so we can start setting 'WIP' and 'costume change'-style labels to our alternates and browsing them more conveniently in the media viewer.
The actual remaining duplicate relations, 'potential', 'same quality', and 'this is better', are still running on the old inefficient system. They will be the next to work on. I would like to have 'same quality' and 'this is better' done for 356, after E3.
the rest
The tag blacklists in downloaders' tag import options now apply to the page's tags after tag sibling processing. So, if you banned 'high heels', say, but the site suddenly starts delivering 'high-heel shoes', then as long as that 'high-heel shoes' gets mapped to 'high-heels' in one of your tag services, it should now be correctly filtered. The unprocessed tags are still checked as before, so this is really a double-round of checking to cover more valid ground. If you were finding you were chasing many different synonyms of the tags you do not like, please let me know how this now works for you.
The annoying issue where a handful of thumbnails would sometimes stop fading in during heavy scrolling seems to be fixed! Also, the thumbnail 'waterfall' system is now cleverer about how it schedules thumbnails that need regeneration. You may notice the new file maintenance manager kicking in more often with thumbnail work.
Another long-term annoying issue was the 'pending' menu update-flickering while a lot of tag activity was going on. It would become difficult to interact with. I put some time into this this week, cleaning up a bunch of related code, and I think I figured out a reliable way to make the menus on the main gui stop updating as long as one is open. It won't flicker and should let you start a tags commit even when other things are going on. The other menus (like the 'services' menu, which will update on various service update info) will act the same way. Overall menu-related system stability should be improved for certain Linux users as well.
The new 'fix siblings and parents' button on manage tags is now a menu button that lets you apply siblings and parents from all services or just from the service you are looking at. These commands overrule your 'apply sibs/parents across all services' settings. So, if you ever accidentally applied a local sibling to the PTR or vice versa, please try the specific-service option here.
full list
- duplicates important:
- duplicates 'false positive' and 'alternates' pairs are now stored in a new more efficient structure that is better suited for larger groups of files
- alternate relationships are now implicitly transitive–if A is alternate B and A is alternate C, B is now alternate C
- false positive relationships remain correctly non-transitive, but they are now implicitly shared amongst alternates–if A is alternate B and A is false positive with C, B is now false positive with C. and further, if C alt D, then A and B are implicitly fp D as well!
- your existing false positive and alternates relationships will be migrated on update. alternates will apply first, so in the case of conflicts due to previous non-excellent filtering workflow, formerly invalid false positives (i.e. false positives between now-transitive alternates) will be discarded. invalid potentials will also be cleared out
- attempting to set a 'false positives' or 'alternates' relationship to files that already have a conflicting relation (e.g. setting false positive to two files that already have alternates) now does nothing. in future, this will have graceful failure reporting
- the false positive and alternate transitivity clears out potential dupes at a faster rate than previously, speeding up duplicate filter workflow and reducing redundancy on the human end
- unfortunately, as potential and better/worse/same pairs have yet to be updated, the system may report that a file has the same alternate as same quality partner. this will be automatically corrected in the coming weeks
- when selecting 'view this file's duplicates' from thumbnail right-click, the focus file will now be the first file displayed in the next page
- .
- duplicates boring details:
- setting 'false positive' and 'alternates' status now accounts for the new data storage, and a variety of follow-on assumptions and transitive properties (such as implying other false positive relationships or clearing out potential dupes between two groups of merging alternates) are now dealt with more rigorously (and moreso when I move the true 'duplicate' file relationships over)
- fetching file duplicate status counts, file duplicate status hashes, and searching for system:num_dupes now accounts for the new data storage r.e. false positives and alternates
- new potential dupes are culled when they conflict with the new transitive alternate and false positive relationships
- removed the code that fudges explicit transitive 'false positive' and 'alternate' relationships based on existing same/better/worse pairs when setting new dupe pairs. this temporary gap will be filled back in in the coming weeks (clearing out way more potentials too)
- several specific advanced duplicate actions are now cleared out to make way for future streamlining of the filter workflow:
- removed the 'duplicate_media_set_false_positive' shortcut, which is an action only appropriate when viewing confirmed potentials through the duplicate filter (or after the ' show random pairs' button)
- removed the 'duplicate_media_remove_relationships' shortcut and menu action ('remove x pairs … from the dupes system'), which will return as multiple more precise and reliable 'dissolve' actions in the coming weeks
- removed the 'duplicate_media_reset_to_potential' shortcut and menu action ('send the x pairs … to be compared in the duplicates filter') as it was always buggy and lead to bloating of the filter queue. it is likely to return as part of the 'dissolve'-style reset commands as above
- fixed an issue where hitting 'duplicate_media_set_focused_better' shortcut with no focused thumb would throw an error
- started proper unit tests for the duplicates system and filled in the phash search, basic current better/worse, and false positive and alternate components
- various incidences of duplicate 'action options' and similar phrasing are now unified to 'metadata merge options'
- cleaned up 'unknown/potential' phrasing in duplicate pair code and some related duplicate filter code
- cleaned up wording and layout of the thumbnail duplicates menu
- .
- the rest:
- tag blacklists in downloaders' tag import options now apply to the parsed tags both before and after a tag sibling collapse. it uses the combined tag sibling rules, so feedback on how well this works irl would be appreciated
- I believe I fixed the annoying issue where a handful of thumbnails would sometimes inexplicitly not fade in after during thumbgrid scrolling (and typically on first thumb load–this problem was aggravated by scroll/thumb-render speed ratio)
- when to-be-regenerated thumbnails are taken off the thumbnail waterfall queue due to fast scrolling or page switching, they are now queued up in the new file maintenance system for idle-time work!
- the main gui menus will now no longer try to update while they are open! uploading pending tags while lots of new tags are coming in is now much more reliable. let me know if you discover a way to get stuck in this frozen state!
- cleaned up some main gui menu regeneration code, reducing the total number of stub objects created and deleted, particularly when the 'pending' menu refreshes its label frequently while uploading many pending tags. should be a bit more stable for some linux flavours
- the 'fix siblings and parents' button on manage tags is now a menu button with two options–for fixing according to the 'all services combined' siblings and parents or just for the current panel's service. this overrides the 'apply sibs/parents across all services' options. this will be revisited in future when more complicated sibling application rules are added
- the 'hide and anchor mouse' check under 'options->media' is no longer windows-only, if you want to test it, and the previous touchscreen-detecting override (which unhid and unanchored on vigorous movement) is now optional, defaulting to off
- greatly reduced typical and max repository pre-processing disk cache time and reworked stop calculations to ensure some work always gets done
- fixed an issue with 'show some random dupes' thumbnails not hiding on manual trashing, if that option is set. 'show some random dupes' thumbnail panels will now inherit their file service from the current duplicate search domain
- repository processing will now never run for more than an hour at once. this mitigates some edge-case disastrous ui-hanging outcomes and generally gives a chance for hydrus-level jobs like subscriptions and even other programs like defraggers to run even when there is a gigantic backlog of processing to do
- added yet another CORS header to improve Client API CORS compatibility, and fixed an overauthentication problem
- setting a blank string on the new local booru external port override option will now forego the host:port colon in the resultant external url. a tooltip on the control repeats this
- reworded and coloured the pause/play sync button in review services repository panel to be more clear about current paused status
- fixed a problem when closing the gui when the popup message manager is already closed by clever OS-specific means
- misc code cleanup
- updated sqlite on windows to 3.28.0
- updated upnpc exe on windows to 2.1
next week
The duplicates work this week took more time than I expected. I still have many small jobs I want to catch up on, so I am shunting my rotating schedule down a week and doing a repeat. I will add some new shortcuts, some new tab commands, and hopefully a clipboard URL watcher and a new way of adding OR search predicates.
Release Tomorrow!
I had a good week doing small work. The error-popup issue with collected thumbnails is fixed, and there are new shortcuts to better allow keyboard-only navigation of the manage tags dialog, an experimental jpeg quality estimator for the duplicate filter, a fun clipboard watcher for easy URL import, and some new options for file viewing statistics.
The release should be as normal tomorrow.
Version 353
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v353/Hydrus.Network.353.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v353.tar.gz
I had a great week. I finished the basics of the new file maintenance system I wanted, cleaned up the duplicate filter a little more, and fixed a bunch of bugs.
file maintenance system
There are a number of large file re-checking jobs the client wants to do, both now and in the future. Going back to figure out more accurate video durations and image rotations, discovering webms that were formerly incorrectly detected as mkvs, eventually integrating videos into the duplicate checking system, all of these will require a combined whack of maintenance CPU that I don't want to hit all at once. I have previously sketched out some disparate systems for these jobs, but none were really doing the trick, so this week I unified it all into one nice system that can handle all sorts of jobs. This new system is simple for now but will get more work in future.
You do not have to do anything, but if you pay attention to your maintenance work, you may notice some new file metadata and thumbnail jobs running in the background or on shutdown. This new system fits into normal maintenance just like database analyzing or repository processing. You can govern whether it is permitted to run in regular idle time and/or shutdown time under options->maintenance and processing and also change its in-built 'throttle', which limits the number of files it will work on (rather than running full bore on what in future may be quite large jobs). The default throttle is 200 files every day, which for most jobs on most machines will be about 30 seconds to three minutes work. Do not expect it to do much work yet.
Existing file regeneration routines now work through this system, and it does its job much better than before. If you hit right-click->regenerate->x on some thumbnails, the job now runs in a regular popup button (rather than the locking 'modal' one from before), letting you keep browsing while it works. And if you select more than 50 thumbnails (think, say, right-clicking on 2,000 video files and saying to regenerate their thumbnails if they are the wrong size), you will now get the option to schedule that big job for later, at which point those 2,000 jobs will end up in the normal idle maintenance queue, to work at 200 files a day or whatever you wish.
This system is fairly opaque at the moment. You can trigger it with the thumbnail right-click, and certain db operations may schedule new jobs for it, but there is no UI to review it yet. In the coming weeks, I expect to write a new 'review' window off the database menu that will let you review total pending jobs, start work manually, and add and remove pending jobs en masse through the regular search interface. I'll slowly integrate more of the client into it as well, letting it add more jobs into the queue by itself.
Let me know how this all works for you!
duplicate filter
The duplicate filter interface got some more work this week, particularly in cleaning up some of my original version's over-engineering. The actions you can choose on the right panel are now split more clearly into 'yes, these files are duplicates, and here is how' decisions vs the 'alternates' and 'not dupes' decisions. Also, 'this file is better' is now split into two buttons for 'delete the worse file' and 'keep both'. This 'delete or not' is split at the shortcut level into two actions as well, if you wish to map both. Existing shortcuts (left-click by default in the filter) will update to the 'and delete the worse file' version.
The complicated 'duplicate action options' object (which governs how to merge metadata across duplicates) therefore no longer handles file deletion. It is also now only attached to the 'better/worse' and 'files are the same' actions–we never found a good reason to merge metadata across all alternates or 'not duplicates', so I have removed it completely. If you want a complicated file delete action, hitting the 'custom action' button now asks you if you wish to delete the file you are looking at, the other one, or both.
Also, to reduce confusion with alternates–which are also technically not duplicates–'not duplicates' is now renamed across the program to the more precise 'not related/false positive'. The 'false positive' action is a record in the db saying 'despite the similar files search thinking these files were related, it was incorrect, so do not bring it up again'.
My hope is that filtering is a bit faster here. If two duplicate files are of very different quality, it is still easy to delete the bad one, but if they are more close and you want to keep both, it is now just one click.
As for the big db-level rewrite, I prepped the duplicate db code for it this week. I am standing at the cliff-edge and feel great about jumping off, so next week I hope to get started on the new code properly and migrate one or both of the current 'alternates' and 'false positive' data to the new system.
the rest
I fixed an issue with the recent 'collect by' session saving where the accompanying sort was not being renewed on a session load. Also, several problems with collected media and sort by 'approx bitrate' are fixed.
There's a new checkbox under options->sort/collect that makes it so the default sort is updated every time you click a new sort in regular browsing. It sounds a pain but is actually pretty neat!
The 'all local files' domain is now hidden from view in new page selection and the tag autocomplete dropdown if you are not in advanced mode. This domain, which is fairly technical and covers both trash and 'my files' and the sometimes-hidden repository update files, is often confusing to new users and is rarely useful even for people who know what it does.
If you use the client's local booru and need to override its host when you copy an external link, this option has moved from options->connection to the local booru's manage services panel. You can also override scheme and port as well! The old host override option is gone completely, and the only other place it was used, the manage upnp dialog, now fetches this info more efficiently and fails more gracefully.
full list
- duplicate filter:
- duplicate action options no longer handle file deletion
- renamed 'not duplicates' across the program to 'not related' or 'false positive'
- 'alternates' and 'not related/false positive' duplicate actions no longer have duplicate action options. no merge content update now occurs on these actions
- the duplicate filter hover panel now splits 'this is better' decisions into two buttons–whether to delete or keep the worse file
- when selecting 'custom action' in the duplicate filter hover panel, it now asks if you would like to delete the current file, the other file, or both
- the 'duplicate_filter_this_is_better' shortcut action will be auto-updated to 'duplicate_filter_this_is_better_and_delete_other'. an alternate 'duplicate_filter_this_is_better_but_keep_both' is now also available
- the 'duplicate_filter_not_dupes' shortcut action will be auto-updated to 'duplicate_filter_false_positive'
- separated the buttons on the duplicate filter hover panel to more carefully split 'yes, files are duplicates' vs other decisions
- in prep for the duplicate db overhaul, refactored all PHash search code and Duplicate management code apart
- misc other prep work for duplicate db overhaul
- .
- file maintenance:
- wrote a new unified manager to handle various long-term file maintenance tasks like regenerating file metadata and thumbnails
- options to govern how this manager can run are now in options->maintenance and processing. you can enable it for idle and shutdown maintenance time and give it a throttle to limit how fast it will work on files, defaulting to 200 per day
- unified the previous db-level attempts at file maintenance to the new system, which supports async job queueing, and moving regen code up to the new manager, out of the db lock
- unified a variety of file and thumbnail regen code to work through the new simpler and saner path
- the right-click->regen thumbnail commands now run through the new manager and no longer need a modal popup. you can keep browsing while they work. they will also not hang the ui as the old system could on big jobs
- when right-click->regenning on more than 50 thumbnails, you now get a dialog asking if you want to do the job now or put it off later
- file maintenance tasks can now run in shutdown time! you will get previews of the jobs with file counts and status progress reports on the shutdown splash
- cleaned up some file extension renaming and dupe-removing code
- in future, I will move the current file integrity check to this new system and have some ui to prompt and set up other big jobs, like fixing various historical misparsing issues
- thumbnail resizing during thumbnail fade that resizes down is now more efficient
- moved the ClientFilesManager to ClientFiles.py
- .
- the rest:
- the 'manage upnp' dialog now moves the duplicated external ip display from the column up to the status text at the top. it fetches the ip after the initial mappings fetch is done. this ip is no longer affected by the external host override option
- cleaned up options->connection page and removed the now defunct external host override option
- the manage services page for the local booru now has optional override for scheme, host, and port for the 'copy external url' function
- fixed an issue with the recent 'collect by' session saving where a restored session that needed a collect was not sorted
- fixed an issue with collections being sorted by approx bitrate
- added a new checkbox to options->sort/collect to set it so the default sort updates every time you choose a new sort anywhere
- fixed an issue with 'remove trashed files from view', which was incorrectly removing on 'all local files' pages
- the 'all local files' file domain, which is frequently confusing to new users, is now no longer an option for new file pages or the autocomplete file domain if the user is not in advanced mode
- the client now searches for versions of urls both with and without a final '/' character when looking up file url import status at the db level and in import lists. system:known_url is unfortunately still an inefficient mess
- improved how the server code deals with some connectionLost errors
- cleaned up and unified some older dialog button code
- fixed a problem in manage tag siblings when petitioning existing pairs and then cancelling when asked for a reason
- fixed a miscount issue when uploading pending tags while many new tags are coming in. progress would sometimes be -754/1,234, ha ha
- db maintenance, repository sync, and file maintenance processing will all now wake on a force idle mode call
- deleted some old code
- misc fixes and cleanup
- some misc gui layout fixes
next week
I have quite a few smaller jobs waiting for me, so other than the new duplicate db tables, that's top priority. Some UI bugs to deal with, maybe some Client API work, an experimental jpeg quality estimator, possibly support for some new filetypes, and hopefully a fun new way to quickly add very complicated OR search predicates thanks to a clever user's work.
Just a note, E3 is coming up soon and I will take my shitposting vacation week for it as usual. I think it'll be 356 that's delayed a week.
Release Tomorrow!
I had a great week. The new 'false positive' and 'alternates' duplicates db storage design and migration worked out well, although it did take most of my time. Other than that, I did a variety of little new options and bug fixes. The annoying bug where a handful of thumbnails sometimes stop fading in seems to be finally fixed!
The release should be as normal tomorrow.
Version 352
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v352/Hydrus.Network.352.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v352.tar.gz
I had a good week. .ico files are now supported, 'collect by' status is remembered in gui sessions, and I fixed a bunch of bugs.
duplicate overhaul plans
I started the duplicate overhaul work this week with some planning and experimentation with existing data. My original thought here had been to exactly replicate existing functionality just with a more efficient database schema, but having gone through the various edge-case insertion and merge operations, I believe the current system is overcomplicated for what we are actually using it for.
Most of all, the current system tries to form a chain of 'better/worse' comparisons so all dupes within a 'same file' group are ranked with each other. A decent number of human decisions are needed to determine this ranking, but the data is currently not displayable, and we haven't really noticed that absence. For most practical circumstances, what we really want to determine is what files should actually be considered dupes, and which of those groups is then the best. Most users delete the 'worst' of a pair in any case. Supporting a system that simply tracks a group of duplicates with a single King is more intuitive and reliable, and it is quicker to work with.
This 'King of a group' idea also maps nicely to how we use 'tag siblings'–having a complicated tree or chain of worst tag to best is not as useful as simply replacing all the lesser members of a group with the King as needed. When I get to overhauling tag siblings, I expect to make a similar change.
But in the meantime, for duplicates, I now have a plan. I expect to spend a few more weeks filling out the full details in code, and then I will switch us all over. The existing workflows should remain the same, just with fewer and easier comparisons. I will not do much specific work on file alternates, but they will be feasible to start on once the db overhaul is done. I will also continue to put time into the duplicate filter ui itself. Overall, I feel good about it all. I'd like the whole thing to be done within 8-12 weeks.
otherwise all misc this week
.ico files should now be supported! .cur files (which are basically the same) should work as well.
'collect by' settings are now, finally, saved in page sessions! If your default collect by settings include any ratings services, they will be forgotten on update, so you will have to reset them in options->sort/collect this one time.
I fixed the stupid issue where media viewer hover windows were popping up over manage tags and some other dialogs. This was due to a flaw in the changes from the new always-on-top duplicate hover panel–I apologise for the inconvenience. Some related OS X specific weirdness should be cleaned up as well.
The 'unclose_page' shortcut (default Ctrl+u) now uncloses pages in the correct order!
When a media fits the media viewer exactly (so 100% zoom fits the width or height exactly), the 'zoom switch' action (default 'z') now correctly restores back to 100%!
'open externally' should work better for some custom program paths. The flash projector (for .swf files) was opening without an ui, for instance. If you have had other programs seem to open in the background from open externally calls, please give them another go and let me know if they now work for you.
full list
- the client now supports importing .ico files! (.cur should be supported too)
- finally, 'collect by' is saved for sessions! if your default collect by previously included ratings services, it will forget them this one time–please reset it under the options->sort/collect
- fixed the issue where the media viewer's hover windows were hovering over child dialogs (manage tags, ratings, or known urls)
- improved some os x hover window focus handling for the new always-on-top duplicate action window
- the entries on the 'sort by' list on gui pages are now subcategorised better. it should be a bit easier to find what you are looking for
- the 'sort by file: approximate bitrate' sort option now sorts still images as well by filesize / num_pixels
- to reduce confusion, sort by mime and system:mime are now renamed to 'filetype'
- fixed an issue where the 'unclose_page' shortcut was restoring pages in reverse order (unclosing least-recently-closed-first rather than most-recently-closed-first)
- improved rigour of video framerate estimation
- stopped the video metadata parser from opting to manually frame count videos with size >128MB or num_frames estimate >2,400
- fixed the forced manual frame count to deal with frame counts >9999
- the 'ffmpeg not found' error on file import will now put up a popup message once per boot informing you of this problem more broadly and steps to address it
- fixed some underreporting issues with subprocess_report_mode
- fixed an issue with some yes/no dialogs returning 'no' on escape/window_close_button rather than 'cancel', which affected cancelability some db maintenance questions
- fixed an issue where media that fitted the media viewer canvas width or height exactly at 100% zoom would not respond to zoom switch events to restore non-100% zoom to 100%
- when a local server's CORS mode is turned on, Access-Control-Allow-Origin is now correctly added to GET/POST requests with an Origin request header
- improved reliability of some timestamp rendering code, which should help some users who had trouble opening cookies management page after malformed cookie import
- I believe I fixed an issue with 'open externally' on certain custom paths where the external program could spawn without an ui (flash projector did this). please let me know if your 'open externally' calls start making terminal windows everywhere
- fixed a runtime stability issue with the new duplicates page and slow-updating counts that come in after the page has been deleted
next week
Next week is an 'ongoing' week, where I work on a medium-sized improvement to an existing system. I think I would like to put some time into my 'background file maintenance' plans, unifying the current prototype systems into one that runs nicely in idle time and adding some ui controls for it. There are several pending file-reparsing jobs I would like to queue up (e.g. checking mkvs vs webms with modern file parsing code, and apng discovery, scheduling background thumbnail regen, and fixing bad old frame counts and durations), and if I want to integrate videos into the duplicates system, I'll need a better framework here to schedule that retroactive CPU work. Otherwise I have a couple little cleanup jobs to be getting on with, and I'll start some new duplicates db code.
Bibanon and WebUI thread
It looks like the Bibanon wants to use Hydrus as a server, and they really want to slap a webUI on top of it… but Hydrus Dev don't have it on priority list.
They have though about using Szurubooru for the WebUI but rr- don't want to update it. See: https://github.com/rr-/szurubooru and rr-@sakuya.pl
Maybe it is time to draft up a sample API set and WebUI for them.
From antonizoon 2 Sep 2018
Hey there, it's been a very long while, but it's good to see that you've
kept working on Hydrus ever since.
One of the issues with hydrus for me in the end, which is why we never
ended up using it, is because of the lack of a webui. As such I ended up
utilizing Szurubooru, as can be seen here: https://eikonos.bibanon.org
However, what I can say is that the dev team at the Library of Congress
was very interested in adding tag support to their image tag system
which is below. It's currently utilized with the aim of crowdsourcing
transcriptions of images, but it lacks tagging or search like we would
have in a booru. Luckily, we already have the hydrus protocol and
szurubooru right? They aim to launch in October.
I find this to be a perfect chance to integrate Hydrus with a webui as
well as obtain some useful features from their existing system as well.
What do you think?
https://github.com/LibraryOfCongress/concordia
Possible CSS frameworks
https://tutorialzine.com/2018/05/10-lightweight-css-frameworks-you-should-know-about
https://dzone.com/articles/top-10-lightweight-css-frameworks-for-building-fas
http://www.creativeweblogix.com/blog/12-lightweight-css-frameworks
https://speckyboy.com/responsive-lightweight-css-frameworks/
Also whether to use Angular/React/Vue (JS web) vs Django/CherryPy/TurboGears/Flask/ Pyramid (Python Web) comes into question
Contacts
Twitter - https://twitter.com/BAnonoma
IRC - irc.rizon.net #bibanon
Discord - https://discord.gg/0kPf8xuFPFa3iihT
Release Tomorrow!
I had a good week. I added .ico file import support, got 'collect by' settings saving through gui sessions, and fixed a whole bunch of bugs (including the annoying media-window-hover-frame-over-manage-tags issue).
The release should be as normal tomorrow.
Version 351
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v351/Hydrus.Network.351.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v351.tar.gz
I had an ok week. I sped up several systems and added a new processing panel to the duplicate filter.
The 'next big job' poll is finished! I will next be focusing on overhauling the duplicate filter's db structure (including sketching out support for file 'alternates') and further improving the ui-side workflow.
duplicate filter
Seeing that the duplicate filter work was popular in the poll, I was happy to put a bit more time into it this week.
Most importantly, the duplicate filter now has a new always-on-top panel to make reviewing differences and making decisions easier. Essentially, I have pulled the 'this has higher resolution' statements and the action buttons out of the top hover window and put them in their own box to the middle-right. It stays on top, so you can always see it, and I have expanded the different statements to explicitly state each file's relevant values, such as '550KB >> 141KB', and to colour their text green/blue/red based on that difference. You can now make at-a-glance decisions for easy pairs. Let me know how it works for you!
I also have tweaked the 'show some random dupes' and 'duplicate filter' database routines to sample results more cautiously, meaning that duplicate filters with very large search domains (like system:inbox or system:num_tags>0) will work significantly faster. The initial search step still has to run every time, but the second 'sampling' stage takes barely any time at all. If your dupe work still takes a long time to count up or load pairs to filter, I'll recommend again to just add a 'creator:' tag.
The next 'big job' work here is to overhaul the duplicate database tables to work in more intelligent 'groups' rather than my initial simple 'pairs' system. This will compact the duplicates data, speed up many operations, massively simplify transitive duplicate logic, and lead to alternate file structure support. I'll also probably copy the basic structure to tag siblings and parents, which are a similar data structure also currently stored in pairs. This job will take some work, but I know the general thrust of what I want to do.
faster code and some misc work
I cleaned and improved a bunch of old code this week. First off, image loading now uniformly uses a faster library, so image imports and some thumbnail creation should be a little bit faster and deal with some rare image rotations more reliably. Secondly, the way the tag siblings and parents managers construct themselves on client boot is significantly faster. And a new 'local' tag cache–which will take a minute to construct when you update the client–will speed up many tag-related operations, particularly file results building, especially right after the client boots. There are many changes here, so please report any bugs you see.
review services now uses nested notebooks instead of my old 'listbook' control. I don't really like how it looks now, but the code behind the scenes is a lot saner. The last instance of this bad listbook is the options dialog, which is even worse behind the scenes but a bit too big to fit into a single notebook–I am still thinking about what to do with it.
It is just a little thing, but subscription popups now list file download progress x/y in terms of the current job rather than total subscription history! So, if your sub has 400 files already downloaded and finds 10 more in a sync, the popup will now say (and display a progress gauge for) a much more helpful 3/10 rather than 403/10.
full list
- wrote a new (always on top!) hover window for the duplicate filter that sits on the middle-right. the duplicate cog button and action buttons are moved to this new window, as are the file comparison statements
- the duplicate file comparison statements now state the relevant actual metadata along with better '>>'-style operators to highlight differences and green/red/blue colouring based on given score. it is now much easier to see and action clearly better files at-a-glance
- improved some hover window focus display calculations to play with the new always-on-top tech
- both the 'show some random dupes' button and finding dupe pairs for the filter should be a bit faster for very large search domains. the basic file search and indexing still has to run, but the second sampling step in both cases will bail out earlier once it has a decent result
- core image handling functions now uniformly use OpenCV (faster, more accurate) by default, falling to PIL/Pillow on errors. image importing in the client and server should be a bit faster, and some unusual image rotations should now be read correctly
- the server now supports OpenCV for image operations, it _should_ also still work with only PIL/Pillow, if you are running from source and cannot get it
- unified all thumbnail generation code and insulated it from suprises due to unexpectedly-sized source files, fixing a potential client-level thumbnail generation looping bug
- gave all image processing a refactor and general cleanup pass, deleted a bunch of old code
- wrote a new 'local tag cache' for the db that will speed up tag definition lookups for all local files. this should speed up a variety of tag and file result fetching, particularly right after client boot. it will take a minute or two on update to generate
- sped up how fast the tag parent structure builds itself
- the review services panel now uses nested notebooks, rather than the old badly coded listbook control. I don't really like how it looks, but the code is now saner
- similar-files metadata generation now discards blank frames more reliably
- subscription popups now report x/y progress in terms of the current job, discarding historical work previously done. 1001/1003 is gone, 1/3 is in
- made the disk cache more conservative on non-pre-processing calls
- cleaned up some file import code, moving responsibility from the file locations manager to the file import object
- updated the ipfs service listctrl to use the new listctrl object. also cleaned up its action code to be more async and stable
- I believe I fixed a rare vector for the 'tryendmodal' dialog bug
- fixed a bug in presenting the available importable downloader objects in the easy drag-and-drop downloader import when the multiple downloaders dropped included objects of the same type and name–duplicate-named objects in this case will now be discarded
- unified url_match/url_class code differences to url class everywhere
- updated some common db list selection code to use new python string formatting
- plenty of misc code cleanup
next week
I'll now tackle this dupe db work. I'll plan and experiment first. Otherwise, next week is a small jobs week. I'd like to add .ico support and add pages' 'collect by' dropdown value to sessions so it gets saved.
Next Big Job Poll
Here is the poll for the next big job:
https://www.poll-maker.com/poll2331269x9ae447d5-67
You can vote for multiple items. I expect to start work on the top-voted item at roughly the time of the v351 release post, in two weeks.
Please feel free to discuss and ask about items on the poll in this thread.
Version 350
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v350/Hydrus.Network.350.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v350.tar.gz
I had an ok week. Some IRL things cut into my hydrus time, but I got some good work done. Some bugs in the new duplicate search system are fixed, and I improved advanced file delete and export handling.
The poll for the next 'big job' is up here: >>12358
duplicate filter
The search addition to the duplicate filter went fairly well, but there were a couple of significant bugs. The 'ghost pair' issue–where a queue would sometimes have a final pair that would never display and lead to high CPU until the filter was closed–is fixed, and safeguards added to catch similar issues in future. The issue with undercounting on large search domains (typically where the dupe page's file query was non-system:everything and covered >10,000 files) is also fixed, but giving the filter 500,000 custom files to work with can be really quite slow. I will keep working here to see if I can speed up big searches like this without compromising accuracy, but if you find your dupe searches are working too slow, try adding a creator: tag to bring the search size down–it works well.
The duplicate filtering workflow itself is still a v1.0 pain. I expect to put some work in here in the coming weeks (and doubly so if dupe work wins the big poll), likely highlighting the differences between the two images with an always-on-top panel and better at-a-glance decision-making for easy comparisons.
advanced delete
This week brings an optional advanced file delete dialog. You can turn it on under options->files and trash and also set custom file delete 'reasons' to assign. Once it is switched on, any file delete under the thumbnail grid, media viewer, or duplicate filter manual file delete will instead give a richer dialog that lets you physically delete files immediately (i.e. skipping the trash) and deleting files without leaving a delete 'record' (which is useful if you want to easily reimport those files later on).
You can also choose from your set reasons or type a completely custom one. These saved reasons reappear when that deleted file fails to import in future (due to being recognised as previously deleted), so if you are interested in tracking why you have previously deleted files, please check this out.
the rest
Export folders now have more timing options, basically the same as import folders. You can pause them individually, tell them not to run periodically, and force them to run manually under a new submenu under file->import and export folders->run export folder now.
The Artstation downloader seems to be not working due to a Cloudflare issue. I am leaving the downloader in for now, in case it starts working again, but if you have a subscription for it, I recommend you pause it for now. The default downloader for new users (and you, if you have it still set as the default) is now safebooru.
If you use the Dolphin file manager, check out this add-on that uses the Client API: https://gitgud.io/prkc/dolphin-hydrus-actions
Advanced users only: Missing file folder recovery is better–if a file folder (one of the subfolders, like 'f39' or 'ta2') is missing on boot but found in another 'known' location, the client will detect the moved folder and propose an auto-update of its records rather than forcing you to go through the complicated manual recovery dialog. If for some reason it is complicated or slow for you to move your folders from A to B through the migrate database dialog, you can now shut down your client, move the folders manually in explorer or wherever, and then reboot the client and be up and working in one click! Just make sure that hydrus knows about B in its locations under migrate database beforehand, and it will do the rest.
full list
- the duplicate filter no longer applies the implicit system limit (which defaults to 10,000 files) on its search domains, which solves the undercounting issue on large search domains. duplicate operations will be appropriately slower, so narrow your duplicate file queries (adding a creator: tag works great) if they take too long
- fixed the duplicate pairs filter's 'ghost pair' issue. it was failing, when 'both files' was unchecked, to remove pairs that included one file that was non-local. this accidental inclusion resulted in incorrect non-zero count and filter/random pairs that could not display correctly
- insulated against potential future iterations of this problem (likely that one of the pair was deleted by another process while a filter is ongoing), with a notification and graceful exiting of the duplicate filter while saving progress
- the 'show random duplicates' button now puts the 'base' of the group (to which all the others are potentially matched) as the first thumbnail
- added a new 'advanced file deletion' section to 'files and trash' options page to turn on a new advanced dialog and set custom file deletion reasons
- if this new dialog is turned on, a delete event from thumbnail grid, regular media viewer, or the duplicate filter's manual delete will launch it. it permits you to delete physically (skipping trash) in one step or delete physically without leaving a deletion record (for easier later re-import) and choose one of the deletion reasons in the new options panel or set a one-time custom reason
- export folders now have more run-controls: 'run regularly', 'paused', and 'run now'
- the file menu now has a 'run export folder now' submenu just like for import folders– it is simple now to set up an export folder that only runs when you tell it to
- updated the on-boot missing file folder recovery process to automatically 'heal' file location mappings when a missing folder is actually in a location that is known (essentially, you can now manually move a bunch of folders from one known location to another while the client is off and it will recover automatically now). error dialogs will appear in this case summarising the problem and proposed fixes with a chance to bail out if you do not want it to happen
- added a new frame type to 'gui' options page called 'regular_center_dialog' for yes/no style dialogs that are better in the center of the parent window
- the custom web browser launch path and file type 'open externally' paths are moved from 'files and trash' to a new 'external programs' options page
- as the superior '–temp_path' program launch parameter now exists for both client and server, I have removed the limited 'BUGFIX: temp folder override' option from the client's 'files and trash' page and use in the actual code. if this option was important to you, please migrate to the –temp_path launch parameter, which covers temp usage more comprehensively
- as the artstation downloader is now non-functional, apparently by a cloudflare issue, the default gug for new users (and anyone with artstation set atm) is now safebooru
- added dolphin file manager add-on link to the client api help
- some misc file metadata fetching cleanup
next week
Next week is a cleanup week. Beyond boring ongoing code cleanup and rewrites, I would like to move some 'default' image parsing to preference OpenCV over PIL uniformly, including for the server (which currently does not need OpenCV). Having image work preference one library over the other at different times causes problems when they disagree about some image metadata (e.g. whether a file is rotated). OpenCV is fairly easy to get for all platforms now, and it generally runs faster and better, so I feel good about adding it to the server and making it the primary choice in all situations.
We breezed past 500 million mappings on the PTR in the past couple of weeks! This is great, and I am really thankful for everyone's contributions, but this growth is putting pressure on several different areas. You may have noticed we finally hit my 256GB/month bandwidth limit on the shared PTR account at the end of April. Regular PTR-syncing client databases are also getting huge, about 18GB. I have multiple plans for how to deal with the various issues here and expect to chip away at them throughout 2019. I may reserve a month or two of future 'big job' time to handle this work.
local variable 'photoshop' referenced before assignment
Hey,
awesome tool, really enjoying it.
I did encounter an error that came up three times when fetching files from Sankaku (out of probably around 10k+ files imported so far, I've only just started using hydrus last week):
local variable 'photoshop' referenced before assignment… (Copy note to see full error)
Traceback (most recent call last):
File "include\ClientImportFileSeeds.py", line 1178, in WorkOnURL
self.DownloadAndImportRawFile( file_url, file_import_options, network_job_factory, network_job_presentation_context_factory, status_hook, override_bandwidth = True )
File "include\ClientImportFileSeeds.py", line 571, in DownloadAndImportRawFile
self.Import( temp_path, file_import_options )
File "include\ClientImportFileSeeds.py", line 790, in Import
( status, hash, note ) = HG.client_controller.client_files_manager.ImportFile( file_import_job )
File "include\ClientCaches.py", line 1144, in ImportFile
file_import_job.GenerateInfo()
File "include\ClientImportFileSeeds.py", line 283, in GenerateInfo
self._file_info = HydrusFileHandling.GetFileInfo( self._temp_path, mime )
File "include\HydrusFileHandling.py", line 233, in GetFileInfo
( ( width, height ), duration, num_frames ) = HydrusImageHandling.GetImageProperties( path, mime )
File "include\HydrusImageHandling.py", line 241, in GetImageProperties
( ( width, height ), num_frames ) = GetResolutionAndNumFrames( path, mime )
File "include\HydrusImageHandling.py", line 274, in GetResolutionAndNumFrames
pil_image = GeneratePILImage( path )
File "include\HydrusImageHandling.py", line 92, in GeneratePILImage
pil_image = PILImage.open( fp )
File "site-packages\PIL\Image.py", line 2690, in open
File "site-packages\PIL\Image.py", line 2676, in _open_core
File "site-packages\PIL\JpegImagePlugin.py", line 783, in jpeg_factory
File "site-packages\PIL\ImageFile.py", line 103, in init
File "site-packages\PIL\JpegImagePlugin.py", line 373, in _open
File "site-packages\PIL\JpegImagePlugin.py", line 139, in APP
UnboundLocalError: local variable 'photoshop' referenced before assignment
The URLs in question are these ones:
https://chan.sankakucomplex.com/post/show/7921258
https://chan.sankakucomplex.com/post/show/7921259
https://chan.sankakucomplex.com/post/show/7921260
I didn't specifically add the URLs, I did a tag search for "1girl lying_on_bed".
I've retried a few times and got the same error. Downloading the actual images and trying to import them manually results in the same error.
I'm using hydrus version 349 on Windows 10 Pro (version 1809).
It's not an issue for me since I don't care about these three files anyway, but I thought you might want to know.
Version 349
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v349/Hydrus.Network.349.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v349.tar.gz
I had a great week. The duplicate filter work went really well, and the manage tags dialog has a neat new button for fixing siblings and parents.
The poll for the next 'big job' is live here >>12358 ! The direct link is https://www.poll-maker.com/poll2331269x9ae447d5-67
duplicate filter
The duplicate filter page has its old 'file domain' button swapped for a full file search context. This allows you to see 'potential pair' counts, show some random pairs, and launch the full duplicate filter on just a subset of your dupes! For instance, you might want only to filter from a certain creator, or only on very small jpgs. You can also check a box to set whether both of the pair must match your search or only one.
The duplicate page has a significant relayout as well. It now has two tabs–one for the maintenance/search side, and one for the actual filtering–and shows its data in a clearer and less laggy way. The new search data is saved with your page, and on a per-page basis, so if you like you can set up several duplicate processing pages for your different queues.
This system works faster and better than I had expected. Please check it out and let me know what you think. It is usually fairly fast unless you give it a very slow search to work with.
One bug I just noticed with my IRL client is if you give it a specific search with more than 10,000 files (like for me system:archive), the duplicate count will be too low and will change slightly on refreshes. This is because of the 'implicit' system:limit on all queries, which caps search results to 10k files (or whatever you have it set to in the options), and causes the duplicate filter to instead be sampling the potential search domain on each query. This is not true in this case for system:everything, which here uses a search optimisation to get the full count. So, I suggest you throw in a <10k tag for this week, and I will see if I can tackle this problem for 350.
I will update the help for this once I know the ui is settled.
manage tags
The manage tags dialog has a new 'siblings and parents' button that auto-hard-replaces siblings and adds missing parents! It even works on multiple file selections, so if you have a bunch you want to fix, you can just ctrl+a and hit the button and it will sort it out for each file. It gives you a little summary yes/no dialog before it fires just to make sure everything looks sensible.
Beyond some additional sibling/parent code cleanup on that dialog, there is also a new 'cog' option to stop triggering 'remove' actions if you 'enter' a tag that already exists. If you turn this on, only a double-click or delete-key press on the list will make for a 'remove' result.
the rest
I spent some time cleaning up file import and thumbnail generation, particularly for videos. Videos have improved metadata parsing, and thumbnail generation should be much faster. A variety of unusual webms should now be importable and have correct frame counts. Thumbnail right-click->regen commands also work faster and update thumbnail metadata more cleanly and reliably.
If you would like pages to always focus their 'input' boxes when you switch to them, there is now an option for this under the 'gui pages' options panel.
full list
- duplicate filter:
- the duplicate filter page now has a full-on real-deal file search object to narrow down the duplicate filter, potential duplicate count, and 'show some random dupes' search domains! it also has a 'both files match' checkbox that determines if one of both files of the potential pairs should match the search!
- the duplicate filter page has multiple layout changes as a result:
- the main management area is now split into two pages–'preparation', for doing maintenance and discovery work, and 'filtering', for actioning the potential dupe pairs found
- the 'filtering' page will select by default, but if 'preparation' needs work, its name will become 'preparation (needs work)'
- the 'filtering' page now has file search ui and the 'both files' checkbox instead of the file domain button. this search data is saved on a per-page basis
- the two pages' status texts are now updated on separate calls which have been rewritten to be asynchronous (with 'updating…' text while they work). both now have explicit refresh buttons to force them to update
- the additional non-unknown pair counts listed on the filter area, which were irrelevant to filtering and sometimes confusing, are now gone. it only lists the 'unknown' pair number
- the duplicate filter page's help button no longer has the awful 'simple help' entry. the full html help will get a pass in the coming weeks to reflect the new search changes
- the duplicate file db code received significant refactoring and improvement to support searching the potential dupe space while cross-referencing the new file search context (and still falling back to the fast code when the search is just blank/system:everything)
- misc duplicate file db code cleanup and refactoring
- while in advanced mode, you can no longer select 'all known files' file domain for an export folder (and now the duplicate filter page) search context
- making a file delete action in the duplicate filter (by hitting delete key or the button on the top hover window, which both trigger a dialog asking to delete one or both) now auto-skips the current pair
- .
- manage tags:
- the manage tags has a new 'siblings and parents' button that will auto-replace incorrect siblings and auto-add missing parents! it works on multi-file selections as well! it gives you a summary yes/no dialog before it fires
- the manage tags dialog has a little logic cleanup r.e. siblings and parents and their cog auto-apply options. the auto-application now only applies on add/pend actions
- the manage tags dialog has a new cog button option to not trigger 'remove' actions from an autocomplete dropdown or suggested tag input action when the tag already exists
- .
- the rest:
- gave video metadata parsing another pass–it now detects 'hidden' incorrect framerates due to advanced 'skip frame' codec settings and is more accurate at determining frame count and duration, including some fixed offset calculations that was sometimes adding or discounting a few frames
- manual video frame count, when needed, is now faster and produces better error text
- fixed a critical bug in thumbnail regen that was sometimes potentially looping regen on files with unusual rotation exif information
- significant improvements to how the client file manager handles thumbnail identifier information, saving a great deal of time for file import and thumbnail regeneration code of videos
- fixed an issue where regenerated file metadata was not propagating up to the ui level in real time
- cleaned up some thumbnail cache initialisation code
- the 'generate video thumbs this % in' option is moved from the 'media' to 'thumbnails' options page
- to simplify code, and in prep for the idle-maintenance-rewrite of this system, the database->regen->thumbnails call is now removed
- all three fields of text on serialised pngs now wrap, and they pad a little better as well
- added a new option to the 'gui pages' options page to force input text box focus on page changes
- fixed a small type issue with the server's session cookie code and some new library versions
next week
I've felt increasingly ill this week, so I am going to take a day or two completely off. Otherwise, next week will be catch-up for smaller jobs and bug reports and other messages, which I have fallen behind on. Please have a think about the big jobs poll, and let me know if you have any questions. Thank you for your support!
Next Big Job Poll Discussion
Ok lads, as I am now finishing up OR search, I am soon going to be free to work on a new 'big job'. I am pleased that I was able to make simple Client API and OR search in much faster iterations than previously. I hope to continue like this, keeping the next big job 8-12 weeks at the most before running a new poll.
The current list is:
Just catch up on small work for a couple of months
Reduce crashes and ui jitter and hanging by improving ui-db async code
Clean up code and add unit tests
Improve tag siblings/parents and tag 'censorship'
Add ways to display files in ways other than thumbnails (like 'details' view in file explorers)
Add text and html support
Add Ugoira support (including optional mp4/webm conversion)
Add CBZ/CBR support (including framework for multi-page format)
Add import any file support (giving it 'unknown' mime but preserving file extension)
Improve 'known urls' searching and management
Explore a prototype for neural net auto-tagging
Add support for playing audio for audio and video files
Add ui for waifu2x and other file converters/processors
Write some ui to allow selecting thumbnails with a dragged bounding box
Add popular/favourite tag cloud controls for better 'browsing' search
Improve the client's local booru (this likely now means a backend migration to the Client API)
Improve duplicate db storage and filter workflow (need this first before alternate files support)
Improve shortcut customisation, including mouse shortcuts
Add ratings import/export, and add 'rating import options' to auto-rate imports
Add more commands to the undo system
Improve display of very large/zoomed files in the media viewer
Set thumbnail border colours on user-editable rating and namespace conditions
Improve hydrus network encryption with client cert management and associated ui
Add tag metadata (private sort order, presentation options, tag description/wiki support)
Improve file lookup scripts and add mass auto-lookup
Add multiple local file services (which will enable true nsfw/sfw partition)
Add an incremental number tagging dialog for thumbnails (for adding page:n etc… to a sequence of files)
Permit custom ordering of thumbnails, through mouse-dragging or otherwise
Allow user to have multiple open split tab columns or separate windows with one or more pages
Improve rating workflow by providing score representatives to compare with
Add file modified/creation timestamp searching and sorting
Write an URL Repository so clients can share known url mappings
Add animated thumbnails for videos (animating on mouseover)
Allow multiple custom 'open externally'-style file launch commands for files
Add version tracking to downloader system objects and explore remote fetching of updates
Expand file notes system
I will put up this poll with the 349 release post and then select whatever seems to be on top by 351, with the proviso that I will try to discount any non-organic (e.g. botted) votes. You will be allowed to vote for multiple items. I am happy to work on any of it.
Please feel free to suggest new items or ask for longer explanations of any of the above. I will edit the list as new items are agreed on.
Release Tomorrow!
I had a great week. The new search system for the duplicate filter went really well and is ready for everyone to use. I also improved and sped up some video import and thumbnail generation code, and I added a neat button to the manage tags dialog that lets you automatically fix siblings and missing parents.
The release should be as normal tomorrow.
Version 348
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v348/Hydrus.Network.348.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v348.tar.gz
I had a good week. It is mostly small updates and fixes.
all misc this week
After the recent weeks' thumbnail work, I became frustrated with how laggy the client's file system could get under heavy load. This week I have written a new file access locking system that has less latency under heavy simultaneous use and is also safer for certain edge cases. Multiple import queues and thumbnail fading and regular media browsing will all interact with less lag now.
The Client API has several improvements: the /add_tags/add_tags call has its 'hashes' parameter fixed and now does sibling-collapse and parent-expansion on its 'add' tags (which can you turn off if you want with the new 'add_siblings_and_parents' parameter). Also, by default the Client API service (and the client's Local Booru) no longer log their requests (you can turn this back on if you like under manage services) as this was very spammy (100MB+ logs!) and not useful for most users. And if you are an advanced user who needs CORS pre-flight for an API interface you are making, please check the new CORS checkbox also under manage services. It defaults to off. This CORS stuff is new to me, and I have made very basic responses–please let me know if it does not do enough for you.
I also wrote a first version of a manager to deal with bitmaps (which are used to draw various custom things to screen) in a centralised, safer, and more efficient way. Some non-Windows clients should now be a bit more stable, and some heavy-load animations may be just slightly smoother (as discarded bmps are now recycled). Let me know if you run into any trouble with this (or if your memory use spikes unreasonably). I expect I will do a bit more work here next week.
An issue with the yiff.party downloader not fetching certain post attachments (they got 'ignored' status previously) should be fixed. You will have to re-search yiff.party again to get the correct URLs (and not just retry any 'ignored' you have atm).
full list
* wrote some OR search help to 'getting started with tags' help page
* wrote a new multi-reader, single-writer lock object for the client file manager, along with some unit tests for it.
* updated the file and thumbnail access and regen and maintenance code to use the new lock. various access is now faster when available and overall safer. there is still work to do here
* adjusted file import to be less aggressive about locking, which should reduce some file/thumbnail access lag during heavy imports
* the thumbnail space estimate in the migrate database dialog is now more adaptive to the new more flexible thumbnail size system and specificies better that it is an estimate
* the client api's /add_tags/add_tags call now collapses siblings and expands parents on an add/pend call. this can be turned off by setting the new optional parameter 'add_siblings_and_parents' to false. the help is updated regarding this and the client api version is incremented to 6
* fixed the client api's /add_tags/add_tags call for the 'hashes' parameter, which was failing to parse, and added an accidentally missing unit test to check this in future
* the client local services (the booru and client api) now have two new options under their 'manage services' panel: 'support CORS', which turns on cross-orogin support (which is experimental for now, so defaults to False), and 'logs requests', which controls whether your log will be spammed with request reports (this also defaults to False), which should clear up some 100MB+ log hassle for people using the Hydrus Companion browser add-on
* hydrus services now respond correctly (albeit sparsely) to OPTIONS requests, and if CORS is enabled, to CORS OPTIONS requests. there are unit tests for this that seem to work ok, but I think we'll need to verify it irl
* finished a first version bitmap manager to handle all wx bitmap creation and destruction, including recycling mid-steps
* updated all simple wx bitmap creation and destruction calls across the client to use the new bitmap manager, improving stability and saving some CPU
* fixed some incorrect button alignment flags that were causing problems for clients set to assert check these values
* added a new yiff.party file url class to the defaults that matches a new file attachment format
* updated the 'url' content parser so if a parsed url is in the form 'summary_text url', as some booru source fields sometimes specify, the preceding summary text is removed, cleaning up the resultant url
* silenced an old server connection lost error that was needlessly loud
* silenced the client network engine from additionally log-printing SizeException errors when a downloading file (usually gif) exceeds file import options rules
* improved misc window destruction code
* updated supported mime list in 'getting started with files' help and website index
* misc cleanup
next week
Next week is an 'ongoing long-term job' week. I would like to (finally) add a file search object to the duplicate filter, which will allow you to restrict a series of potential dupes to a certain tag, or only archived files, or whatever else you would like. While I am in that code, I will also see if I can do some easy duplicate system de-jank work overall.
The first version of OR search is now complete. If you missed the boat, there is some official help for it now here: https://hydrusnetwork.github.io/hydrus/help/getting_started_tags.html A thread to discuss what 'big job' I might work on next is here >>12152 . I will catch up on this thread next week and make the poll alongside the v349 release thread next Wednesday.
Version 347
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v347/Hydrus.Network.347.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v347.tar.gz
I had a good week. OR search is essentially finished, and I cleaned up and fixed a variety of other things.
A new 'big thing to work on next' poll will be going up soon. If you are interested, please check out the discussion thread here:
or search
As previously discussed, I have moved OR predicate construction to the standard dropdown list below the tag input. It now appears as the top result, where you can hit enter on it to submit it as-is. Also, while under construction, a 'cancel' button and 'rewind' button (to remove the most recent OR-term added) will appear on the same panel. You can also hit Esc to cancel a currently under-construction OR predicate.
As a reminder, hold shift when you enter a tag to start an OR chain. Further shift+enter events will append new tags to the chain, and a bare enter will cap it off. I will write out some proper help for this.
OR search is basically finished as a v1.0 now. I still have some last tidy-up jobs to do, but I am overall happy with it.
the rest
I built on the past weeks' thumbnail experiments and have written a two-stage thumbnail rendering system that gets thumbs on screen faster (even if they are the wrong size, so will look fuzzy), and regenerates any needed clearer versions in the background and replaces them in-place on screen over the following seconds. It is much smoother and faster than before, and it is pretty neat to see a fuzzy thumb suddenly fade into a clearer version, but I still have a little work to do here.
Now, when you trash a file, a context-appropriate 'deletion reason', such as 'Deleted from Media Page.' will be saved. These statements are mostly trivial, but duplicate filter actions will specify a bit more about the duplicate processing action type. This text will be recovered in an import status window for 'deleted' status results, just as a help if you want to investigate closer (e.g. perhaps you are not sure why a particular file failed to import, but then you see the reason is you already decided you have a better duplicate version of it). Any files deleted before this system will just give "Unknown deletion reason."
Adding OR search caused a couple of search flaws: bare system:rating searches were delivering since-deleted files, and some searches with combinations of OR predicates with regular tags were delivering subsets of the real results. I believe I have fixed both of these, and now many previously slow OR searches should run quite a bit faster, especially when accompanied by with non-OR predicates.
I gave export folders a pass and fixed several bugs and inefficiencies, particularly for 'synchronised' folders that produce subdirectories from their filenames, which were often deleting those subdirectories. Also, and export folder or manual export event that attempts to produce a file path above the base export directory (e.g. if the generated filename begins with ..\ or ../) will now fail with some error text to explain what happened. If you use export folders a lot, particularly 'synchonised' ones, please let me know if you still get any unusual behaviour.
full list
- or search:
- under construction OR predicates now present at the top of the regular tag results list, prepended with 'OR: ', and skipping default selection
- this new OR line is enter-able, which will present it as-is, rather than adding new preds
- hitting escape on a 'search' tag input box that is empty but has an under construction or predicate will cancel the or pred
- hitting escape on a 'search' tag input box otherwise should more reliably kill its focus when the dropdown is a float window
- improved OR search efficiency significantly with dynamic OR search triggering based on other search predicates. OR searches including negated '-tag' components should be massively faster when paired with non-OR tag or file search predicates
- I believe I fixed a search issue that would sometimes return insufficient results when OR preds are mixed with certain other combinations of tags
- improved reliability of some thumbnail refresh calls
- cleaned up a bunch of OR handling ui code
- .
- the rest:
- after previous weeks' experiments, wrote new double-layer thumbnail loading system–now too-small thumbs will quckly scale up fuzzily straight to screen, and then in the coming seconds, the nice regenerated full-size thumb will be made and drawn in place as ready. it presents much faster and looks better, but there is some cleanup to do here that I will tackle next week
- all local file trashing events now record a context-appropriate deletion statement such as "Deleted from Media Viewer." this value is recovered in 'deleted' import status 'notes'. You will mostly see 'Unknown deletion reason.', for files deleted before this new system, but it will populate with appropriate info over time
- fixed a search optimisation that was not cross-referencing with file domain, meaning for instance that bare system:rating calls were returning since-deleted files
- upnp management window now uses new listctrl
- cleaned up some old custom page-naming code
- added a 'data' debug call to clear out all cached thumbnails and force an instant ui thumb reload
- fixed the trash bmp misalignment, ha ha
- removed e-hentai login script from the defaults, since this testing script is not appropriate for new users
- dejanked some media viewer video transitions by cleaning up animation bar rendering and smoothing out video buffer initialisation
- cleaned out some surplus subprocess wait calls that were hanging some systems on various 'open externally' calls
- fixed multiple syncing problems with 'synchronise' export folders that produce files with subdirectories. subdirectory structures should now be synced correctly and empty folders deleted
- export folders that collapse multiple file results to the same duplicated name should, after the next run, do less overwriting to this same name
- if an export folder or the regular export dialog makes a file destination path that is above the chosen directory (e.g. if the path starts with ../ or ..\), the export job will error out with an explanation
- big manual file exports _should_ be politer to the ui and cause fewer hangs
- doing page tab drag and drops may have less post-drop ui jank on linux, continued feedback would be appreciated
- moved 'reason' handling for all content updates to its own area, which neatens many content update data handling issues
- fixed petitioning a tag via a shortcut, which had bad reason handling
- fixed an issue with committing pending ipfs items that was overchecking service permissions
- fixed some remaining bad wx code in the unit tests
- misc file status reporting cleanup
next week
I'll tidy up some last OR search stuff and clear out some small jobs. I would like to reduce some lag when the client file manager has a lot of competing access (e.g. when lots of new thumbnails need to be generated), and I would also like to improve some Linux stability with some unified bitmap management.
Hydrus software optimization thread
ITT: create proposals for making Hydrus more optimized.
Proposal: Why can't Hydrus switch to MariaDB?
If it is faster, then it should be better. The only trouble is having the need to rewrite the queries, which from an SQL standpoint should be a non-issue, right?
List of Databases with Open Source License and Open Source APIs:
SQLite - Currently used in Hydrus, has minimal features
MySQL - A more well-rounded SQL Database with user management
PostgreSQL - An SQL with complex features with less performance
MariaDB - SQL/NoSQL database with heavy optimizations
ElasticSearch - A literal search engine instead of a normal Database
Teradata - IDK
https://www.infoworld.com/article/2611812/mysql/mysql-face-off--mysql-or-mariadb-.html
Release Tomorrow!
I had a good week, mostly working on small jobs and fixes. File and thumbnail access should be less laggy, some drawing code should be more resource efficient and stable, and the client api has a bit of work as well.
The release should be as normal tomorrow.
Version 346
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v346/Hydrus.Network.346.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v346.tar.gz
I had a great week. There's a bunch of fixes and thumbnail improvements, a more powerful manage urls dialog, and a prototype OR search to try out.
thumbnails
Last week's big thumbnail change seems to have gone fairly well overall. There do not seem to be any huge errors anywhere, and the simpler file system and code is great, but I was annoyed by how slow certain regeneration scenarios turned out to be, and how some thumbnail resizes were a bit blurry.
To address this, this week I have cleaned and improved a bunch of thumbnail loading code. It was much nicer to do now I only have one thumbnail file to juggle. Thumbnails should now be a less blurry when they scale up, and they should generally load a bit faster whenever a lot of regenerations need to be done (you should notice CPU-expensive video thumbs load last of all, while easy ones like pdfs and pictures should come in first).
I was also annoyed that even small changes in desired thumbnail size would force a complete regen, which was particularly laggy when scaling up. So, to smooth out the loading time when you have recently changed the desired thumbnail resolution, I have added an experimental regeneration throttle. When the difference between what is available and what is desired is just a few pixels (and so a scale will not be too fuzzy), rather than regenerating from source, the client will now mostly just scale up or down what it already has. When the difference is great, it is more worth spending the CPU on a full regeneration to get a clear thumbnail, so it will regenerate more often. It still eventually regenerates everything, but when the difference is slight, the real workload will be spread over several loads. For some example numbers, if the thumb on your disk is 120x120 and your options want 125x125, the client will expensively regenerate from source only about 29% of the time and scale up the original 120x120 about 71% of the time. But 120x120 vs a desired 250x250 will regenerate/rescale about 84%/12% of the time. This is a big experiment, but I am generally pleased with it: it does speed up thumbnail load in many situations that were previously needlessly slow, and the relative fuzziness is generally not a noticeable problem. I hacked the percentage ratios out of some statistical polling that I can definitely tune in future, so please let me know if your situation produces too many very fuzzy thumbs.
Also, if you are in 'advanced mode', the thumbnail right-click menu now has more 'regenerate' options: as well as doing a complete file reparse and thumb regen, you can now choose to just regenerate thumbnails or only regenerate thumbs if their size is wrong. If you are an advanced user, please feel free to play with these new commands, particularly if you notice fuzzy thumbs from the throttle above, and let me know how it goes.
I expect to do more work on thumbs in the coming weeks. As well as tuning the new regeneration throttle, I would like to expand the new database-level thumbnail regeneration modes to work during maintenance time. I have long planned a 'file reparse' db maintenance mode so I can finally fix all the ancient mistaken animation frame counts and mime mismatches like apngs/pngs and mkvs/webms, and now this same maintenance command can also just do thumbs, I would love to let users say 'please regen all thumbs that are the wrong size in the background during idle time, no more than 20mins/day'. There's no reason why this stuff should be only on command for advanced users, so now I need to build the maintenance structure for it.
I'd also like to extend the current thumbnail location override under migrate database to work just like the files, so you can balance thumbnails over multiple weighted locations.
Linux and OS X users: You may not have received an error, but it is decently likely that last week's 'rxx' folder delete did not work for you. This was due to a permissions-setting mistake on my end. I have fixed this permissions code, and on this 346 update, if you still have some rxx folders in your db, the client will attempt to re-do the delete. If it wants to do this, it will give you another popup message as you update, informing you of the situation before it goes ahead.
or search
Several things came together this week, and I managed to get an early prototype of OR search working completely. I am keen to get users' thoughts on how to finish it off.
To start an OR search, hold down shift when you enter a tag in a search page. Rather than adding this tag to a search, it will now place it in a new box just above the input box. Further shift-enter actions will increase the OR 'chain', making 'tag A OR tag B OR tag c', and a final non-shift enter will cap the chain and send the OR search predicate up to the main search. It works! You can even add system predicates!
However, this entry method is not finished. Have a play with it and you will see what I mean. There is no neat way to cap off an OR chain early (i.e. without typing anything new: if you hit a bare enter on it, you'll get system:everything or system:inbox on the end), nor to 'go back one' if you put the wrong tag in. Also, system predicates that need additional panel infomation before 'going through' do not hold on to the shift status. Beyond the bugs, I am interested in what you would like for the workflow here–would you rather the OR-chain under creation be inserted as the top item in the list below, rather than in the new box above? Should it be highlighted as under construction somehow? Should it have a dangling OR while under construction, such as 'blue eyes OR green eyes OR' rather than 'blue eyes OR green eyes'? How should it be cancelled, or undone a step? What would be the best way to say 'ok, I have A OR B OR C entered, and I don't have a D to add, so I want to enter what I have now'?
the rest
I made two more stupid errors last week–one that draws white frames over animations on certain media update events, and one that occaisonally set the 'include current tags' value on search pages to 'false' on reloads. I have fixed both of these issues, and as a one-time for this update, all 'include current tags' values are forced back to true. I am sorry for the inconvenience here. The white frame stuff drove me nuts all week.
Beyond that, I extended the 'manage urls' dialog to support multiple thumbnail selections, which allows for easy mass-deletion of bad URLs. The URL list on this dialog also now supports multi-selection and accepts the delete key. It also supports adding URLs to multiple files, but when opened on multiple files it presents a little red-text warning to remind you that adding a URL to multiple files only makes sense for gallery URLs. I expect to do a little more work on this in future, cleaning the workflow a little more and detecting the url class of urls you attempt to add to multiple files, specifically testing what is and is not appropriate to add. In any case, like everything else this week, please give it ago and let me know how it works for you!
Also, the 'move files now' button on database->migrate database now lets you limit the max time it will run to 10/30/60 minutes! This is a nice stopgap if you, say, have 600,000 thumbnails to move and don't want to do it all at once.
full list
- or search:
- extended the search predicate object to handle more OR stuff
- extended the tag list to handle list objects that have multiple colours
- extended the new OR search predicate to report multiple text-snippet-and-colour pairs based on sub-predicates
- extended tag search input to handle prototype OR predicate creation–hold shift when you enter the tag, and it'll start an OR chain. shift-enter continues the chain, enter alone completes it
- fleshed out the predicate unit tests to cover more of this
- wrote unit tests for OR search predicates. it seems good!
- improved some search logic to apply system preds better in certain edge cases and spend less CPU on OR-search-only searches
- .
- thumbnails:
- thumbnails will now queue for load in a more intelligent order based on estimated difficulty to regenerate, which will tend to put more thumbs on screen faster
- the decision to regenerate a thumbnail from source is now tempered by how different the current thumbnail is from what is desired–the more similar the two sizes, the more (randomly) likely the client will decide to just use the current (resized) this time. this smooths out change-lag while limiting the number of really fuzzy thumbs you get. feedback on how this works IRL would be appreciated–it uses some voodoo distribution polling to figure it out, which I can definitely tweak
- improved visual quality of thumbnail scale-up optimisations
- fixed an issue where a multipage thumbnail grid would incorrectly recalculate the new virtual height after a thumbnail size change event, leading to a bit of invalid extra scrollspace (with noclip rendering errors) at the bottom
- the thumbnail right-click menu's reparse files entry is now extended to a new 'regenerate' submenu with three options: reparse file and regen thumbs (the old action), force regen thumbs, and regen thumbs if wrong size!
- the new 'regen if thumbs wrong size' action sends how many thumbs needed resize up to the popup window, as well
- moved some old thumbnail regen code responsibility out of the db and into the files manager
- cleaned out some old redundant file/thumbnail code
- cleaned and refactored a bunch of general image handling and resizing code
- .
- the rest:
- fixed some bad serialisation code that was making file search objects set their 'include current tags' value to false/true on interleaving loads. on this update, all 'include current tags' values are blanket reset to true
- fixed an issue that was drawing animation canvases pure white on various media update events
- extended manage urls dialog to support multiple files when launched from a selection of thumbnails. there is a warning in this case, noting that only gallery-style urls are appropriate to be added to multiple files
- manage urls dialog now supports multiple selections, including shift-select, and accepts delete key presses for easy mass deletion
- when you ask the database migration dialog to move some files, it now pops up a confirmation dialog that also asks if you would like to limit the max time for the job as 10, 30, or 60 minutes
- improved file permission setting code across the program to be more sensible for non-Windows
- if you are a non-Windows user and were hit with directory permission problems last week on the thumbnail update–which resulted in the rxx directories not being deleted–the update this week will attempt to do the delete again, this time correcting the now missing execute permission bit. if it finds outstanding rxx directories to delete, it will give a popup beforehand summarising the situation and giving you a chance to bail out
- fixed yet another problem that was stopping client api url requests from finding the correct page by name
- when a client api url request includes fixed tags, these tags should now propagate in all scenarios where the single url produces multiple files
- updated sqlite dll and console for windows
- misc fixes and cleanup
next week
Next week is a 'cleanup' week. I'll keep chipping away at the listctrl rewrite and other boring jobs, and I'll see if I can finish up the OR search entry workflow in a sane way.
Thanks everyone!
Version 345
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v345/Hydrus.Network.345.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v345.tar.gz
I had a good week. The big thumbnail update is done, which means for an important update. Users with smaller databases have less to think about, but anyone with 1,000,000+ files should definitely read all of the following:
thumbnails
This is a great week to make a backup before you update!
Until now, the hydrus client has used two simple thumbnails–one 'full-size', one 'resized'–for each file. Hardware has moved on since then, and so has my code, so it is now feasible to have a single smarter thumbnail that will resize itself on demand. The work for this was done this week. This makes for a simpler and less wasteful storage system and more flexible thumbs all around.
The main thing is that on this update, the client will delete the surplus thumbnails. You will get a popup before it happens to remind you. Specifically, it will delete the 256 'rxx' 'resized' subfolders in your database's file store. If you have moved these folders through the migrate database dialog, this is no problem–the client will delete them wherever they are. Depending on your collection and platform, it could take a few seconds or several minutes. Your new thumbnails may load just slightly slower the first time, but this will return to normal in time.
The best news is that the old 200x200 limit is gone! You can now set your thumbnails' size under options->thumbnails up to 2048x2048! I am now running 360x240 on my 4k machine and like it a lot.
Note that if you previously moved your 'resized' thumbs to an SSD but not your old 'full-size' thumbs, you will want to recheck the migrate database dialog at some point to make sure you move your newly active thumbnail folders to fast storage. This is likely to be a big job, so plan for it.
The migrate database help regarding all this has been updated:
https://hydrusnetwork.github.io/hydrus/help/database_migration.html
Also, be prepared for your next backup run to have significant additional delete overhead as all your backup's 'rxx' folders will be deleted as well. If you have fewer than, say, 250,000 files, I do not think you need to think much of this, but if you have many more, you might like to manually delete these folders from your backup yourself right before your next run. There are 256 of these folders, named in hexadecimal from r00 to rff. Preemptively deleting them will save you backup-scanning time and keep your recycle bin sane (if your backup program deletes to recycle bin). Make sure you do a permanent delete (Shift+Delete on Windows) to skip over the recycle bin yourself.
I tested the new system thoroughly, but as always, some unusual scenario may present problems. The system is more complicated than 'if the thumb is the wrong size, resize it and save it back to disk', so some edge cases may throw an error. Please report any bugs, and I'll try to have them fixed for 346.
full list
- or search:
- set out a plan to achieve some simple conjunctive normal form (e.g. (blue eyes OR green eyes) AND (blonde hair OR red hair)) OR search support
- started work on the object extension and search code to support this search in a very basic (and likely inefficient-for-some-scenarios) way–we'll work on this as we discover the most common inefficiencies
- .
- thumbnails:
- the client no longer uses both 'master' and 'resized' thumbnails–it uses a single, smarter thumbnail
- only the 'txx' thumbnail directories (formerly referred to as full-size) are now used, and the thumbnails inside will regenerate and scale themselves as needed on demand (and will be careful to not save changes to disk when when their source file is non-local)
- the old 'rxx' 'resized' thumbnail directories are no longer referred to anywhere in the code or ui
- the old 'rxx' thumbnails directories will be permanently (i.e. no recycle bin) deleted on update. this is a big job, and you will be prompted on update before it happens
- if you have migrated your db to put 'resized' thumbs on an SSD but not the formerly 'full-size', you will want to recheck the 'migrate database' dialog once you have booted and set a new thumbnail override to move the txx directories over
- due to the smarter thumbnail, 200x200 is no longer the hard limit for hydrus thumbnails! you can now set up to 2048x2048
- all file storage location information is now stored directly in the client db (rather than the options object), which should make for more easily export/importable options in future and improve manual fixing as needed
- added more thumbnail-resizing related popup spam to file report mode
- fixed a windows-only issue that was making the migrate db dialog close after a file move event concluded
- updated database migration help for new concepts and ui
- cleaned up some misc storage code
- .
- the rest:
- fixed a problem in the client api with fetching file identifiers from file_ids
- fleshed out 'help my db is broke.txt' with more specific clone recovery examples
- fixed import support for a variety of single-frame music webms
- fixed an edge-case preview viewer initialisation bug that was trying to draw the canvas before any media was set
- network report mode now states url classes of urls about to be parsed
- misc small fixes and cleanup
next week
I got started on OR search this week. I have a decent plan and feel good. I hope to have some sort of prototype ready for 347. Otherwise I will do some small jobs. I may look into adding multiple-files 'known urls' management.
Release Tomorrow!
I had a great week. With the new simpler thumbnails, I improved how thumbnails load and added some new thumbnail maintenance actions. I also extended the 'manage urls' dialog to work for multiple files and got the very first simple version of OR search working and ready for user testing!
I also fixed the unusual 'include current tags' and white-frame animation bugs.
The release should be as normal tomorrow.
Version 344
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v344/Hydrus.Network.344.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v344.tar.gz
I had a mixed week, but I got some good hydrus work done. It is basically all misc this week.
highlights
The Client API v1.0 got its last polish this week. I fixed some bugs and added a new parameter to control page selection to the /add_urls/add_url command. Future Client API work will be entirely in regular weekly work. I'd like to add wildcard and namespace tag searching, more system predicates, autocomplete tag searches, optional https to encrypt communication outside a LAN, cookie.txt import, and I am sure many other things. The basic bones are decent now though–I now need only hang new things off it.
Animation scanbars (the draggable area just below a gif or webm in the media viewer) now have millisecond-precise x/y progress timestamps! This was fun to do, and it worked out pretty neat–it even works for variable framerate gifs.
I have added barebones .psd (photoshop document) file import support! This is a complicated format that supports image layers and so on, so I cannot provide any native rendering, but it will parse width and height. It presents as 'application/x-photoshop'. PSB files (which support huge 300k width/height) are also recognised, if you happen to have any. Hydrus will treat them as .psd as well. This has been asked for several times, so please let me know if you find useful workflows here. Note that hydrus assumes its file contents will never change, so do not try to load a hydrus-stored .psd in Photoshop and then save any changes back! This is best for 'finished' psds that you are effectively archiving.
There is a new 'open_known_url' shortcut under the 'media' shortcut set. It should work on a focused thumbnail or the media viewer: on activation, if the file has one 'recognised' URL (the ones listed in the top-right of the media viewer), it will launch in your browser, and if it has multiple, you will get a list of all known URLs you can quickly navigate with arrow keys and enter to choose which you want to see.
full list
- final v1.0 client api polish:
- added optional 'show_destination_page' arg to '/add_urls/add_url', defaulting to False, to control whether an URL-add will select (i.e. jump to) the destination page in the ui. this changes the default behaviour for this command
- simplified the routine that finds or creates a watcher or url import page and fixed a bug in the api that was not creating new pages when destination_page_name was specified
- some misc cleanup
- fixed fetching file_metadata by hashes
- fixed the client api help regarding file_metadata response example tags
- client api version is now 5
- .
- the rest:
- psd support added! because of this format's potential multi-layer complexity, it will not render natively, but width and height are parsed. it is treated as 'application/x-photoshop'. PSB is also recognised and treated as psd
- added a 'open_known_url' shortcut to the 'media' shortcut set that lets you quickly open URLs for files. if there is one recognised known url, it will be launched, and if there are multiple, a list with all known urls will appear to select which one you want
- animation scanbars now show an x/y current timestamp! it includes millisecond timings and even works for variable framerate gifs. whether to show a second/shadow caret for timestamp position on variable frame rate is a new discussion to have
- fixed an issue where animations would sometimes not resume animation for several seconds after a big scanbar drag
- when the thumbnail manager cannot produce a thumbnail due to a storage error (like a missing file), it now only puts up a single, more informative error popup on the first problem. subsequent errors are printed silently to the log. (these errors tend to come in en masse, so this cuts down on spam and error-related ui lag that was making loading a bad session difficult)
- improved error reporting when an upload pending command would fail due to service non-functionality–it should now give a popup with error info imediately, rather than obscured through the login system
- added temp_dir parameter to the client and server that will override which temporary directory the program will use
- cleaned up how no_daemons and no_wal mode are handled internally
- no_wal mode now has to be called from the command parameter, the no_wal file hack in the db directory no longer works
- missing ffmpeg errors now prompt the user to check if it is installed
- searching for numerical ratings should now work for files that were rated when the service had a different number of stars (ratings now searches in 'bands' rather than exact values)
- reduced the min height of the new import files frame's list
- doubled the decompression bomb test to permit files up to ~179 megapixel, we'll see how it goes
- misc cleanup
next week
I have some IRL stuff going on atm. I hope it will not affect my hydrus time, but it may steal a day or two from me. If things line up well, the situation should be resolved within a week or two.
With Client API done for real now, I will move completely to OR search. Otherwise, next week will be an 'ongoing work' week. I hope to pull the trigger on eliminating hydrus's double-thumbnail system (i.e. moving to just one thumb per file). The db update next week will thus likely be a big deal, maybe several minutes' of deleting for users with larger clients.
Release Tomorrow!
I had a good week. The big thumbnail improvement is done, making for smarter thumbnail access and storage across the program. Also the max possible size of thumbs, previously 200x200, is now uncapped! The update tomorrow will take a little while, as the now-surplus half of your thumbnails will have to be deleted.
The release should be as normal tomorrow.
Release Tomorrow!
I had a mixed week, but I got some good hydrus work done. I added some neat x/y timestamps to the animation scanbar, made a way to launch file urls with a shortcut, and added barebones .psd import support.
The release should be as normal tomorrow, but perhaps a little late if some IRL stuff comes up.
Version 343
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v343/Hydrus.Network.343.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v343.tar.gz
I had an ok week. The Client API v1.0 is essentially finished, and I did some code cleanup as well.
client api
The Client API now has full 'file search' capability, including fetching file metadata like size and resolution and tags, and also file and thumbnail fetch. As before, the help for these new commands is here:
https://hydrusnetwork.github.io/hydrus/help/client_api.html
But for some tiny last cleanup jobs, the Client API v1.0 is now complete. From now on, I will fit adding additional commands to the API in normal weekly 'small job' work.
I am overall pleased with how this has gone. If you have been waiting for a 'working' version of the API to play with, now is the time. For regular users, there is a link to a browser add-on in that help file, and it seems a phone app is also in the works.
the rest
The media viewer is now smarter about recycling its internal windows, so browsing between video and static image types (in either direction) is now faster and smoother. If you often get the one-frame 'tiny grey square' when transitioning between video and static images, this should now be eliminated.
More types of tiff are now supported, and I fixed an issue that was mis-detecting soundless webms as mkvs.
The list of file paths in the manual file import window is now sortable, and it will preserve this sort order in the subsequent file import or path tagging dialog (useful if you wish to set up sequential number tagging, or you would like for some reason to import by largest file first). It also has a new '#' column to let you return to the original parse sort order.
full list
- client api:
- fixed an int/str type mismatch issue with service_names_to_actions_to_tags in /add_tags/add_tags in the client api that meant that argument was not working
- fixed up some last /get_files/search_files stuff
- added /get_files/file_metadata
- added /get_files/file
- added /get_files/thumbnail
- added help and unit tests to reflect the above
- updated client api version to 4
- .
- the rest:
- the list of paths in the manual file import dialog is now sortable. this order will be preserved in regular and 'add tags' ok events for this dialog. it has a new '#' column so you can return to 'parse' order if desired
- animation and static image windows in the media viewer canvas are now recycled through media type transitions, making for slightly smoother browsing between mixed media
- increased aggression of media viewer image prefetch
- added support for 'MM' Tiffs
- fixed webm mime parsing for webms with no audio (these were falling back to mkv)
- improved the error reports when a serialised png fails to import
- the mass-open-urls popup is now pausable as well as cancellable
- fixed several recently broken ui unit tests
- misc old code cleanup
- some misc test controller/constant refactoring
next week
I'll finish up the last Client API work and move on to tag OR searching in the client as the new 'big job'. My hope is this will not be a gigantic project, but we'll see how it shakes out. Beyond that, I'd like to fit in some small jobs–maybe some time units on the animation scanbar, maybe some new shortcuts.
This week got away from me a bit, so I also have a ton of bug reports and other messages to catch up on.
Version 342
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v342/Hydrus.Network.342.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v342.tar.gz
I had a good week. There are several new ui features and bug fixes, and webp and tiff now have basic support.
tiff and webp
Tiff files are now supported! Tiff is an old format typically used for photographic and scientific uses (think 12,000x12,000px images of the moon). They'll import to hydrus like any other image and will work on the duplicate file system. I tested 24bit and 8bit (monochrome), but could not find a 48bit example–even the easily accessible NASA stuff was all 24bit–so I don't know if high colour-depth tiffs work. If you have an example of one, please send it to me, and if it doesn't work in hydrus, I'll see if I can figure it out. EDIT: Some tiffs are not supported. I know basically what this is and will fix it for next week.
And webp files are now supported! Webp is a newer initiative from Google to replace jpgs and pngs. It has some impressive file size reductions, but hasn't picked up all that much yet. They'll now import to hydrus like any other image and will work on the duplicate file system. WebP also supports image-animations, much like gif, but I haven't written support for this yet. For now, hydrus will assume all webps are static images, and animations will only give you the first frame (just like how apngs used to import when hydrus considered them pngs).
I did not have to do all that much work to get tiff and webp working–I just hooked in my existing image libraries, PIL and OpenCV. If you run from source and have slightly older libraries here, your support may not be complete. I hope to keep working here and try to get animated webps working. If you are enthusiastic about webp, let me know how you get on–they mite b cool for some purposes, and if the library support for saving them is there, I might start using them in hydrus's thumbnail system, including for the long-planned animated thumbnail system.
otherwise all misc this week
Tag autocomplete search should be a bit quicker to update when you temporarily pause on a large search (e.g. if you pause on 'title:' and then type 'gar', the big laggy search for 'title:*' will now be cancelled much faster so the quicker 'title:gar*' can replace it).
Every one of the complicated 'copy' URL thumbnail menu entries is now replicated on the 'open' menu! So, if you select five files that all have twitter URLs, you can hit right-click->known urls->open->these files' twitter tweets to open them all in one go! It will open them one at a time, one per second, so as not to nuke your browser. If asked to open more than a few URLs like this, the client will warn you with a yes/no dialog and throw up a popup that will let you cancel the operation (just in case you accidentally try to open 500 URLs!).
By default, 'system:everything' is now hidden in search contexts with more than 10,000 files! This search predicate is sometimes useful to new users, but experienced users hit it more by accident than not. If you would rather have it anyway, you can force it to appear by hitting the new checkbox under options->default system predicates.
Gallery download pages now have a more informative 'status' column–they'll report gallery fetch status and say 'done!' when all work is complete.
After many attempts and some great help from users, I believe I have finally fixed the Linux 'similar files search with >0 distance' crash. One user and I managed to isolate the program area and figure out a cleaner layout, and it seems to be fixed. If you have had this problem, please give it a hesitant go and let me know what happens.
Also, I cleaned up some video rendering logic and memory management. Videos should render a little smoother and more responsively, and they will clean up after themselves better as well. This may have fixed a long-running but rare all-platforms crash related to launching or closing the media viewer on a playing video.
full list
- added support for webp import. it does not yet support animated webps, which, if the local platform supports, will import like apngs used to: just the first frame
- added support for tiff import. it works ok for 24bit and 8bit (monochrome) tiffs, but I am not sure how well it will do with 48bit
- both webp and tiff should work on the duplicate files system
- improved webm detection to include opus audio (previously, these files were falling back to mkv)
- fixed an issue where unusual formats with duration but no frames or frames but no duration were being sorted and otherwise presented incorrectly
- improved autocomplete job cancelability. this job can now cancel much faster on large jobs, meaning typing searches with large result sets will hit less CPU and return faster on subsequent keystrokes
- _all_ of the complicated 'copy url' commands from the thumbnail right-click->known urls menu are now available on the 'open' submenu! if there is more than one url to open (e.g. 'open all of these files blahbooru post urls' on a selection of 50 files), you will be presented with a yes/no dialog to confirm, and it will open one url in your browser every second (with a cancellable popup if num_urls > 5)
- by default, system:everything is now hidden if its total files is >10k. you can force it to always show under options->default system predicates
- the gallery downloader's list's status column now shows gallery status (deferring to active file status) when appropriate and shows 'done!' when all work is complete
- after working back and forth with a user, I _believe_ the linux similar files >0 distance search crash is finally fixed
- fixed sorting by media views/viewtiming with collections
- a single-selected collection right-click now shows total media views for all files in the collection! you can now see how long you have been viewing an artist!
- fixed an issue that lead to export folders not running on always-on clients as often as they should
- updated the gelbooru 0.2.5 file page parser to pull rating tag from the correct location (previously, it was pulling from what appears to be a site-wide 'mature' browser hint)
- improved memory cleanup stability when animations and other parts of the video rendering pipeline are deleted–this _may_ fix some rare crashes
- increased animation rendering aggression overall and particularly in 'future' of frame buffer
- if a video renderer that is asked to start some way into the video fails to render anything, it will now fall back to trying to render from the beginning. this is slightly hacky atm and leads to out of phase rendering frames, but it is better than an error
- added a '–no_db_temp_files' launch parameter that will force the client or server to return to the recent old behaviour of exclusively using memory for journalling. this is useful if your temp directory is small and/or your available ram is very large. if running in this mode, the client will attempt to check available memory (instead of free space on your temp dir) before performing very large transactions
- with the new lighter-weight update transactions, the client now tests for less free space for journalling before running repository update processing
- added /get_files/search_files to the client api, which does the first half of file searching. it allows tag search (including -tag negation) and system inbox/archive. since the second half, which will fetch file metadata, is not yet in, this can't do anything interesting yet
- updated help and unit tests to support this, client api version is now 3
- some misc refactoring
next week
I got the first half of Client API file search done this week, but it is useless without the other half, so that's the top priority next week.
Also, after thinking about it a bit and talking with some users, I am going to try changing my schedule just a little. I always enjoy stability and cleanup work, and I am happy I found time recently to do some more, but it is often difficult to remember this kind of work with the pressure of a hundred smaller (and often sexier) jobs. With this in mind, I will try having a rotating schedule where I do one week focusing on stability and cleanup (including improving unit tests), one on larger ongoing work (like the recent autocomplete async work and the thumbnail experiment last week), and two on regular smaller jobs (like most of what I did this week). I'll go cleanup, small, ongoing, small. I will still work on bugs and the 'big job' as normal every week, but the rest of my time will be more focused on one thing. This will force me to spend at least some time on each important area every month without turning my week inside out trying to cram a bit of everything in.
Anyway, I'm going to give that a go and see how it feels. I will do cleanup next week. There are a ton of broken unit tests that need catching up to current code, and I'd love to move forward the listctrl object replacement. Both of these are boring as anything and have been put on the back burner, so let's see if I can concentrate next week and move them forward!
Version 341
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v341.tar.gz
I had a great week after being ill at the end of the previous week. There is a bit more than usual in this release because of the extra time. The Client API has some new commands, and there are a variety of fixes and stability improvements.
client api
The Client API now does some fun stuff. If you would like to play with one user's browser extension, which recognises hydrus-compatible URLs–including whether you have the page's file already–and lets you send them straight to hydrus for downloading, please check it out here:
https://gitgud.io/prkc/hydrus-companion
And the Client API help itself is here:
https://hydrusnetwork.github.io/hydrus/help/client_api.html
There is a little info at the top telling you how to turn the API on.
This release brings some more API commands. You can now edit a file's tags, edit a file's URLs, and the 'add_url' command now lets you attach tags for any files coming from the given URL and also lets you determine which page (by page name) the URL will go to (for instance if you have multiple watcher pages working on different check timings for different boards).
the rest
I spent some time this week cleaning things up. Repository update processing uses less memory and now regularly saves its progress on big jobs, reducing transaction bloat, improving error recovery, and giving control back to you much faster when it finishes. Memory use all around should be reduced.
I also replaced several areas of messy UI code, which has resulted in improved stability, particularly for Linux. Some CRITICAL logspam is gone, and I believe I have finally fixed a lingering ui crash on the duplicates processing page (at least for search distance 0–I will still work on >0, which seems to be a db-level problem for linux). Several boot crashes due to incorrect boot-error handling (such as when catching that some client_files directories are missing) are also fixed. I would appreciate your continued feedback here.
Duplicate processing now treats ratings copy/move/merge actions a little different: Before, it would only make a change if the 'destination' file had no rating set–now it will do it if it has no rating or if the rating is smaller than what the 'source' has. For instance, merging two files that have rating 2/5 and 3/5 will result in both files having rating 3/5.
After talking with a user about different ways of storing and presenting thumbnails, I put some work into a thumbnail drawing experiment that only needs one set of thumbs to draw on screen (the current thumbnail system stores two copies for different purposes). If you would like to try it temporarily, please turn it on under help->debug->gui actions->thumbnail experiment mode. Instead of using your 'resized' thumbs, it will only use the 'master'. It makes no changes to your file structure, and you can turn it off at any time. Try loading up some random files in your client and let me know if you get any errors or if any thumbs take an extremely long time to load. I've wanted to drop the two-thumb storage system for some time but never had good numbers to feel good about it, but my tests here are promising–the experiment mode only takes a few hundred µs more per initial thumb load for me, and if I were to switch over to it permanently, I could reduce it even further. Assuming there are no big problems here, I expect to chip away at this problem over the coming weeks and hopefully make a significant simplification in hydrus file storage.
full list
- client api:
- added /add_tags/add_tags, which does several kinds of tag content updates
- added /add_tags/clean_tags, which shows how hydrus will handle potential tags
- added /add_urls/associate_url, which allows you to associate urls with files
- added 'destination_page_name' to /add_urls/add_url, which will choose which destination watcher/url importer to place the url (or create a new one with that name)
- updated client api version to 2
- updated client help and unit tests for the above
- added a linked contents to the client api help
- improved some server error handling, mostly moving 403s to more correct 400s
- improved how missing parameter 400 errors are reported from the server vs deeper keyerrors that should be 500
- .
- the rest:
- tag repository update processing now saves progress to disk every million rows or every minute, whichever comes first. this reduces journaling bloat, improves recovery when the process quits unexpectedly, and makes for significantly faster cancel when requested by the user
- when processing duplicates and copying/merging/moving ratings, the 'source' file will now also overwrite the 'destination' file's rating if that destination rating is lower (previously, the rating would only go over if the dest had no rating set)
- added a new 'thumbnail experiment mode' under help->debug->gui. this will load fullsize thumbs and resize them in memory, please see release post for more details
- reduced menubar replacement flicker while, I believe, keeping and strengthening recent menubar indexing stability improvements
- the tag autocomplete dropdown will now always embed (instead of floating) in non-Windows
- when data seems non-decodable, the fallback encoding format is now that given by chardet, rather than utf-8
- improved serialisability of some pending tag data
- watchers can now hold and pass on fixed pending tag data
- gallery log objects can now hold and pass on fixed pending tag data
- file import objects can now hold and action fixed pending tag data
- hard drive imports now store their paths-to-tags info in this new format, directly in the file import objects
- improved some url-import page drop-target-selection logic
- improved error reporting when dropping/api-adding urls
- adjusted some url import workflow so big 'already in db' download lists should work a bit faster
- attempting to start the program with some external database files but not the main 'client.db/server.db' file will now cause a boot-fail exception with an explanation before any stub db files can be made
- tightened up some hydrus service login-capability-testing code that was previously stopping certain error states from recovering promptly, even on a force account refresh, while the service was maxed on bandwidth
- fixed a source of linux CRITICAL logspam related to several common dialogs
- improved ui stability on boot when file folders are missing (particularly for linux)
- improved stability for the various async tasks on the duplicates processing page, particularly for linux. I am not sure I got everything here, but it is definitely better
- did some more misc stability improvements, particularly in various boot fail scenarios
- completely removed an ancient and janky focus catcher widget from the main gui frame
- now various db caching is improved on the python side, removed a sqlite instruction to force temp information to always stay in memory–hot data returns to staying mostly in memory to start and then spools to disk if the transaction gets too large
- fixed approx bitrate sorting for malformed video files with explicitly '0' duration
- daemon_profile_mode now spams some more info about export folders
- fixed an issue that meant client db maintenance was firing its jobs too aggressively, regardless of idle status
- updated windows build to cv 4.0
- misc refactoring and fixes
next week
We are in the home stretch of the Client API now. I just want to write some simple file searching and it'll be done. I do not expect to be done in one week, but I think I should have something working for 342.
It felt good this week to clear out some long-time bugs doing code cleaning, so I'd like to focus on that as well.
Release Tomorrow!
I had a good week. I fixed several bugs (including I think the Linux >0 distance similar files search crash, finally!), did some neat ui work like better tag autocomplete responsivity, more stable and slightly smoother video rendering, and more 'open urls' options on multiple selections, and added some basic webp and tiff file import support.
The release should be as normal tomorrow.
Release Tomorrow!
I had a great week after being ill at the end of last week. 341 is a big release because of the extra time. There are several new Client API commands, bug fixes and more memory reduction, and some important stability improvements, particularly for Linux.
The release should be as normal tomorrow.
Version 340
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v340.tar.gz
I had a great if busy week. The Client API does more, several bugs are fixed, some new features and options are in, and stability and memory use should be a bit better.
client api
This is still somewhat advanced, but users who are interested may want to start looking in.
The first version of the Client API last week went ok! There were a couple of bugs, but thanks to feedback from some advanced users, I've improved reliability and carved out a better spec. This week brings JSON everywhere, fixes the add_file crash, and adds two neat calls:
/add_urls/get_url_files now looks up which files are attached to a URL
/add_files/add_file lets you import a file, either from a path or raw bytes
Please check the updated help here for more details:
https://hydrusnetwork.github.io/hydrus/help/client_api.html
the rest
I fixed the stupid 'page of pages' closing bug. I apologise if you were hit by this. I got a little blackpilled about all these typo errors not being caught, so I wrote a new testing suite to better test core ui functionality. I will expand this over the coming weeks and hope to completely eliminate the most blockheaded problems.
I have added nijie.info to the downloader defaults. It needs a login to access. I also updated the default danbooru parsers to get 'rating:' tags. I'd like to fold in some more downloaders to the client defaults in the near future. sfw FurAffinity should be doable next week. If you have a simple working downloader that you think would be worth rolling in for all new users, please suggest/submit it.
I added a semi-hacky checkbox to options->files and trash that pauses all new file/thumbnail requests for 15 seconds after your computer wakes from sleep. If you store your client's files over a NAS or other network solution (that may take a couple of seconds to reconnect after wake), give it a go and let me know if you like it.
I did a bunch of little work on stability and memory management and failure recovery this week. If you have had trouble with the menubar or a bloaty client or invalid website encodings or subs that fail to load, see if any of it is better here. OS X users with retina (i.e. high-dpi) screens may also see less blurry ui–I'd particularly like feedback in this case, good or bad.
full list
- client api:
- fixed up some api permissions object stuff so that /verify_access_key response can always serialise correctly
- fixed the 'add_url' api call's instability
- the API will now always return JSON on 200. anything else should be presumed to be raw text
- '/api_version' now returns JSON, and after talking with users, it will now start incrementing with every api change. it remains 1 just for this week
- '/request_access_permissions' now returns JSON
- '/add_url' now results JSON on success with more info, 403 on failure
- '/get_url_info' now returns the 'normalised_url' in the response JSON
- added '/get_url_files', which returns 'url_file_statuses', listing known hashes and file import status for that url
- added '/add_files/add_file', which can import a file from a path or bytes
- added '/add_tags/get_tag_services', which will return info on the client's tag services
- updated client api help to reflect the above changes and fleshed out the intro a bit
- fixed the client api permissions enum values in the help, which I somehow transcribed wrong first time
- updated the client api tests to check the above
- refactored client api tests to be neater and in their own file
- .
- the rest:
- fixed the page of pages close bug
- added a downloader for nijie.info to the client defaults (it needs a login)
- updated danbooru file page parsers to get 'rating' tag
- added gelbooru 0.1.11 parser for future application
- fixed an issue that was stopping advanced content updates from fully copying all the desired mappings in the transaction
- added a semi-hacky checkbox to 'options->files and trash' that will delay all new file/thumb requests for 15s after the computer resumes from sleep (useful if your files are on a NAS that takes a few seconds to reconnect on wake)
- wrote some more graceful fallback decoding handling code that attempts original assumed encoding and 'utf-8' if different and returns the one with the fewest ' ' replacement characters
- the network engine and the ffmpeg info parsing now use this new 'safe' decoding, so even if a site has borked bytes or the video file has unexpected Shift-JIS title metadata, it'll still go through, albeit with some question marks
- moved some more old daemons to the new job scheduler, deleted some old daemon code
- improved some daemon job wake and shutdown code
- wrote a proper upnp manager object and improved all-around reliability of the auto upnp-service-mapping code
- simplified the upnp check code so it now only ever checks/does anything if the respective services actually want upnp mappings. surplus mappings are now wiped immediately on service update
- fixed upnp mapping fetching to cope with ipv6 results
- improved some memory clearing code to deal with some semi-stubborn objects
- improved some 'iterate through this giant list of single numbers from the db without using a lot of memory' code and applied it to the autocomplete cache regeneration routine
- improved menubar stability, both in finding menus and swapping them out
- if a serialised json object fails to load from the db, this is now caught, the bad object deleted and written to a new file in the db dir, and all logging info captured along with an explanatory popup thrown on screen. so, if a subscription fails to load, it will now be extracted so that a subsequent subscription edit/run will work with the remaining good objects. in the case of backed-up objects (gui sessions atm), reattempting the load should restore the next most recent backup
- fixed an issue with login script validation when the given credentials have surplus ( key, value ) pairs to the script's credential definitions
- fixed two login invalid cookie error handling bugs
- maybe made some dupe filter searching more stable
- fixed a py2 datatype issue that made the client unbootable when updating the client from <296
- the client now pauses to nag and moan about backups if you try to update more than 15 versions in one go
- slightly sped up discord bugfix file drag and drops and expanded file limit up to 25 files/200MB
- added experimental secret discord bugfix dnd mode checkbox
- improved how html parsing deals with some unexpected bad tag data
- turned on primitive high-dpi support for OS X. let me know if it fixes any blurry issues on retina displays
- wrote a new 'ui test' under the debug->gui menu to help catch common-action bugs that slipped through weekly work
- improved how the test code does some wx/ui stuff, but also broke some more and ran out of time to clean it up–this is an ongoing project
- improved how some text import line splitting works
- misc fixes
next week
I pushed it too hard this week, and while I am overall happy with the work, I am going to return to normal schedule so I don't burn myself out. I'd like to keep pushing on the Client API, probably getting add_tags done, and otherwise just flesh out the new test code and do some small QoL stuff.
Version 339
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v339/Hydrus.Network.339.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v339.tar.gz
I had a good week. I fixed some bugs and finished one part of the Client API.
client api
(This is only for advanced users for now.)
The Client API is now ready for some experimentation! Although most of this work has gone much faster than I expected, the last step of actually writing the server commands along with some half-decent test code and help documentation is proving a little slower. This week rolls out 'set up API access' and 'add urls' functionality. In the coming weeks, I will roll out 'add file', 'add tags', and 'search files' commands.
The basic (under construction) outline is here:
https://hydrusnetwork.github.io/hydrus/help/client_api.html
If you are interested in writing a script to talk to the Client API, please have a play with the new service under manage services and review services and see if you can get some fun going. I am interested to hear your feedback and to help if you run into trouble.
misc
I messed up several things last week, thank you for the quick reports. The bugs with ratings service sorting, the manage import folder dialog, and setting a UPnP port on the local booru/client api are now all fixed.
I also added a button to the ratings' review services panel that allows you to delete ratings en masse based on several file categories. If you have wanted to clear all the ratings for your deleted files, check it out!
full list
- client api:
- wrote some ui to handle client api permissions adding and editing
- wrote a 'catch a permissions request' mini-dialog for external api permissions adding
- wrote api calls:
- GET
- /api_version
- /request_new_permissions
- /verify_access_key
- /add_urls/get_url_info
- POST
- /add_urls/add_url
- and made a new 'client api' help page to describe in detail what these do
- wrote fairly comprehensive unit tests for the new client api
- refactored a bunch of 'hydrus network' specific stuff away from general server code that the client api now uses
- neatened up 401 vs 403 error handling across the program, and replaced some clientside error handling that was inelegantly borrowing these network errors
- deleted very old prototype file/thumbnail client server fetch code, which was no longer in use
- .
- misc and bug fixes:
- added a 'clear ratings' button to the ratings service 'review services' panels. it can clear out ratings for deleted, non-local, or _all_ files
- the '3 downloaders are working, is it ok to close the client?' message is now folded into the 'confirm client exit (auto-yes in 15s)' dialog. this merged dialog will still appear for users who have the regular confirm client exit dialog turned off (and still auto-yeses in 15s)
- the file url downloader now reports 'downloading file' and 'importing url' text status separately
- fixed a typo bug from last week that was breaking asc/desc ratings service sorting
- fixed a typo bug from last week that was stopping manage import folders from opening
- fixed a typo bug from last week that was breaking setting upnp port on the local booru/client api service management panel
- the advanced file reparse-and-re-thumbnailing routine now correctly moves a file to its new extension if its mime changes (e.g. png->apng, or webm/mkv distinctions)
- the client file manager now silently detects and auto-repairs instances of missing files where the file actually does exist, just with the wrong extension
- fixed a url parsing issue that was normalise-mangling url classes with no path but some query parameters
- the network engine now uses utf-8 decoding when no specific encoding is set (previously ISO-8859-1)
- fixed an ffmpeg video parsing bug when the video included Shift-JIS metadata. it should work for other unusual encodings as well
- maybe cleaned up some menubar management code
- the filename tagging dialog now uses a notebook for service choice, like the manage tags dialog, rather than the janky old listbook
- fixed a py2-to-3 issue with the admin-only 'is server currently busy' check while a backup is running
- improved some dialog button event handling. it may completely fix the 'trytoendmodal' issue some users run into
- improved some JSON db serialisation error reporting code, trying to pin down an issue several users have seen with session save
- improved thread-safety of serialisable objects as they serialise
- misc improvements and cleanup
next week
More Client API stuff. I did 'add urls' this week because it was the simplest. I think I can do 'add file' and 'add tags', and we'll see how much of 'search files' I can do as well.
I had meant to try out some high-resolution support for the OS X release this week, but it kept getting put off. I realised I had missed it again just now and tried to slip it in, but my first attempt did not go perfectly. Rather than try to rush it together in the last moments of the week, I have decided to instead give it another go for 340. Sorry for the delay!
Release Tomorrow!
I had a great if busy week. I fixed the stupid page of pages bug (and a whole bunch more), fleshed out the new client api, added some small new options, improved stability, and added several new downloaders to the client defaults.
The release should be as normal tomorrow, maybe a little late.
Bugs Thread
Gonna start another thread because >>173 is becoming too big for most people.
In the tag manager, whenever multiple tags are selected for multiple files with different tags, if you press the del key, the program will still ask you if you want to delete them all or put them all in the selected files. This does not make sense, if I'm pressing the del key it's obvious that I want them gone.
Version 338
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v338/Hydrus.Network.338.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v338.tar.gz
The recent v335 was an important update. If you are updating from v334 or earlier, please go read that release post for your special one-time update instructions and update to v335 before you try this week. Once you have updated to v335, the process for updating to anything newer is back to normal.
I had a great week. There's some bug fixes and neat new tag stuff.
tags
After talking to some users this week, I decided to knuckle down and improve autocomplete tag input. This has been in the works for a long time, but I have had trouble just putting the time in.
So, the autocompletes for both file searching (on pages) and tag management (on manage tags dialog) now work asynchronously! They search in the background, letting you continue typing smoothly without hanging you up. Large and laggy searches are cancelled mid-job as needed. Hitting enter before results come in is much smoother and more reliable, so if you just want to add simple tags like 'green eyes' by typing them in and hitting enter, it is much easier and less janky.
I am very happy with this change. Improving the core experience is something I want to do more. That said, I did this in a slightly rough way (I am wrestling with some terrible ancient code here), so there may be some bugs for certain workflows. If you discover you are getting false positives when typing fast or any other strange behaviour, let me know. This new system may want some different fetch timing settings, as well. All feedback welcome, basically.
Also, any 'selection tags' taglist on a regular page now lets you right-click->select all files with (selected tags)! If you select multiple tags for your right-click, you can either select the files with all the tags (AND), or those with any of the tags (OR). This is pretty neat, so give it a go!
misc
Pages now have a 'sort by approx bitrate' in their sort dropdown. This does a very basic (filesize / duration) calculation (putting durationless files on the smaller end) and lets you prioritise bloated gifs and other short, high-filesize (i.e. easy and useful to filter) vids.
Gelbooru changed up how some of their URLs work, so they suddenly started giving 403s for the hydrus default downloader. This release folds in a new parser that works again and a couple of new downloaders for gelbooru user favourites and pools.
I fixed an issue that meant preview view duration was ofter being counted in the new file viewing statistics system even while previews were in hidden pages.
full list
- after talking with some users, put a bit of time into tag autocomplete wolkflow. both read (search pages) and write (manage tags) autocomplete inputs now operate ~asynchronously~, with the tag fetch working on a separate thread. tag jobs can also now cancel at certain checkpoints in the tag search process if overwritten by a new request. therefore, a variety of tag lookup scenarios _should_ be less painful. this change was executed in a semi-inelegant way, so please report if you encounter bugs from fast typing etc…
- I also improved some of the "I hit enter before results were in" code as a result of this. not sure I have it totally nailed, so please give feedback on errors here
- wildcard search tags now have an explicit '(wildcard search)' after their label
- taglists that have an attached page of media (basically the 'selection tags' box and the 'active search predicates' up top) now have a right-click menu entry to 'select files with (all these tags)' and, if more than one tag is selected, 'select files with (any of these tags)'! This is pretty neat in action, so give it a go!
- added a 'sort by approx bitrate' file sorting option that does a basic filesize/duration so you can filter out dense gifs and other short-but-big vids a bit easier. anything without a duration is shoved to the 'smallest' side
- some of the file sort options now default to their respective 'biggest' first, see how you like it
- folded in updated gelbooru file page parser (fixing the 403 errors that just appeared) and added searches for gelbooru user favourites and pools
- if a gallery or watchable url ends up in the file processing queue (and hence fails), it now says the believed-to-be url class name in the error, which should help some false positive url class matching debugging
- fixed a focused-file selection issue that meant preview viewtime was frequently counting in file viewing statistics even while a page was not currently in view
- fixed the local booru review service panel, which wasn't fully deleting shares when the button was clicked
- wrote a service object and basic server skeleton for the client api (basically refactoring the existing local booru code). client api now appears in manage and review services and can boot and present the normal hydrus browser welcome page at '/'
- the client api and the local booru now have an 'allow non-local connections' checkbox! defaulting to off and on respectively
- updated a wx-thread call function to more safely and universally deal with instances where the responsible window died before the call could be made
- removed some old experimental crypto code that isn't used any more–pycryptodome is no longer needed to run the program from source
- improved some misc client service code
- deleted some old unused code
- misc cleanup
- updated to ffmpeg 4.1 on windows release
- updated to sqlite 3.26 on windows release
- updated to wxpython 4.0.4 on all platforms
next week
The Client API continued to go better than expected this week. If you explore manage/review services, you'll see the stubs of the new 'client api' service and can even boot a skeleton server. I'll work on a bit more ui and access code this week, and with a bit of luck, I'll actually have it doing something fun for v339.
中文帖子(Chinese Thread)
你們有甚麼問題也可以在這裡問
https://hydrusnetwork.github.io/hydrus/ (主頁)
https://github.com/hydrusnetwork/hydrus (開源軟件)
https://t.me/hydrus_network (Telegram即時通訊)
Version 337
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v337/Hydrus.Network.337.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v337.tar.gz
The recent v335 was an important update. If you are updating from v334 or earlier, please go read that release post for your special one-time update instructions and update to v335 before you try this week. Once you have updated to v335, the process for updating to anything newer is back to normal.
I had a good week. A bunch more bugs are fixed, and I added some little things as well.
all misc this week
I fixed a variety of bugs this week. A unicode log printing bug still hanging over from the py3 update, some unusual webms (mostly ones that tend to be a single frame with music), and some network download file size checking.
I also believe I have completely fixed the shutdown memory explosion some users were seeing. I managed to recreate the bug and figured out some bad shutdown scheduling that was causing an event processing routine to go crazy. I cleaned a bunch of other shutdown code as well, including a couple instances of subscription processing not stopping promptly on a client shutdown. Overall shutdown (particularly the last moments just before the splash screen closes) should be a bit smoother for all users.
I also added some little new things. The duplicate system now merges/copies pending tags as well as 'current' tags (and the advanced service-wide content update routine does as well). The 'lain' image on the downloader easy-import page is now clickable to launch a file selection dialog. And the manual export files dialog has a new experimental checkbox for advanced users that tries out exporting symlinks (which requires running as admin mode in Windows, so I don't really recommend it for that). If you try this symlink export, let me know how it goes!
full list
- fixed another couple of unicode encoding problems with the logging and profiling code
- the logger now sticks a unicode BOM at the top of new log files to help text readers guess the utf-8 encoding
- fixed musical webm import when the video stream has no stated duration but the audio stream does and typically stretches out a 'single frame' video
- fixed some 'max size' download file size testing
- 'waiting on bandwidth' statements on network job controls now show the specific network context (like 'web domain: somesite.com') they are waiting on (you might need to hover over to see the tooltip for this)
- the downloader easy-importer lain image is now clickable to launch a file selection dialog
- if you are in advanced mode, the manual file export dialog now lets you export symlinks with a new checkbox. this is experimental, so if you are interested, give it a brief test and let me know how it works for you
- duplicate content merging now applies to pending as well as current tags
- the duplicate filter now counts pending tags when saying which file has more tags
- advanced content updates now _copy_ both current and pending tags. the other actions now state what they do more clearly
- stopped printing long server error text on 304 and 404, where actual response content is uninteresting
- removed wx import that accidentally came into server space due to 2to3 check
- improved the path-fixing code that helps environment construction when launching external programs from non-windows frozen builds
- fixed a critical pubsub processing bug that kicked in at a certain stage of client shutdown. this should stop the post-shutdown-processing memory explosion certain users were seeing and should stop any ui jank in the last 0.2s of the program for everyone else
- improved some other shutdown memory cleanup that was sometimes leading to double-log-printing of exit statements
- did a full pass over the daemon scheduling code. it now reacts more responsively to various shut down situations
- reduced db disk cache aggression significantly and added more memory maintenance to the cache population process
- fixed an issue where subscriptions were not promptly responding to shutdown events
- fixed an issue where some delayed network jobs (e.g. while all network traffic is paused) were also not responding to shutdown events
- added a 'pubsub report mode' debug mode for simpler pubsub review
- the db is now less redundantly spammy on certain behind the scenes update notifications
- wrote a first version of the client api manager and permissions handling objects
- misc fixes, cleaned some shutdown code
next week
I started the Client API for real this week. The first 'manager' component is basically done, and it wasn't as complicated as I feared. I'll get started on some ui code for service review and authorisation management this coming week and otherwise just do some more small work to keep catching up on things.
Release Tomorrow!
I had a great week. As well as fixing a bunch of bugs, I radically improved how tag search works and feels and I added a neat way to select files by their tags. I moved the client api forward more than expected, but it is not ready for use yet.
The release should be as normal tomorrow.
increasing the public tag repo bandwith
Hey i just installed the hydrus client. It's perfect to organize my memes, props to the dev. My problem is that the bandwith volume is too low. I added the public repo mentioned on the github.io site. After 1932 tags downloaded my client bandwith was exceeded. Maybe the bandwith volume so anyone, whos adding the public repo can download anything in on download and not waiting 9 hours to get more volume.
Release Tomorrow!
I had a good week. I fixed several more bugs, including some more unicode issues and shutdown scheduling, and the big shutdown memory explosion some users were seeing, and added a couple of new small features. The client api work started well but is not ready for use yet.
The release should be as normal tomorrow.
Version 336
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v336/Hydrus.Network.336.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v336.tar.gz
Last week, v335, was an important update. If you are updating from v334 or earlier, please go read that release post for your special one-time update instructions and update to v335 before you try this week. Once you have updated to v335, updating to anything newer is back to normal.
I had a good week. A bunch of py3-related bugs are fixed, and I improved some heavy-load throttling.
py3 fixes
Overall, last week's update went well. CPU and memory use and overall jank is down for all users, and afaik there weren't any permanent disasters in the update. This was a big relief for me.
That said, as expected, several unusual bugs related to py3's new unicode handling slipped through. Thank you for the reports. A common one was related to windows logging, another was in pixiv searching with kanji tags, another with videos with unicode metadata. I believe I have cleaned most if not all of it up. There may be a couple more out there, so please let me know if you get anything new. Otherwise, please resume any broken subs or retry any broken webm imports.
I also fixed the numerical tag ui, which was coercing all clicks to to zero or max stars, and which I messed up by accident!
better heavy workflow
The client now has a strict thread-level throttle on the number of importers that can work at once. It seperates the different sub-parts into their own max limits, but the overall total limit is about 20-40. This is in an effort to stop ui hang after accidental heavy import situations such as pasting 100+ queries into a gallery downloader. This is a 1.0 throttle and doesn't have any ui to report 'waiting on a slot' or anything. I don't expect 99% of users will even notice it, but if it affects you seriously, please let me know.
In a related note, for advanced users, gallery downloaders now have the ability to 'bundle' mass-pastes into one downloader, queueing up all the different gallery queries sequentially in the same object (rather than in many in parallel, which is what previously cluttered the ui and boshed the CPU). If you sometimes paste a bunch of single-file md5-queries at once, please check this new check menu item under the gallery downloader's cog menu.
full list
- fixed an issue where the numerical rating control was coercing all clicks to either the minimum or maximum allowable rating (e.g. 3/5 stars wasn't working)
- fixed some text file and process i/o, which was handling some unicode decoding/encoding incorrectly. it now mandates utf-8 in all cases
- fixed a referral url encoding problem that was stopping pixiv from downloading when the gallery page url had kana/kanji characters (from a search term)
- fixed a str vs bytes issue when loading the filename tagging panel
- fixed the delete button on the filename tagging quick namespaces panel (the edit and delete buttons are also now 'live' and will disable when nothing is selected)
- improved some json dump deserialisation code
- fixed a data-sorting issue that would appear with certain parsers in the edit parsers panel
- improved video metadata parsing, fixing an issue when the video has a 'title' row containing inconvenient data
- fixed some hex character processing for system preds
- added an advanced check item to the gallery downloader cog icon menu that will 'bundle' multiple query-pastes to the same single gallery downloader (this is helpful if you are pasting a whole bunch of md5 queries in one go and would rather one downloader work through them sequentially than 50+ separate ones blat your CPU simultaneously)
- the different kinds of importer worker threads now have several limits on the max number that can be working at once, to stop accidental ui overload when a hundred or more are in memory and all want to work at once (like after a big paste event or resuming after computer sleep). during periods of heavy import activity, the importers will now naturally space themselves out to smooth out the spike. the limits are hardcoded for now, let me know if it noticeably bottlenecks your situation
- made some menubar update code a bit less complicated and reduced how often it'll spam during heavy update
- the 'what to do?' buttons that appear in manage tags sometimes on a tag action got a simplification pass and are now on the new dialog system
- simplified my new dialog code significantly, clearing out redundant code and classes and pushing all okable/cancellable/vetoable closing checks through one single method
- wrote some new help.txt in the db dir about hanging startups
next week
I still have a bunch of small stuff going on, but I'd like to start on the Client API, the next 'big thing', as well. I expect this will involve several weeks of ui and clientside server prep before anything exciting happens.
Experimental decentralized booru software
Hello Hydrus Network community.
I am a programmer familiar with booru development. I have written my own private booru software that scrapes public websites for images much like Hyrdrus Network, it currently has 10 million files, but this isn't what I am here to talk about.
I am very interested in Hydrus Network's goals of decentralized zero-authority sharing and collaborative organization of images as developments towards these goals benefits my own booru.
I have recently created a proof of concept decentralized booru called "Decboo", it uses IPFS for image distribution and the Ethereum blockchain for tag distribution.
Decboo is not the booru I spoke of earlier, it is an extremely basic program made only to demonstrate blockchain-based tag distribution, functionality that I hope the Hydrus Network developer(s) will integrate into the Hydrus Network software.
A zip file containing Decboo's software can be downloaded here: https://ipfs.io/ipfs/QmZvYUrXFwpqCognaLm6ecSixQRuXbeXScrZKA71VRUsgE
Version 335
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v335/Hydrus.Network.335.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v335.tar.gz
When I first made this release, Github’s file upload was not working right, and I used Mediafire instead. Github is now working and I have updated the links above.
I had a great four weeks updating hydrus to python 3. The update went well, and the releases today are ready for all users, but there are special update instructions just for this week.
python 3
The client and server now run completely and exclusively on python 3, updating from python 2. The new version has a variety of benefits, mostly in better unicode vs. data handling, but for hydrus it also runs a little faster, uses less idle CPU and significantly less memory, and has a little less ui-jank.
I am pleased with the update. None of it was extremely difficult, but there was a lot to do, a few headaches, and some new stuff to learn. I am glad I took the four weeks. I also appreciate the users who tested the preview releases in the past couple of weeks.
I have squashed a ton of little bugs, and everything normal like file downloading and browsing seems to work completely fine, but there are likely a couple of small issues still out there. If a dialog breaks for you and you get some popups about some datatype being mishandled, please send it to me and I'll get it fixed up for v336!
some notes
Unfortunately, for technically difficult reasons, I could not compile 'debug' versions of the executables, so these are gone this week. I will revisit this, but the original debug builds were a bit hacky and may no longer be practically possible in py3.
Also, I am not certain if the database menu's 'set a password' will have kept correct password data for unusual keyboards, so if you use this function, the client will, just for this v335, forgive incorrect passwords! If this happens to you, the client will give you a popup with more information and tell you how to try to fix it. I would appreciate feedback here, if you encounter it.
Due to a natural library update unrelated to py3, your hydrus sessions will likely be wiped (this also happened to some running-from-source users a little while ago), which means Hydrus will have to re-login to any sites you have logins set up for. If you have special cookies you need to save or re-import from your browser, get set up before you update!
Now, because py2 and py3 are incompatible, the new version cannot be run in a folder that still has old .dll files hanging around. Please follow the following to update:
update instructions for windows installer
Just for this week, I have added a routine to the installer to delete the old files (but obviously saving your db directory where your database and files are stored!), so you shouldn't have to do anything. I nonetheless recommend you still make a backup before you update. Backups are great, and if you don't make one yet, this week is a great time to start.
If you are a particularly long-time user and the installer fails to clear everything out, you may need to delete the old files yourself, like the extract users will have to:
update instructions for windows and linux extract
You will have to perform a clean install, which means deleting everything in your install folder except the db directory before extracting as normal. This is simple, but do not get it wrong. Do not delete your db directory–this is where your database and files are stored.
As always, if you have a recent backup, you don't have to worry about any possible accident, so make sure you have one.
update instructions for os x
Due to technical limitations, the OS X release is now App only. Furthermore, this App release will no longer store the db inside itself! The default location for your db is now ~/Library/Hydrus (i.e. /Users/[you]/Library/Hydrus). This also means that the future update process will be as simple as replacing the existing Hydrus Network App in Applications, just one action. I apologise that this important change has taken so long to come out, but we got there in the end.
If you are updating this week, you will need to make the Hydrus folder under your Library yourself and move your existing db there so the new Hydrus can pick up where you left off. If you use the tar.gz, you'll be moving the contents of your install_dir/db, and if you use the App, you'll want to right-click->Show Package Contents on your old py2 App and navigate to Hydrus Network/Contents/MacOS/db. You want the contents of db, not the db folder itself, so the path to your new client.db file should be ~/Library/Hydrus/client.db, for instance.
If you cannot see your Library folder, check this: https://www.macworld.com/article/2057221/how-to-view-the-library-folder-in-mavericks.html
If you have trouble with this, please let me know and we'll figure it out together.
update instructions for running from source
You'll need to make a new py3 venv and make new shortcuts. I now use pycryptodome instead of pycrypto and dropped some libraries, so I recommend you go to the 'running from source' help page again and just paste the new pip line(s) to get set up.
I don't think 3.4 will work, but 3.5, 3.6, and 3.7 all seem ok. Obviously contact me if you run into trouble. I'm also interested in success stories!
full list
- important:
- hydrus now runs completely and exclusively on python 3!
- for users who are updating, the client has special install instructions for just this week:
- if you are a windows or linux user who extracts to install, you will have to delete your old install's files (but keep your db folder!!!) before installing/extracting the new version so there are no 2/3 dll/so conflicts (don't delete your db folder!)
- if you use the windows installer to install, this v335 installer will do the clean install for you! there is absolutely no way this could go wrong, so no need to make a backup beforehand :^)
- if you are an os x user, I am now only releasing the client in the app. furthermore, the default app db location is now ~/Library/Hydrus (i.e. /Users/[you]/Library/Hydrus). you will have to move your existing db to this location to update, and thereafter you'll just be replacing the app in Applications!
- if you try to boot a non-clean mixed 2/3 install, the client will try to recognise that and give an error and dump out
- please check the release post for more detailed instructions here
- .
- semi-important:
- the db password feature may be one-time broken for unusual keyboard languages, so failures this version will be forgiven with an appropriate error message explaining the situation. feedback from чики брики lads appreciated
- I may have fixed the issue some linux/os x users were having launching external programs, including OS ffmpeg (it was a child process environment issue related to pyinstaller)
- although I did most of my devving here on py 3.6, the client seems to run ok on 3.5. I doubt 3.4 will do it, if you mean to run from source
- I moved from the old pycrypto to the new pycryptodome, so users who run from source will want to get this. I also dropped some libraries
- .
- misc bug fixes:
- fixed the 'load one of the default options' button on manage tag import options when a set of default options is orphaned by a deleted url class
- removed some popup flicker related to long error messages
- fixed some parsing testing ui error handling
- cleared up some bad text ctrl event handling that could sometimes cause a recursive loop
- listctrls should now sort text that includes numbers in the human-friendly 2 < 10 fashion
- cleaned up some bad external process calling code and improved how child process environment is set up
- finally figured out the basic problem of a long-time nested dialog event handling error that could sometimes freeze the ui. I may have fixed it in one case and will keep working on this
- .
- boring details:
- ran 2to3 to auto-convert what could be done
- updated environment to python 3
- went over a whole ton of unicode encoding/decoding manually to update it to python 3
- removed all the old tobytestring/tounicode calls in favour of new python 3 handling
- fixed all the file io to do bytes/str as appropriate
- corrected a bunch of / vs // int/float stuff
- fixed up twisted, which has some str/bytes stuff going on
- fixed all the listctrls to deal with column sorting None values amongst ints/strs
- fixed png export/import, which had some fun bytes/bytearray/int/str issues
- updated the swf header parsing code to py3 (more str/bytes stuff)
- misc float/int fixes
- fixed up some http.cookies handling, which has changed in py3
- improved some ancient loopback connection code that was only really checking to see if ports were in use
- cleaned up a bunch of now-invalid parameter tuples that 2to3 helpfully marked
- numerous misc other refactoring and so on
- updated the new network engine to now decode non-utf-8 responses correctly based on actual response header
- removed some old py2 manual http multipart code
- removed the old py2 'matroska or webm' parsing py, replacing it with some direct ffmpeg format inspection
- replaced all % formatting with the new .format system. I will slowly move to this rather than the current endless concatenation mess
- deleted some more misc old code
- tightened up some spammy network error reporting
- converted all /r/n to /n in my environment project, ha ha ha
- the ui seems to better support rarer unicode characters like
- updated some of the install/update/backup help for all this, and some misc other stuff as well
- fixed misc bugs
next week
A lot of small stuff piled up over the holiday. I will spend a week or two trying to catch up and also planning out the client API, which will be my first big job of the year.
I hope you had a good Christmas and New Year. Mine were great, and I am looking forward to 2019. Let's keep pushing and see if we can do some fun stuff. Thank you for your support!
Release Tomorrow!
I had a good week. I have fixed several bugs in the new py3 release (numerical ratings, some file and network unicode i/o, and some list sorting), cleaned and improved some long-time ui jank-code, and wrote in some limiters and a new workflow to smooth out mass-multiple import jobs.
The release should be as normal tomorrow.
Important Release Tomorrow!
I had a great four weeks updating the software to use python 3, the new version of the language I write hydrus in. Overall, it went very well. The new client does not look different, but its guts are newer and neater. Some things run faster, and some long-time py2 bugs are gone.
The release will come at the normal time tomorrow, but it will have special one-time update instructions for all existing users. It is important you read the release post.
Version 334
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v334/Hydrus.Network.334.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v334.tar.gz
I had a good week. I mostly fixed bugs in order to make a clean 'final' python 2 release.
all misc this week
I wrote a system: predicate for the new file viewing statistics. It works great!
Also, you can suspend file viewing tracking and clear all records entirely under the new database->file viewing stats menu.
I shuffled some options around in the options dialog and created a new 'gui pages' entry.
The thread watcher should now correctly recognise url and url#12345 are dupes in all cases.
I wrote some login info to the downloader help and completely rewrote the subscription help:
https://hydrusnetwork.github.io/hydrus/help/getting_started_downloading.html
https://hydrusnetwork.github.io/hydrus/help/getting_started_subscriptions.html
full list
- wrote a system:file viewing stats to comprehensively search the new viewing stats–it _should_ also be synced with the exact current values
- but for system:everything, inbox, and archive, which remain where they were, system predicates are now sorted alphabetically!
- added a _database->file viewing stats_ menu that lets you suspend file view tracking and clear all records permanently
- mr. bones now welcomes all users under the help menu
- fixed mr. bones's confusion at those who have yet to board the ride
- also mr. bones now makes sure to get the latest file viewing stats
- moved confirm trash/archive from _options->gui_ to _options->files and trash_
- moved a bunch of 'pages' related stuff from _options->gui_ to the new _options->gui pages_
- added an option to options->gui pages to change the number of session rolling backups
- subscription popups now provide an x/y query progress string in their popup text
- the edit subs/sub panels are now a bit shorter by default and the edit sub has its own frame position data, under 'edit_subscription_dialog', and remembers its last size and position by default
- fixed an issue where some dupe watcher urls (like url and url#12345) were not being correctly merged on a mouse drag and drop watcher-import
- the client will now print up to 512KB of server error info to the log (previously 4KB)
- removed the youtube download prototype–if it returns, we'll do a proper youtube-dl solution. as a result, pafy is no longer needed to run the client
- network report mode now shows more network error information
- gave the 'getting started with subscriptions' help page a complete pass. it now reflects the new system and has up-to-date advice based on my new experience
- wrote a 'logins' section to the bottom of the 'getting started with downloading' help page
- misc fixes
the next four weeks or so
As planned, I will now concentrate on updating the software to python 3. My rough guess has budgeted four weeks for this, but I can't say with great confidence. I will stop putting out normal releases, but I will try to keep up with threads and messages, go to the discord on Saturdays, and make progress updates on Tuesday evenings like my 'release tomorrow' posts. I hope we'll be working on new code and back to normal schedule in the new year.
I had a great 2018, and I am now looking forward to 2019. I am still in a position to work on hydrus, and I still enjoy it. I really appreciate your support–thank you!
Python 3 Update Progress 3
Ｈａｐｐｙ Ｎｅｗ Ｙｅａｒ！
I had a great week. The client works well, all big tests passed, and I now have functioning builds for Windows and Linux. I will spend the last week figuring out an OS X environment and doing last tests and cleaning.
I feel great about the schedule. v335 is completely on track for a Jan 9th release.
Python 3 Update Progress 1
I had a good week. I was able to get the client booting and closing cleanly on a new Python 3 environment on Windows. I have changed hundreds of tiny things, and there are still still a bunch more to go, and then I have to hammer out the 'freezing' process to make the executable release, and then figure out the same environment and build for the two other platforms.
I feel good about the schedule. It involves a lot of bashing my head against the keyboard, but I am moving forward.
Database Encrypted Sync
I haven't seen this discussed so I thought I'd make a thread on it.
We are all aware that Hydrus runs fine inside a Truecrypt container, but having one huge file can be a problem.
For instance if you backup your 500GB database inside a truecrypt container and then download even just 1 image you basically need to backup the whole 500GB all over again if you run an automated backup to an external HDD or a NAS. Another problem with Truecrypt/Veracrypt volumes is you have to set the container size at the time of making it, leaving it too big or too small all too easily.
Well I've been experimenting with different crypto programs that feature encrypting directories while keeping files separate. The filenames and contents are unreadable, but they are still their own files so syncing and backup programs know exactly what to copy and what is the same.
The problem is that Hydrus doesn't want to run at all within these encrypted directories, but luckily we can have our installed client separate to the db folder.
>install an encryption program (I used cppcryptfs but there are a bunch for various platforms)
>create a new encrypted folder on any drive
>mount it as a Drive letter (you will need to use the same Drive letter each time)
>Move your Hydrus db folder to that mounted drive (or you can make a new one by ignoring this step)
>create a shortcut to your client.exe adding -d="path to db folder on virtual drive" (e.g "C:\Hydrus Network\client.exe" -d="Z:\db")
>use shortcut
it should find your database and start like normal, with any new files being encrypted as their own files that can be synced individually.
I haven't tried it but theoretically you could even mount a cloud provider as a virtual drive and store your entire database that way, probably stupid but I might try it for fun.
Python 3 Update Progress 2
𝕸𝖊𝖗𝖗𝖞 𝕮𝖍𝖗𝖎𝖘𝖙𝖒𝖆𝖘!
I had another good week. The client works about 95%, and I can get it into a proper executable release that runs fine. I now need to iron out the last issues and sort out Linux and OS X environments.
I feel great about the schedule. I am still aiming for a Jan 9th release for v335.
hi i've had hydrus a while back and am getting back into it now
i'll have some questions about migrating db's from old to new
but first i wanna ask about what storage hardware is best for hydrus?
because i think i broke me old hdd by using hdyrus (seagate's being prone to crashes)
the tags that i had through the public tag repository should re-appear when i import files, right?
Version 333
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v333/Hydrus.Network.333.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v333.tar.gz
🎉🎉 Merry v333! 🎉🎉
I was slightly short on time, but I still had a great week. There's some fixes and speedup and a bit of fun.
file viewing statistics
The client now records how often (and for how long) each file has been viewed in the preview window and main media viewer! You will see how many times you have viewed a file in its normal thumbnail and media right-click menus. You can customise how this info displays in the menu–including hiding it completely–under options->media.
You can also sort by total views or viewtime under the normal file sort dropdown (which is now itself alphabetical and has improved labels).
This is only a first version, so it isn't perfect. The total viewtime isn't updated until you finish looking at a file, for instance, and counts will sometimes be temporarily desynchronised. I expect to revisit it, maybe adding the stats to the main media viewer's top-right window, and making the current viewtime live, so the seconds count up correctly as you look at something. I'll add a system predicate for it as well, so you'll be able to search for things you've seen at least x times and so on. I will add an option to turn these stats off completely next week.
What do you get here? Do you go back to the same twenty files over and over, or are things more evenly distributed? Once we have fleshed this data out, I could draw some distribution graphs or do some more innovative 'show me some infrequent stuff' searching.
another cache
After the good file and tag work of the past few weeks, I've written a higher-level cache that significantly streamlines how gui-level media is stored in the client. There is now only ever one copy of each media object, with all thumbnails and media viewers pointing at that same one. This reduces total memory usage and CPU in many situations, makes it possible to immediately show content changes after advanced updates like tag repository processing, and it speeds up certain searches as duplicate media objects do not have to be recreated from scratch at the db level.
I am quite pleased with this cache. I have been thinking about it for a long time. Please let me know if you have any problems with it (likely it would be some variation on "I changed a tag/rating/whatever, but the file doesn't show the change, even when I refresh the search").
tumblr madness
If you use tumblr for lewd purposes and missed the news this week, tumblr have gone nuts and decided to ban all nsfw content off their platform on Dec 17th. There's been some corporate drama related to the tumblr app, but no one knows what's really going on with this overbroad new decision–my assumption is Verizon haven't been able to find a workable business model, so they are seizing this chance to reduce liabilities rather than continue throwing money away. Maybe they'll try to find clean ads to run like 4chan are currently doing with 4channel, or maybe they expect to kill the whole thing softly over the next few years. And perhaps the outcry will convince them to reverse the decision, but don't plan on it.
So! If you had plans to download from some nsfw tumblr, get it going now. Go visit the creator's blog and see where they are migrating to so you can figure out new subscriptions.
If you have a big tumblr yourself that is about to get semi-nuked due to your reblogs, there may also be a clever one-time way to get higher quality versions of what you reblogged as well, as per here: >>10905
full list
- added a first version of file viewing statistics! the client db now keeps track of how many times a file is loaded in the preview and full media viewers, and for how long!
- you can see the media and preview stats on any single media right-click menu. there are multiple options for how this displays, including hiding it completely, under options->media
- viewing stats update as they happen! (although viewtime typically only updates on the end of viewing. I'll likely make this more live, especially if I end up showing this info in the media media viewer)
- you can now sort files by total media views/viewtime!
- mr. bones's wild ride continues, as well
- deleted the old 'file list' way of updating in-ui media objects in favour of a long-planned global media cache. there is now only ever one active copy of any particular media, and all data-level updates need only occur once on that single copy. this saves a bunch of CPU, memory, and overall hassle behind the scenes! various search results/lookups for media already loaded elsewhere now load super fast!
- tag siblings refresh is quicker and less memory heavy thanks to this as well
- furthermore, the complicated tag changes from tag repository processing and advanced content updates are now reflected immediately in the gui on the job's completion! (as long as you have fewer than 10k files open, ha ha) previously, these required a search refresh to show the results
- the file sort choice dropdown on all pages is now sorted alphabetically. it has always been a mess picking what you want from here, so let's see if this helps!
- tag and rating sort options are now listed as 'tag:' and 'rating:' respectively
- fixed some misc file sort choice code, which was failing to keep certain defaults in certain situations
- fixed the tag import options' new 'load from defaults' button to correctly load the tag blacklist
- the keyboard icon on the media viewer's top hover window now permits activation of current/default shortcut sets under submenus. it now also omits these entries if no custom shortcut sets exist
- cleaned up some of the hover_window-canvas interaction code
- fixed some long-time sperg-out buffer-drawing when changing position in a long video
- the database->backup actions are now hidden if the current db has non-default file/thumbnail locations. for now, in these cases, only a custom backup is appropriate
- fixed some ancient repository admin code that fetches summary account info given an account key
- the filename tagging dialog now has a much shorter listctrl by default, so should fit better on smaller monitors
- fixed the 'review session cookies' dialog's clear button, which was not deleting sessions after clear. it now also wraps the operation in a yes/no confirmation
next week
I am still planning to take about four weeks over Christmas for a big conversion to python 3. This break starts next week, December 12th. I will use this last week just to tidy up, fix any stupid bugs, and write some more help so there is a clean 'final' v334 python 2 build.
The poll for the 'next big thing' is finished, and prototyping a new client API won out. I will start this in the new year and hope to make it an 8-12 week job before starting up a new poll (or possibly going straight on to OR searching, which was a very close second).
Release Tomorrow!
I had a good week. I mostly fixed bugs and wrote some more help, trying to make a clean 'final' python 2 release.
The release should be as normal tomorrow. It will be the last release of the year, as I will be spending the next few weeks updating to python 3.
Version 332
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v332.tar.gz
I had a great week catching up on a whole bunch of small jobs.
rolling session backups
The program now automaticaly makes up to ten rolling backups (i.e. it keeps the ten newest saves) of your sessions! This includes the 'last session' backup used to restore the program on boot, so if you accidentally close something very important and can't get back to it, or if your session breaks for some reason, you can now rewind a bit further.
You can access the backups under pages->sessions->append session backup. They are listed by the time they were made.
The client also now saves an 'exit session' on a clean client shutdown. This is different to the 'last session', which is updated every few (default 5) minutes, and provides a longer-term backup.
This works for the entire in-client serialisation system, so if this works out, it may make sense to apply it to other objects such as the bandwidth tracking system, just to provide an easy additional layer of safety.
misc
There's a bunch of small stuff this week:
Gallery import pages now have a cog button beside the query input that lets you control if new imports start with files and/or search paused! It remembers its values through a session reload.
Tag import options that can be 'default' now have a button to specifically load a default, so if you want to do a specific download that has exactly the normal default options but one little thing different, you can just load it up and edit it rather than having to recreate the whole thing from scratch.
The new tag id cache is expanded, and I've written a similar cache for file ids. A whole bunch of database operations should be just a bit faster.
Export folders can now be told to delete the exported files from the client after they run. (So you can, for instance, tell your client to send all files with 'nsfw' to an import folder for another client.) If you decide to do this, it'll throw up some warnings just to make sure you are ok with it.
When running database->check->file integrity, the client can now optionally export a .txt file with all the missing files' known urls! This same list can be pasted right back into a new url downloader in the client to try to get them back!
full list
- the client serialisation system now supports multiple rolling backups!
- client sessions (like the 'last session' that typically loads on boot) are now automatically backed up to ten times in rolling backups! you can review and append the backups if you need to recover from _pages->sessions->append session backup_, where they are listed by their timestamp!
- when the client closes, an additional 'exit session' is now saved. this differs to 'last session', which is overwritten every x minutes, and is now available (especially with the new backup) for various error recovery situations
- gallery import pages now have a little cog icon to control if new import queues will start with files and/or galleries paused. these states persist through a session reload
- tag import options that are 'defaultable' now have a button to let you load in a specific default, so you can easily quickly edit in a one-time slight alteration of the default rather than having to create everything from scratch
- under options->downloading, you can now set the 'delay' times on gallery/watcher network error, subscription network error, and subscription other error, now defaulting to 90mins, 12hours, and 36 hours respectively
- attempting to launch a client with db version > software version now spawns a blocking messagebox on pre-launch informing the user of the risks and advising task manager force-kill of the process
- did a little cleanup on the new tag id database cache and merged some other, older semi-laggy tag-fetching code to use the same system
- wrote a similar 'file id' database cache for caching file hashes and also merged some old hash-fetching code into it. a variety of file operations are now significantly faster
- export folders can now delete files from the client after export. the edit panel will warn you on selecting this and oking the panel. it can't be set if the export type is 'synchronise'.
- fixed the edit export folders dialog's old buttons, which were semi-working due to some recent update work
- when checking for file integrity, you can now choose to export a .txt file listing all the missing files' known urls, so you can try to recover by feeding them all back into a new url downloader!
- the physical file deletion process is now a little simpler and deals with larger jobs in smaller batches, no longer hogging the file read lock the whole time. clearing a large trash should no longer hang other media loads
- adding ngugs in the 'export downloaders' dialog now attempts to add the respective gugs as well
- fixed an issue where the listctrl would accept certain kinds of duplicate data and hence confuse its indices
- fixed all the add buttons on the export downloader panel to exclude all items already in the list when figuring out what additional objects to add
- the 'scroll thumbs at this rate per tick' option now _rounds_ the pixel result, rather than always _floor_ing it
- wrote a new mass-selection database routine that should reduce memory footprint of autocomplete regeneration. if it works out, I will use it in some other places
- improved the errors when a network job that cannot wait fails on an invalid login, and added a separate error when it refers to a hydrus service
- fixed a small typo bug when trying to auto-add url classes and parsers
- updated an old deprecated checkboxlist call
- clarified the integrity section of 'help my db is broke.txt'.
- misc fixes
next week
Next week should be more small stuff like this. Maybe some fun/new for 333 if I can fit it in. I mostly want to tidy up for a 'python 2 final' 334 on the 12th Dec. On the 12th, I will break up for Christmas to convert the program to python 3.
The poll on the next big thing to work on
Due to some funny voting, I am considering the poll finished earlier than expected and will start work on an API prototype in the new year. (details >>10845) Thank you everyone for voting–I really appreciate the feedback.
Here is the poll if you would like to review what was overall popular and not:
https://www.poll-maker.com/poll2148452x73e94E02-60
This thread remains available for discussion of anything related to the poll.
Thanks everyone!
Running from source
I'm having a lot of trouble getting hydrus to run from source.
Could not import lz4.
I have lz4 installed.
Traceback (most recent call last):
File "H:\Hydrus Network\client.py", line 20, in <module>
from include import ClientController
File "H:\Hydrus Network\include\ClientController.py", line 14, in <module>
import ClientCaches
File "H:\Hydrus Network\include\ClientCaches.py", line 1, in <module>
import ClientDefaults
File "H:\Hydrus Network\include\ClientDefaults.py", line 2, in <module>
import ClientData
File "H:\Hydrus Network\include\ClientData.py", line 3, in <module>
import ClientDownloading
File "H:\Hydrus Network\include\ClientDownloading.py", line 12, in <module>
import pafy
File "C:\Python27\lib\site-packages\pafy\__init__.py", line 7, in <module>
from .pafy import new
File "C:\Python27\lib\site-packages\pafy\pafy.py", line 52, in <module>
"pafy: youtube-dl not found; you can use the internal backend by "
ImportError: pafy: youtube-dl not found; you can use the internal backend by setting the environmental variable PAFY_BACKEND to "internal". It is not enabled by default because it is not as well maintained as the youtube-dl backend.
I have youtube-dl installed as well. Setting the PAFY_BACKEND fixes this.
2018/12/01 21:22:43: Traceback (most recent call last):
File "H:\Hydrus Network\include\ClientController.py", line 1241, in THREADBootEverything
self.InitModel()
File "H:\Hydrus Network\include\ClientController.py", line 602, in InitModel
session_manager = self.Read( 'serialisable', HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_SESSION_MANAGER )
File "H:\Hydrus Network\include\HydrusController.py", line 491, in Read
return self._Read( action, *args, **kwargs )
File "H:\Hydrus Network\include\HydrusController.py", line 177, in _Read
result = self.db.Read( action, HC.HIGH_PRIORITY, *args, **kwargs )
File "H:\Hydrus Network\include\HydrusDB.py", line 861, in Read
return job.GetResult()
File "H:\Hydrus Network\include\HydrusData.py", line 1498, in GetResult
raise e
DBException: ImportError: No module named ordered_dict
Database Traceback (most recent call last):
File "H:\Hydrus Network\include\HydrusDB.py", line 527, in _ProcessJob
result = self._Read( action, *args, **kwargs )
File "H:\Hydrus Network\include\ClientDB.py", line 8656, in _Read
elif action == 'serialisable': result = self._GetJSONDump( *args, **kwargs )
File "H:\Hydrus Network\include\ClientDB.py", line 5479, in _GetJSONDump
return HydrusSerialisable.CreateFromSerialisableTuple( ( dump_type, version, serialisable_info ) )
File "H:\Hydrus Network\include\HydrusSerialisable.py", line 129, in CreateFromSerialisableTuple
obj.InitialiseFromSerialisableInfo( version, serialisable_info )
File "H:\Hydrus Network\include\HydrusSerialisable.py", line 206, in InitialiseFromSerialisableInfo
self._InitialiseFromSerialisableInfo( serialisable_info )
File "H:\Hydrus Network\include\ClientNetworkingSessions.py", line 76, in _InitialiseFromSerialisableInfo
session = cPickle.loads( str( pickled_session ) )
ImportError: No module named ordered_dict
I have no idea what to do here. Any help appreciated.
Thanks.
Version 331
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v331/Hydrus.Network.331.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v331.tar.gz
I had a good week. The login manager is done, I have added proxy support, and some tag stuff is faster!
login manager
The 'manage logins' dialog has some small usability improvements, and it also has a neat 'do login now' button to easily manually attempt logins.
The network->data->review session cookies panel now hides 'empty' sessions by default and allows for cookies.txt drag-and-drop import!
And login scripts can now be rolled into the 'easy import' downloader pngs! They get listed like all the other objects.
With these last jobs, I have finished my last big jobs for the login manager. I would still like to write some decent help, and I'll fold new login scripts into regular updates as users create and share them, but this phase's bulky work is all complete. Sites with captcha and other complicated login systems (like exhentai.org) are not supported in this first version (so you'll have to manage without, or use manual cookies.txt import from your browser), but I am overall really happy with how it has gone.
proxy support
A user reminded me last week to add proxy support back in. This was an important objective of the big network engine rewrite (I had previously removed my tentative support as the old engine became too convoluted to support), and adding it this week couldn't have been smoother–the 'requests' library for python is just great. If you have an http or socks4/socks5 proxy you would like to use, please go to options->connection and fill in the simple new options. It has some text to explain it (including how to add socks support if you are running from source and lack the right library).
This is client-wide for now, so if you set a proxy, all client requests will go through it. Let me know if this causes any problems. I may revisit this and have it work on a per-domain basis, like how bandwidth rules work.
other
Searches that include 'system:num_tags' are now much faster and have accurate tag counts even when multiple services share the same tag! This is still a pretty CPU-expensive predicate, so I recommend you reduce the search domain by including a tag or a simple system predicate like 'system:size', but it should nonetheless run a lot quicker. I wrote a new and unusual query to pull this off, so please let me know if it fails in some situations. If it works well, I will use it elsewhere.
I have also started an extremely basic tag cache for the client db. It makes autocomplete results and file search results appear significantly faster in many situations. Like the new system:num_tags query, I am going to play with it a bit on my IRL client, see if there is any 'I noticed the difference in a good/bad way' feedback, and likely do a similar cache for file data and iterate on them a bit.
There are better and more "show all these things' files" menu entries on the gallery import and thread watcher management panels' lists' right-click menus. You can now choose between showing the 'presented', 'new', and 'all' files, and they now show more quickly in the current page.
I updated last week's e-hentai.org login script to one that works better, and I added a 4channel.org thread url class to permit watching the soon-to-be-separated sfw 4chan boards.
full list
- added a 'do login now' button to the manage logins dialog. it only enables when the selected logins are active and not invalid and so on, and will ok the dialog and queue up some login attempts, which will make report popups as normal
- 'review session cookies' panels now support drag and drop cookies.txt import! cookies.txt importing will also handle errors a bit better and report total number of added cookies
- the 'review session cookies' panel now defaults to not showing sessions with zero cookies. a new checkbox controls this
- login scripts can now be rolled into easy import pngs! should work for export and import just like the other objects (although they won't be auto-added based on domain in export dialog)
- brushed up some of the 'change login script' code–particularly, it now puts login scripts that have matching domains first in the selection list, for easier selection
- after striking a reckless bargain with a daemon from the database-plane, system:num_tags now runs significantly faster and produces accurate tag counts even when searching over multiple tag services that have duplicate tags. if this works out, the immaterial beast promises greater gains for similar jobs with no possibility of anything going wrong
- prototyped a new tag cache in the db that affects (and should speed up) many tag fetching routines. let's see how it goes
- added complete, global proxy support for the new network engine! there are new options under options->connection (with some explanation text) to handle it. if pysocks is installed, socks4/5 proxies are also available!
- updated the e-hentai.org login script to the new one on the github. your existing mappings for e-hentai.org _should_ all be updated right. exhentai.org is likely too difficult to properly support in the current system
- the different panels where you enter system predicate information now all run on the new panel sizing system–if you have had problems with these, please let me know how they size now!
- added a '4channel thread' url class to support watchers for the new 4channel sfw domain. it works for now, but let's see if their api changes when the split actually happens
- the list right-click menu on gallery import and thread watcher panels now has three options to show combined importers' files–presented, new, and all. it also now shows the files (more smoothly) in the same page, clearing any existing highlight.
- misc ui improvements
- updated 'running from source' in help
next week
I have three more weeks until I break up for the holiday and start the big python 3 rewrite. With the login manager work done, I am now just going to catch up on small jobs. I'd like to focus on some code cleanup, particularly, and maybe untangle some hellish db siblings/parents code. If you missed the 'next big thing' poll last week, please check it out here: >>10654
you guys know that there are people who actually use hydrus for searching their files instead of just downloading every trash they can find? so since people exist who want to actually use hydrus, it's way more useful to have or-search instead of a shitty api that only retards would benefit from (that's not true but or-search is easier to implement and IT IS USEFUL.. especially if you use your own tagging style instead of the bs ptr tagging system which is just stupid imo) - end of discussion
Version 330
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v330/Hydrus.Network.330.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v330.tar.gz
I had a great week. There are some more login scripts and a bit of cleanup and speed-up.
The poll for what big thing I will work on next is up! Here is the thread: >>10654
login stuff
The new 'manage logins' dialog is easier to work with. It now shows when it thinks a login will expire, permits you to enter 'empty' credentials if you want to reset/clear a domain, and has a 'scrub invalid' button to reset a login that fails due to server error or similar.
After tweaking for the problem I discovered last week, I was able to write a login script for hentai foundry that uses username and pass. It should inherit the filter settings in your user profile, so you can now easily exclude the things you don't like! (the click-through login, which hydrus has been doing for ages, sets the filters to allow everything every time it works) Just go into manage logins, change the login script for www.hentai-foundry.com to the new login script, and put in some (throwaway) credentials, and you should be good to go.
I am also rolling out login scripts for shimmie, sankaku, and e-hentai, thanks to Cuddlebear (and possibly other users) on the github (which, reminder, is here: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System ).
Pixiv seem to be changing some of their login rules, as many NSFW images now work for a logged-out hydrus client. The pixiv parser handles 'you need to be logged in' failures more gracefully, but I am not sure if that even happens any more! In any case, if you discover some class of pixiv URLs are giving you 'ignored' results because you are not logged in, please let me know the details.
Also, the Deviant Art parser can now fetch a sometimes-there larger version of images and only pulls from the download button (which is the 'true' best, when it is available) if it looks like an image. It should no longer download 140MB zips of brushes!
other stuff
Some kinds of tag searches (usually those on clients with large inboxes) should now be much faster!
Repository processing should also be faster, although I am interested in how it goes for different users. If you are on an HDD or have otherwise seen slow tag rows/s, please let me know if you notice a difference this week, for better or worse. The new system essentially opens the 'new tags m8' firehose pretty wide, but if that pressure is a problem for some people, I'll give it a more adaptable nozzle.
Many of the various 'select from a list of texts' dialogs across the program will now size themselves bigger if they can. This means, for example, that the gallery selector should now show everything in one go! The manage import/export folder dialogs are also moved to the new panel system, so if you have had trouble with these and a small screen, let me know how it looks for you now.
The duplicate filter page now has a button to edit your various duplicate merge options. The small button on the viewer was too-easily missed, so this should make it a bit easier!
full list
- login:
- added a proper username/password login script for hentai foundry–double-check your hf filters are set how you want in your profile, and your hydrus should inherit the same rules
- fixed the gelbooru login script from last week, which typoed safebooru.com instead of .org
- fixed the pixiv login 'link' to correctly say nsfw rather than everything, which wasn't going through last week right
- improved the pixiv file page api parser to veto on 'could not access nsfw due to not logged in' status, although in further testing, this state seems to be rarer than previously/completely gone
- added login scripts from the github for shimmie, sankaku, and e-hentai–thanks to Cuddlebear and any other users who helped put these together
- added safebooru.donmai.us to danbooru login
- improved the deviant art file page parser to get the 'full' embedded image link at higher preference than the standard embed, and only get the 'download' button if it looks like an image (hence, deviant art should stop getting 140MB brush zips!)
- the manage logins panel now says when a login is expected to expire
- the manage logins dialog now has a 'scrub invalidity' button to 'try again' a login that broke due to server error or similar
- entering blank/invalid credentials is now permitted in the manage logins panel, and if entered on an 'active' domain, it will additionally deactivate it automatically
- the manage logins panel is better at figuring out and updating validity after changes
- the 'required cookies' in login scripts and steps now use string match names! hence, dynamically named cookies can now be checked! all existing checks are updated to fixed-string string matches
- improved some cookie lookup code
- improved some login manager script-updating code
- deleted all the old legacy login code
- misc login ui cleanup and fixes
- .
- other:
- sped up tag searches in certain situations (usually huge inbox) by using a different optimisation
- increased the repository mappings processing chunk size from 1k to 50k, which greatly increases processing in certain situations. let's see how it goes for different users–I may revisit the pipeline here to make it more flexible for faster and slower hard drives
- many of the 'select from a list of texts' dialogs–such as when you select a gallery to download from–are now on the new panel system. the list will grow and shrink depending on its length and available screen real estate
- .
- misc:
- extended my new dialog panel code so it can ask a question before an OK happens
- fixed an issue with scanning through videos that have non-integer frame-counts due to previous misparsing
- fixed a issue where file import objects that have been removed from the list but were still lingering on the list ui were not rendering their (invalid) index correctly
- when export folders fail to do their work, the error is now presented in a better way and all export folders are paused
- fixed an issue where the export files dialog could not boot if the most previous export phrase was invalid
- the duplicate filter page now has a button to more easily edit the default merge options
- increased the sibling/parent refresh delay for 1s to 8s
- hydrus repository sync fails due to network login issues or manual network user cancel will now be caught properly and a reasonable delay added
- additional errors on repository sync will cause a reasonable delay on future work but still elevate the error
- converted import folder management ui to the new panel system
- refactored import folder ui code to ClientGUIImport.py
- converted export folder management ui to the new panel system
- refactored export folder ui code to the new ClientGUIExport.py
- refactored manual file export ui code to ClientGUIExport.py
- deleted some very old imageboard dumping management code
- deleted some very old contact management code
- did a little prep work for some 'show background image behind thumbs', including the start of a bitmap manager. I'll give it another go later
next week
I have about eight jobs left on the login manager, which is mostly a manual 'do login now' button on manage logins and some help on how to use and make in the system. I feel good about it overall and am thankful it didn't explode completely. Beyond finishing this off, I plan to continue doing small work like ui improvement and cleanup until the 12th December, when I will take about four weeks off over the holiday to update to python 3. In the new year, I will begin work on what gets voted on in the poll.
Version 328
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v328/Hydrus.Network.328.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v328.tar.gz
I had a good week. The login manager is 95% done, and I got some neat ui stuff finished as well.
I am now considering what 'big thing' to work on next. Please see 'next week' below for more details.
login manager
The login manager is first-draft finished. It works and is turned on, but I have not added any new scripts, and I have yet to remove the hardcoded pixiv and hentai foundry login, so I can do some final testing. I would like advanced users to play with making new scripts and seeing where they fail, as I am not totally confident I have the 'this login messed up, what to do now?' error handling fixed yet.
I will also play with it next week, and assuming I can iron out any big problems, I will remove the hardcoded stuff and start adding login scripts for the existing big sites and some new ones like FurAffinity.
You may get a new error message this week if a HF or pixiv login fails. It may pause a related subscription as part of that. Please let me know how this works for you.
ui stuff
The options dialog now has a 'thumbnails' page, which collects several thumbnail related options. It also now has 'thumbnail border' and 'thumbnail margin'! If you would like a thicker–or thinner; it should even support 0 pixels for both–spacing between your thumbs, please check it out.
The manage tags dialog has had a layout pass. The clunky checkboxes and advanced buttons are collected into a cog icon, and the copy/paste/remove buttons are put into a single line. The whole thing also uses my new sizer, so as well as being cleaner and more compactable, it now expands better as well.
Manage tag siblings and parents now has reason 'suggestion' buttons when you petition a tag repository! Using these pre-defined reasons are helpful for the janitor of the repo (for the PTR, this is me, hydrus dev), as it collects all the same kind of petitions together into a single actionable chunk. If you regularly make sibling and parent petitions to the PTR, please give these buttons a go. It should save everyone time.
The network->pause menu now has some checkbox entries to pause file import queues, gallery searches, and watcher checking globally. This is a little ugly (the downloaders won't say this cleverly in their statuses, and they may take up to 30s to resume after being unpaused from this), but it works well as an temporary emergency stop-gap. I also expect to add an option to gallery and watcher pages in the coming weeks to start file import queues in a paused state by default.
full list
- wrote test ui for edit login script panel
- the login system now works and is turned on, although the legacy hardcoded pixiv and hf logins remain in place. it will not do anything very new this week–it is strictly only for advanced users to experiment with for now
- cleaned up some messy network code
- all subscription and hydrus jobs will no longer wait indefinitely on an invalid login–they will cancel immediately
- network jobs will report a bit more info when they are cancelled
- subscriptions will now attempt to test login validity before and during file downloads and syncing. if they fail, the sub will pause and stop work and a message will be presented to the user
- made a 'thumbnails' options page and moved some things to it
- added thumbnail border and margin to that thumbnails page! you can even set 0 border and/or margin and it works
- fixed up a heap of bad thumbnail drawing code that didn't work with thicker borders
- the tag sibling and parent dialogs now have suggestion buttons in their 'give a reason' dialogs! if your petitions are simple and fit into one of these categories (which is most of them), please use these buttons as they will let janitors (e.g. hydrus dev for the PTR) process them in batches, in fewer clicks
- manage tags dialog's checkboxes and advanced buttons are now wrapped into a cog icon! remove/copy/paste buttons are compacted and put on the same row!
- manage tags dialog's copy button now only copies selected, not all tags
- manage tags dialog now uses the new sizer. some components are smaller by default but will eat up spare pixels better
- misc manage tags code cleanup
- added 'paged file import queues' to the network->pause menu. this will pause any, hdd, url, simple, gallery, or watcher page from processing its file import queue. it is a bit hacky and will take up to 30s to unpause unless you joggle the respective downloader to wake it up–see how it goes!
- added a similar 'gallery searching' to that menu, to pause any url, simple, or gallery page from actioning its gallery-side queue
- and 'watcher checking' for watchers
- fixed a stupid bug that was causing false-positive and _mostly_ harmless errors for certain pixiv and artstation multi-page downloads
- fixed an issue where multi-page data was being mis-metadata'd (mostly, this meant thread watchers were giving the last filename tag to all files, and I think similarly getting the latest source time) due to a mistake in some recent de-duping code
- fixed the new pixiv file page api parser to parse source time, which I must have accidentally deleted previously
- fixed a no-expand bug in my new sizer when in horizontal orientation
- fixed a small bug when making an easy-import downloader png and cancelling the add url class dialog
- misc fixes
next week
The login manager work is basically done. I need to iron out some last error situations so I am happy, and then I can write/integrate some new login scripts in my normal work time. I am thus 'finished' with my big work focus and very happy to start something new. My current plan is to spend mid-December to mid-January doing the python 3 update, which leaves a decent bit of time to have a discussion about what to work on next and put a poll up for voting. Please check out the thread at >>10429 for the current list.
Version list
Hey do we have a compiled list of every version and it's features?
I haven't updated in AGES and I imagine there are a lot of new things, but I don't want to dig through every individual OP to find out.
Can you make a compiled list of versions with major changes, kindof like lists of minecraft updates as an example, where each version will broadly list "bug fixes" or "added desert biome" longer explanations for larger feature updates.
Or even, can that happen going foreword somehow?
Reposting from >>>/tech/457184
Maybe relevant to your interest.
Japan did it.
Automatic anime images tagging with neural networks.
THE FUTURE IS NOW
http://illustration2vec.net/papers/illustration2vec-main.pdf
Big things to work on next
With the download engine and login manager coming to a close, I will need something new to be anxious about and near-overwhelmed by. I will put up a poll in a few weeks for everyone to vote on a big list of possible new features that are too large to fit into my normal weekly work. The poll will allow you to vote on multiple items. I hope to work on the most voted-on item for two to three months before starting the cycle again.
This thread is for discussion of the list, which at current looks like this:
- Just catch up on small work for a couple of months
- Improve tag siblings/parents and tag censorship
- Reduce crashes and ui jitter and hanging by improving ui-db async code
- Speed up tagging workflow and autocomplete results
- Add ways to display files in ways other than thumbnails (like 'details' view in file explorers)
- Add text and html support
- Add Ugoira support (including optional mp4/webm conversion)
- Add CBZ/CBR support (including framework for multi-page format)
- Add import any file support (giving it 'unknown' mime)
- Improve 'known urls' searching and management
- Explore a prototype for neural net auto-tagging
- Add support for playing audio for audio and video files
- Add OR file search logic
- Add an interface for waifu2x and other file converters/processors
- Write some ui to allow selecting thumbnails with a dragged bounding box
- Add popular/favourite tag cloud controls for better 'browsing' search
- Improve the client's local booru
- Improve duplicate db storage and filter workflow (need this first before alternate files support)
- Improve shortcut customisation, including mouse shortcuts
- Import/export ratings, and add 'rating import options' to auto-rate imports
- Add more commands to the undo system
- Improve display of very large/zoomed files in the media viewer
- Set thumbnail border colours on user-editable rating and namespace conditions
- Improve hydrus network encryption with client cert management and associated ui
- Add tag metadata (private sort order, presentation options, tag description/wiki support)
- Write a repository-client refresh/resync routine to clear out junk data and save space
- Prototype a client api for external scripts/programs to access
- Support streaming file search results (rather than loading them all at once once the whole query is done)
- Increase thumbnail size limit (currently 200x200)
- Add an optional system to record why files are being deleted
- Improve file lookup scripts and add mass auto-lookup
- Cleanup code and improve practises
- Add multiple local file services
- Add an incremental number tagging dialog for thumbnails
I am happy to work on any of these items. If you have questions, please ask, and if you have suggestions for new items, go ahead.
Version 329
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v329/Hydrus.Network.329.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v329.tar.gz
I had a great week. The login manager is now ready for everyone, and I did some neat other stuff as well.
login manager
I have fixed up how the login manager deals with various errors and am happy for everyone to give it a go. This release also introduces login scripts for Deviant Art, Danbooru, and the Gelbooru sites. >Please check it all out under network->downloaders->READY FOR CAUTIOUS USE: manage logins. This ui needs a bit more work to make it user-friendly, but you basically enter some credentials (username & password) for a domain and then 'activate' the login script for it. Thereafter, the client will try to log in to that site whenever it needs something from it and doesn't think it is logged in. It will make a little popup window while it logs in just to let you how it is doing. If it fails, it will 'block' that domain, and any network jobs will hang on it and any subs will pause on it. Most logins last from 30 days to five years, so after the initial success, you likely won't see it ever again, particularly if you use subscriptions. It seems to work ok!
That said, feedback would be appreciated, particularly for the individual scripts. If you discover the client does not notice when it fails to log in, or fails to notice when it has naturally been logged out, or keeps reattempting a subscription that has an invalid login or something, please let me know.
Caveats:
As it says on the dialog: With hydrus, use throwaway accounts or accounts you otherwise do not care much about. The credentials are stored in plaintext, and if something goes wrong with the login system or you binge one night and download too much, you don't want to get a black mark on an account you care about.
The deviant art login page only seems to work if your client has some history of downloading from Deviant Art previously. Otherwise, it puts up a 'Are you a robot?' captcha challenge, which this first login manager cannot deal with. If you have been downloading from Deviant Art for a while, you should be fine (otherwise, try downloading a few SFW files first and then try activating the login). The DA parser is also updated to correctly download NSFW images (which you will have access to once logged in).
I don't know when this happened, but it seems that Pixiv now also allow guests to access SFW images. This is a neat development, but it means my login script as-is doesn't detect when it fails to log in (since pixiv now give you a 'guest' login regardless). If you have had trouble getting NSFW pixiv images and can't figure it out, please check your credentials are all correct and try deleting your cookies on the login dialog to try again. I will revisit the script next week to see if I can better detect a failed login.
I looked at FurAffinity this week, since it has been highly requested, but found they require a captcha even on regular login attempts (and not, say, just sign-ups). This throws a wrench in our plans, as this first version cannot deal with captcha. Unless there is a simpler alternate login system, there is likely no good solution here for now, and FA users will have to copy cookies across from their browser. (which I still intend to make more user-friendly and reliable in the coming weeks)
I had hoped to roll out a Hentai Foundry login this week, but it turns out their login cookies is dynamically named, so I will have to tweak the login manager to deal with it. Not a huge issue, but it will be delayed a little. The click-through guest login is now working entirely on the new system and appears to be fine.
other stuff
Texts in the program that ellipsize (Like this…) when they are too short now present tooltips with the full text. This includes the status on the network job control, which will help you review any long login errors.
The manage tags dialog's buttons get a bit more tweaking this week: both the remove and copy buttons now work differently depending on whether any tags above are selected or not: if tags are selected, only those will be affected; if none are selected, all will be affected. Also, the remove button is now wrapped in a yes/no dialog that can be turned off in the cog menu.
The filename tagging options panel now allows you to additionally tag files by their last, second last, and third last directory names!
The export files panel now allows you to delete the files from the client after export. This checkbox value is remembered and presents a WARNING yes/no dialog on export just to make sure.
Manage tag siblings and parents now handle large 'add' batches as one transaction, whether that is by a big .txt/clipboard import or when adding multiple parents. Everything is now bundled into one big list and you are only asked for the pend 'reason' once.
full list
- login:
- the login manager is fully turned on! hentai-foundry click-through and pixiv login now occur fully on the new system
- wrote a Deviant Art login script for NSFW downloading–however, it only seems to work on a client that has done some logged-out downloading first (otherwise it thinks you are a robot)
- updated the DA file page parser to only NSFW-veto if the user is currently logged out
- wrote a danbooru login script for user prefs and special files if you have a gold account
- wrote a gelbooru 0.2.x login script for user prefs
- pixiv recently(?) allowed non-logged in users to see sfw content, so the login script is updated to reflect this. the login script doesn't detect a failed login any more, so I will revisit this
- logging in in the regular order of things now makes a temporary popup message with the overall login status and final result. ~it is cancellable~–and if cancelled, future login attempts will be delayed
- logging in in the regular order of things now prints simple started/result lines to the log
- deleted old network->login menu and related code such as the custom pixiv login management. gdpr click-through is now under downloaders
- subscription login errors will now specify the given login failure reason
- subscription login tests will now occur at a better time, guaranteeing the sub will be correctly saved paused if the test fails
- login errors will now always specify the domain for which they failed
- testing a login script on a fresh edit login script dialog now pre-fills the alphabetically first example domain
- the login script test ui now restores its 'run test' button correctly if the test is abandoned early
- misc improvements to login error handling and reporting
- .
- other:
- any texts across the program that ellipsize when they are too thin to display what they have will now tooltip their text (this most importantly includes the status on the network job control, which will now display full login problem info)
- the copy button on manage tags goes back to copying all if no tags are selected
- the remove button on manage tags now removes only selected if some tags are selected. it still removes all if none are selected
- the remove button on manage tags is now wrapped in a yes/no dialog (as is hitting the delete key on the list's selection). this can be turned off under the cog button
- filename tagging panels now support directory tagging for the last, second last, and third last directories. the related code for handling directory tagging is cleaned up significantly
- the export files panel now lets you delete the files from the client after export. this value will be remembered, and if on will prompt a capital letters warning on export, either via the button or the quick-export shortcut
- in manage tag parents, where there are multiple parents in a pending action (either by importing via clipboard/file or by putting multiple parents in right-hand box), the action will now be treated as one transaction with one 'enter a reason' confirmation!
- in manage tag siblings, when multiple 'better' values are pended in one action via a clipboard/file import, they will now be treated as one transaction with one 'enter a reason' confirmation!
- .
- misc:
- added a new url class that api-links .gifv-style imgur links so they are downloadable like regular imgur single media pages
- the pixiv manga page url class now redirects to the new api, so mode=manga pages should now be drag-and-drop importable and generally downloadable if you have any still hanging around in any queues
- clients now come with an additional danbooru parser that fetches the webm version of ugoiras
- after discovering a pdf that ate indefinite 100% CPU while trying to parse, I have decided to stop pulling num_words for pdfs. it was always a super inaccurate number, so let's wait for a better solution at a later date. hydrus hence no longer requires pypdf2
- fixed an issue with monthly bandwidth estimates rolling over to the new year incorrectly
- in an attempt to chase down a duplicate files content move/copy bug, the duplicate action content updates got a bit of cleanup work. if you have noticed duplicate actions not copying tags/urls, please let me know the exact process in the ui, including services and merge options, you went through
- tag lists should now update their sibling appearance correctly after a tag siblings dialog ok–previously, they were checking for new sibs too early
- tag siblings and parents should now refresh their data more efficiently when spammed with new data notifications (this usually happens janitor-side, which approving dozens at once)
- copy queries/watcher urls on the download pages' lists' right-click menus no longer double-spaces the copied texts (it just does single spaces)
- fixed an issue where certain initialised watchers were erroring out when asked to provide next-check time estimates–in all cases, null timestamps will be dealt with better here
- misc tag parents/siblings ui code cleanup
- wrote some code to catch and report on an unusual dialog dismissal error
next week
I'll do a bit more login manager work, figuring out easy solutions for the difficult cases where I can and rolling out more scripts, and otherwise catch up with small work. I have five weeks left to make a nice 'final' python2 release before I take my holiday break to go up to python3.
Release Tomorrow!
I had a great week. There's some more login manager work, including better ui and login scripts for several new sites, a bunch of bug fixes, some layout improvement and cleanup, and some faster search and tag processing.
The release should be as normal tomorrow.
Source code documenting / project structure cataloging
For some time now I have been meaning to start looking into the source code of Hydrus to see what it can do and what might be improved upon without bothering Hydrus Dev too much.
Problem is, according to my knowledge, the source code is not documented(or commented) at all and the only one who knows what's really going on within the program is Hydrus dev himself, bless his soul.
I was wondering if I missed something and there is something like documentation out there and if there isn't I wondered if there were people willing to help document the whole project, or at least try to figure out what's going on exactly.
So far I have come up with the following topics if anyone were to write something like a documentation, feel free to add to them:
-How to set up your development environment.
-How is the program structured, how are objects/modules linked
-What are the functions of the objects/modules
-How are the objects/modules structured
-How does the program as a whole function
-How to interact with said objects/modules
-How to add/write your own objects/modules to the hydrus client and server
I don't know if I nor anyone else has the resources, time or spirit to do this but I wanted to at least ask. Having documentation would help people to inject personal customization into hydrus to make it fit their needs and maybe even spawn a small modding community. It would allow the program to flourish as a whole without forcing everything on a single point/person.
Release Tomorrow!
I had a great week. The login manager is ready for everyone to try out, and I've written login scripts for Deviant Art, Danbooru, and the Gelbooru sites. I also did a bunch more ui work, so there are also improvements to manage tags, tag siblings, tag parents, and more.
The release should be as normal tomorrow.
sankaku channel login script
i've created a very simple login script for sankaku channel: https://pastebin.com/Qb4sWnna
it seems to work fine. feel free to use/adapt as needed.
logging in on sankaku channel is required to circumvent the default 25 page limit. while being logged in, you can access up to 50 pages. it should in theory to get unlimited pages if we are able to pass the next parameter that is needed for page 51 and beyond (e.g., https://chan.sankakucomplex.com/?next=7285636&page=51), but that would need adaptions in the page parser and i'm not sure how to do it (or even if it can be done at all with the current configuration options). if anyone has an idea about that, i'd love to hear it!
Version 327
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v327/Hydrus.Network.327.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v327.tar.gz
I had an ok week. The login manager is moved forward, and I did a few other things as well.
login manager
Still advanced users only.
The login manager now saves itself, and I've don the dialog for regular users to set up their login credentials, under network->downloaders->manage logins. Feel free to poke around. It stil isn't switched on, so custom login scripts will not run yet.
The advanced side is missing some good test ui, and the regular side is missing a similar 'attempt login/test these credentials'. I'd like to roll something out next week and turn the system on at the same time.
misc
The spammy/false-positive 'periodic file limit' warnings that subscriptions started giving last week with the new, more liberal syncing check are now gone. You'll now only get a warning about the file limit when it matters (when the sub only found new files on the check).
Subscription merge is a bit more helpful. It lets you select which is the 'prime' sub, into which to merge the others, and has some other behind-the-scenes data improvements as well.
Several new pixiv downloading issues, mostly to do with multi-page content like manga, are fixed.
The manage tag censorship, siblings and parents dialogs are now all moved to the new panel system, a notebook instead of the buggy old listbook, and also use my recent new sizer. They should fit better on smaller screens and expand better on larger ones. Let me know if you still have trouble with them, as I can tweak a lot more easily now.
I have finished some new 'getting started' help for using the new download system. If you have yet to download anything in client but want to try, please check it out here:
https://hydrusnetwork.github.io/hydrus/help/getting_started_downloading.html
full list
- login stuff:
- finished off some login script data stuff
- fleshed out how login credentials and other linked data is stored in the login manager, including script link recovery when the script changes but name does not
- improved some initialisation login validation error handling
- improved login failure validation error handling
- wrote a dialog panel for managing login credentials and reviewing validity and so on
- a heap of related session and login tie-in/fix-up work
- the login manager will now save changes to the db. it will get the HF and pixiv scripts on db creation/update, and if you have a pixiv login, the login system will pre-fill that info and 'activate' the script (although the login manager will not fire any login scripts yet–if so configured, it'll just delay on a polite error message)
- .
- other stuff:
- with the subscriptions' new more liberal syncing logic, the periodic file limit will now only pop up if the sub does not see any already-seen files
- to give more buffer for the new syncing logic, file import caches will now store 250 entries minimum on compaction (was 100 previously)
- subscription merging now lets you choose the primary subscription into which the other subs will be merged
- cancelling a subscription merge action mid-merge is now safely nullipotent
- post urls that use subsidiary page parsers (such as the new pixiv manga parser) will now correctly insert (rather than append) their manga urls into the file import cache
- removed a couple of places where urls could accidentally be duplicated in a file import cache
- cleaned up some areas where successful file import objects were presumed to have file hashes when they might not (this was causing errors when importing urls that split into multiple url children, like pixiv manga, while also having 'additional tags' set)
- updated tag censorship, parents, and siblings dialogs to the new panel system
- tag censorship, parents, and siblings panels now use a notebook instead of the layout-borked listbook
- tag parents and siblings panels now use the new small-resolution-friendly sizer, are more tight by default, and expand more neatly
- refactored a bunch of tag ui code to clientguitags
- the client video renderer will now deal with videos with (invalid) duration of 0 more gracefully
- finished the 'getting started with downloading' help page, sans the login stuff
- bit of other help work
next week
I would like to 'finish' the login manager, which I think will mean turn it on at a 1.0 state. I suspect I will have to go back a bit for a week or two more to clean things up, and I'd also like to roll out some login scripts (and downloader updates) for Deviant Art and Fur Affinity and some other big sites. We'll see how it shakes out.
Release Tomorrow!
I had a good week. The login manager is turned on for advanced users, and I did some neat ui stuff, including custom thumbnail border and margin thicknesses and greatly improved layout for the manage tags dialog.
The release should be as normal tomorrow.
Version 326
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v326/Hydrus.Network.326.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v326.tar.gz
I had a great couple of weeks. I was able to get some good prototype login stuff out for advanced users to play with and a bunch of other stuff besides.
Some login behind-the-scenes changed this week. Please report any problems you have with hanging/failing Pixiv, Hentai Foundry, or Hydrus service (i.e. the PTR) jobs.
login
Advanced users only for now!
On Tuesday the 9th, I had a bunch of login work 75% complete and not much else. I decided to make this a two-week cycle, and I am happy to have finished off the 25% and some more. Please check out the new dialog under network->downloader definitions->manage login scripts.
As simple as I have attempted to make it, the new login manager is complicated, and the new dialog matches. Just as with the downloader, I expect to write some 'how to write a login script from scratch' help, but for now, please feel free to explore. Hit the add defaults->add them all to see some first-draft replacements I have knocked up for the Hentai Foundry click-through and Pixiv login to see what I am going for here. It basically hits a number of GET/POST requests in order, drawing from user credentials like username/pass and temp variables parsed and passed from one page to another.
None of this code is functional yet. The dialog will not save any changes, and there is no test panel, which I sorely missed as I put the example scripts together. I may add or tweak a thing here or there, but hope to turn somo of it on next week, maybe along with some regular-user-friendly ui to manage what scripts are active and store credentials and so on. I hope to swap out the hardcoded HF and Pixiv logins with finalised scripts and write a couple more for the big sites myself.
subs and downloaders
Pixiv is still in flux, and it seems they broke our manga page parsing for a subset of users this week. Thankfully, user kourraxspam on the discord figured out a neat API fix to our problems and also wrote a pixiv tag search. I tweaked the new objects a bit and folded them into the update. Since we are now mostly pulling from APIs, I hope Pixiv will be a bit more stable. Please let me know if you have any more trouble.
The new Pixiv downloader no longer pursues mode=manga URLs. If you ran into big Pixiv problems this week and have a lot of failed 'mode=manga' pages hanging around in your subs, you can try 'try again'ing the neighbouring failed/skipped mode=medium URLs, or try to re-run the queries in a manual download page, or if you are comfortable with the new download system, there is more technical url-class-based fix in the discord, under #parser-creation.
Subscriptions get some QoL improvements this week. Firstly, the way they determine if they are 'caught up' to the last sync now permits occasional deeper searches that will find files that were tagged late. In my final IRL testing here, I found it worked well but sometimes hit the 'periodic file limit' and gave me annoying little popups about it. These popups are false positive and you can dismiss them. I will improve the subscription logic next week to recognise this situation and not make the popups.
Also, subs can now publish to 'labels'. In the edit subscription panel, you can set a specific name for a sub to publish its button or page files to (overriding the default, which is subscription name). So, if you have several 'character' subscriptions for different sites, you can have them all publish their files to the same 'char subs' page!
The watcher and downloader pages also have better en-masse file handling in their lists' right-click menu, which now has a 'show all in new page' entry that lets you combine the results of multiple watchers or gallery imports into one page, letting you handle multiple finished queues all in one go!
layout improvements
After thinking about it a long time, I have written a custom sizer that lays ui items out in a more conservative way. I have applied it to several locations, particularly the 'management' panels on the left of any page, and hope it will make these panels size a bit more sanely on smaller screens. This first step is limited, and it isn't perfect by any means, but it should be an improvement, with fewer 'why is this thing so tall?' moments.
If this sizer doesn't cause huge problems, I expect to apply it across the program and then start tweaking the minimum sizes and related sizer flags to take better advantage of it. If you are on a small screen, please let me know if you notice any big changes, for better or worse, for this week and to come. There is plenty of other related cleanup to do here. I'll also be tackling the manage tag parents/siblings dialogs' layout-hellscape soon.
misc
If you right-click several tags, you can now open separate new search pages for each of them separately!
Autocomplete tag searches should now return faster! My tests showed a 33% reduction! They also handle some unusual wildcard searches better!
The gallery log button can now be right-clicked to restart a recently failed search! Restarted searches should be more reliable!
Under options->downloading, you can now have the little 'x/y' import summaries say 'x/y - zN', where z is the number of x that were 'new' (as opposed to already in db)!
full list
- login:
- finished the new login objects. they can deal with multi-step single second-level domain login problems, can pass variables from step to step, and use cookies as success verification
- wrote an ton of ui for the new login objects, now under network->downloader definitions and network->logins. it is not 'active' yet, but advanced users are invited to check it out. there is no good test ui yet, which I think I'll have to figure out in the coming weeks
- wrote a first attempt at HF and pixiv replacement login scripts–please try importing from defaults on the manage login scripts dialog and look through them to see what I am going for. once the system is flipped on and we are happy these work, I'll remove the old hardcoded legacy login stuff
- when a network job that needs a login cannot login, it now waits (rather than bombing out completely), presenting the related error, and checks again every 60 seconds
- if a network job thinks it can login but fails to generate a login process, the network engine now catches the error safely and recovers. the job is put on hold as above
- .
- subs:
- the subscription 'have we caught up to where we were before' test is now more complicated–rather than just stopping after five 'already seen' urls are found, it now only stops if at least the _last_ five contiguous urls of the page are already seen. this will catch more late-tagged files that get inserted out of order
- fixed the 'get quality info' button on edit sub panel to only get the current selection, not all queries wew
- subscriptions can now optionally publish/present their files to a specific label! this is a great way to merge multiple subs to the same final landing page
- .
- layout:
- after a long time thinking about it, wrote a new custom boxsizer that handles resizing multiple expanding items of different reasonable min size by expanding them _beyond minimum size_ by their proportion, rather than forcing them all to have total proportional width/height. I expect to polish this and apply it in multiple locations around the program where tall things were being too tall because something else was forcing it to be (the management panel on the left of most pages was terrible at this, causing a giganto taglist just because the upper panel was tall as well).
- changed my custom boxsizer (the box with a bold header) to the new custom boxsizer, so it is all over now–please report any bad layouts you see
- in an effort to improve layout, the manage tag parents and siblings panels' preview boxes have shorter minimum height–it will get a bigger layout overhaul soon
- .
- bigger misc:
- thanks to work of user kourraxspam on the discord, fixed the pixiv downloader to use a more stable api and added pixiv tag search
- watchers and gallery imports now have a list right-click menu entry to show all selected importers' files in a new page! use this to clear out a bunch of finished queues all at once!
- the tag right-click menu now offers 'open new search pages for each in selection' if multiple tags are selected–this will open three search pages each with one tag, as opposed to the original entry, which would only open one page with all three
- the edit nested gug panel now uses a checklistbox rather than the menu to select gugs to add, which is more reliable and allows for multiple selections
- sped up autocomplete tag fetches' tag sibling integration–irl this may be a reduction in total a/c search time of approx 33%
- page parsers will now generate next gallery urls absent any file/post urls if the only type of url they can generate is gallery urls (so a meta-gallery-search like board->threads that only generates subsidiary gallery pages will now work, whereas before it never could because it was missing post urls)
- the gallery log now provides a shorthand way to restart and resume failed searches from its right-click menu (if the most recent log entry failed)
- 'try again (and allow search to continue)' reattempt jobs will now generate next page urls even if no new urls are found (which can happen if a search stopped due to the file limit exactly lining up with the number of files found, for instance, so a reattempt finds nothing new)
- gallery downloaders will now specify their 'delay work for a bit' error states in the ui. this usually means 'could not connect', which has a 4-hour timer (I'll prob add a scrub delays button here at some point)
- the watcher will now show its 'delay work for a bit' error state in more places in the ui
- added a 'media' shortcut 'export_files_quick_auto_export', which will open the export files frame and give you a quick yes/no to confirm you want to export as set. if yes, it will export. then it will close the frame
- added a 'show a "N" to short import summaries' option to options->downloading, which will extend the typical 'x/y' status string to 'x/y - zN' for z 'new files' (as opposed to already in db)
- improved how the video parser estimates frame rate–it _should_ fix some of those low-framerate, low-framecount slideshow-vids where at current they render everything in a rush and then sit on the last frame for ten secs
- .
- smaller misc:
- network report mode now reports url_to_fetch and parser-to-parse-with info
- when the server fails to accept a file upload due to a file parsing issue, it now prints the hash of the file in the error
- if the client sees a possible file hash in a server error message from a file upload, ~it will try to show that file in a new page~
- fixed an issue where wildcard searches were not finding results if the search text included the normally discarded characters [](){}"'
- fixed some domain handling for localhost and other undotted network names
- content parsers will now only launch with permissable content types, which for the legacy 'lookup scripts' scripts system means only tags and vetoes, and for the new login system means only temp variables and vetoes
- as compaction now happens automatically on sync, removed the 'compact' button from edit subs panel
- an unusual network error related to hydrus update files sometimes being cut off mid-stream is now glossed over silently, with the download reattempted after a delay
- the initial gui session load now occurs after a 0.25s delay–let's see if it cleans up some initial layout issues some users have had
- maybe fixed an odd dictionary-initialisation error related to tag siblings/parents dialog boot
- ruggedised against an unusual bandwidth load bug
- gave some of the index help a pass
- did most of a 'getting started with downloaders' page in the help–I'll finish it next week
- updated discord share link to https://discord.gg/3H8UTpb , which should not expire
- some listbox add/edit code cleanup
- some listctrl delete code cleanup
- misc help work
- misc cleanup
next week
More login manager. I'd love to plug my new system in and have it do some work, but we'll have to see how doable that is–I might need to add some more supporting ui first.
I have been thinking about my longer-term schedule. I had thought to cram the login manager into the next two weeks and then take four weeks (basically November) off to do a big python 3 update, but I am now leaning towards putting the rewrite off so it bridges the holiday. Christmas is the best time to not be putting out releases for everyone's convenience, and since it is so close, I think it makes sense to just delay it that few more weeks. This will also give me a buffer to ensure the login system isn't exploding and catch up on whatever small work still needs doing. I could even do some prep work on the next big thing, which I still expect to put up a poll for once the login system is done.
The Struggle Is Real
Hey everybody, this is just a thread to talk about our everyday life with Hydrus.
How are you coping with your file hoarding addiction now that you're being enabled?
Are you remembering to take some time to tag what you download?
You ARE making a monthly backup next week, right?
What kinds of files are you saving, and what sorts of tags are you committing to the repo?
Right now I'm sorting through the downloads from a few of my NSFW subscriptions pre-tagging. They are all about half stuff I want and half gay shit I don't want, so I put "impure" in their subscription tags and I just go through 100 at a time and purify them. It does tend to break about even with me keeping 50 files out of each 100. Pray for me anons.
…AND YOU?
Version 325
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v325/Hydrus.Network.325.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v325.tar.gz
I had a difficult week, but I got some great work done. Save for some final help revisions, the downloader overhaul is complete.
final downloader work
So, I managed to finish 13 of my 15 final jobs in the downloader overhaul. All that remains is a help pass for subscriptions and a better intro to gallery and watcher downloading, which I will fold into normal work over the coming weeks. This has been a longer journey than I expected, but I feel great to be done. This final work is mostly unusual stuff that got put off.
For instance, subscriptions can now run without a working popup (i.e. completely in the background)! It works just like import folders, as a per-subscription checkbox option, and still permits the final files being published to a popup button or a named page. I recommend only trying this after the initial sync has completed, just so you know the sub works ok (and isn't accidentally downloading 2,000 garbage files in the background!).
Also, subscription queries can now take an optional 'display name'. This display name will appear in lieu of the actual query text in most display contexts, like the edit sub panel or a popup message or a publishing destination. A query for pixiv_artist:93360 can be more neatly renamed to and managed as 'houtengeki', and 'xxxxxxxx' can be renamed 'family documents, DO NOT ENTER' and so on.
And subscription queries now have individual tag import options that only support 'additional tags'. So, if you want to give a particular query a blog-related creator tag or a personal processing tag, this is now simple.
If you 'try again' on a 'deleted' file import, the client will now ask if you want to erase that deletion record first (i.e. overriding it and importing anyway)! This is obviously much quicker and simpler than having to temporarily edit the file import options to not exclude previously deleted.
Gallery and Watcher pages now have quick 'retry failed' buttons and list right-click menu entries.
advanced stuff
If you are in advanced mode, subscription edit panels now have a 'get quality info' button. If you select some queries and hit this (oh fug, I just tested it IRL and discovered it does it for all queries, not just selected, wew, I will fix this for next week), the client will do some hacky db work and present you with a summary of how many files currently in those queries are inbox/archive/deleted, and a percentage of archived/(archived+deleted)–basically "after processing, you kept 5% of this query". This should help you figure out which queries are actually 'good' for you and which are just giving you 98% trash. I can do more here, but this is just a quick prototype. Feedback would be appreciated.
The downloader easy-import pngs now support custom http headers and bandwidth rules! This is a bit experimental, so test it a bit please before you roll it out for real. If you have custom headers or specific bandwidth rules for the domains in your export downloaders gugs, they will be added automatically, and there's a button to add them separately as well. Exporters and importers will get detailed previews of what these new 'domain metadata' objects include.
If you are in advanced mode, file import options now have options to turn off url- and hash-based 'skip because already in db/previously deleted' checks. They are basically a "I don't care what you think the url is, just download it anyway and see if it is a new file m8". If you have a particular url conflict that was causing an incorrectly skipped download that I have previously discussed with you, please try these options and reattempt the problem file. Don't use them for regular downloads and subs, or you'll just be wasting bandwidth. Advanced file import options now also allow you to turn off source url association completely.
full list
- added a 'show a popup while working' checkbox to edit subscription panel–be careful with it, I think maybe only turn it off after you are happy everything is set up right and the sub has run once
- advanced mode users will see a new 'get quality info' button on the edit subscription panel. this will some ugly+hacky inbox/archived/deleted info on the selected queries to help you figure out if you are only archiving, say, 2% of one query. this is a quickly made but cpu-expensive way of calculating this info. I can obviously expand it in future, so I would appreciate your thoughts
- subscription queries now have an optional display name, which has no bearing on their function but if set will appear instead of query text in various presentation contexts (this is useful, for instance, if the downloader query text deals in something unhelpful like integer artist_id)
- subscription queries now each have a simple tag import options! this only allows 'additional tags', in case you want to add some simple per-query tags
- selecting 'try again' on file imports that previously failed due to 'deleted' will now pop up a little yes/no asking if you would like to first erase these files' previously deleted file record!
- the watcher and gallery import panels now have 'retry failed' buttons and right-click menu entries when appropriate
- the watcher and gallery import panels will now do some ui update less frequently when they contain a lot of data
- fixed the new human-friendly tag sorting code for ungrouped lexicographic sort orders, where it was accidentally grouping by namespace
- downloader easy-import pngs can now hold custom header and bandwidth rules metadata! this info, if explicitly present for the appropriate domain, will be added automatically on the export side as you add gugs. it can also be bundled separately after manually typing a domain to add. on the import side, it is now listed as a new type. longer human-friendly descriptions of all bandwidth and header information being bundled will be displayed during the export and import processes, just as an additional check
- for advanced users, added 'do not skip downloading because of known urls/hashes' options to downloader file import options. these checkboxes work like the tag import options ones–ignoring known urls and hashes to force downloads. they are advanced and should not be used unless you have a particular problem to fix
- improved how the pre-import url/hash checking code is compared for the tag and file import options, particularly on the hash side
- for advanced users, added 'associate additional source urls' to downloader file import options, which governs whether a site's given 'source urls' should be added and trusted for downloaded files. turn this off if the site is giving bad source urls
- fixed an unusual problem where gallery searches with search terms that included the search separator (like '6+girls skirt', with a separator of '+') were being overzealously de/encoded (to '6+girls+skirt' rather than '6%2bgirls+skirt')
- improved how unicode quoted characters in URLs' query parameters, like %E5%B0%BB%E7%A5%9E%E6%A7%98 are auto-converted to something prettier when the user sees them
- the client now tests if 'already in db' results are actually backed by the file structure–now, if a the actual file is missing despite the db record, the import will be force-attempted and the file structure hopefully healed
- gallery url jobs will no longer spawn new 'next page' urls if the job yielded 0 _new_ (rather than _total_) file urls (so we should have fixed loops fetching the same x 'already in file import cache' results due to the gallery just passing the same results for n+1 page fetches)
- in the edit parsing panels, if the example data currently looks like json, new content parsers will spawn with json formulae, otherwise they will get html formulae
- fixed an issue with the default twitter tweet parser pulling the wrong month for source time
- added a simple 'media load report mode' to the help debug menu to help figure out some PIL/OpenCV load order stuff
- the 'missing locations recovery' dialog that spawns on boot if file locations are missing now uses the new listctrl, so is thankfully sortable! it also works better behind the scenes
- this dialog now also has an 'add a possibly correct location' button, which will scan the given directory for the correct prefixes and automatically fill in the list for you
- fixed some of the new import folder error reporting
- misc code cleanup
next week
Now I will finish a simple login manager. Fingers crossed, I hope to spend a total of three to four weeks on it. I don't expect I'll have anything interesting ready for it for v326, but maybe I'll have some dummy ui for advanced users to play with.
Thanks everyone!
Release Tomorrow!
I had a great couple of weeks. The login manager is ready for advanced users to play with, and I've done a bunch of other work besides, including plenty of fixes and ui additions, some tag autocomplete speedup, layout improvements for small-screen users, and better en-masse downloader & subscription file management.
Due to the two-week cycle, there is a lot more than usual to test. The release may be a bit late tomorrow.
No Release Tomorrow!
I had a good week, but the login manager work (of which there is a lot!) took up so much time that I only fixed a handful of bugs otherwise. I think it would be better if I spend tomorrow hacking away at more of the login stuff than putting out a small release with some not super-urgent work.
So, I am moving v326 forward to the 17th. Everything else is as normal.
Version 324
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v324/Hydrus.Network.324.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v324.tar.gz
I had a great week. The downloader overhaul is almost done.
pixiv
Just as Pixiv recently moved their art pages to a new phone-friendly, dynamically drawn format, they are now moving their regular artist gallery results to the same system. If your username isn't switched over yet, it likely will be in the coming week.
The change breaks our old html parser, so I have written a new downloader and json api parser. The way their internal api works is unusual and over-complicated, so I had to write a couple of small new tools to get it to work. However, it does seem to work again.
All of your subscriptions and downloaders will try to switch over to the new downloader automatically, but some might not handle it quite right, in which case you will have to go into edit subscriptions and update their gallery manually. You'll get a popup on updating to remind you of this, and if any don't line up right automatically, the subs will notify you when they next run. The api gives all content–illustrations, manga, ugoira, everything–so there unfortunately isn't a simple way to refine to just one content type as we previously could. But it does neatly deliver everything in just one request, so artist searching is now incredibly faster.
Let me know if pixiv gives any more trouble. Now we can parse their json, we might be able to reintroduce the arbitrary tag search, which broke some time ago due to the same move to javascript galleries.
In a similar theme, given our fully developed parser and pipeline, I have now wangled a twitter username search! It should be added to your downloader list on update. It is a bit hacky and may be ultimately fragile if they change something their end, but it otherwise works great. It discounts retweets and fetches 19/20 tweets per gallery 'page' fetch. You should be able to set up subscriptions and everything, although I generally recommend you go at it slowly until we know this new parser works well. BTW: I think twitter only 'browses' 3200 tweets in the past, anyway. Note that tweets with no images will be 'ignored', so any typical twitter search will end up with a lot of 'Ig' results–this is normal. Also, if the account ever retweets more than 20 times in a row, the search will stop there, due to how the clientside pipeline works (it'll think that page is empty).
Again, let me know how this works for you. This is some fun new stuff for hydrus, and I am interested to see where it does well and badly.
misc
In order to be less annoying, the 'do you want to run idle jobs?' on shutdown dialog will now only ask at most once per day! You can edit the time unit under options->maintenance and processing.
Under options->connection, you can now change max total network jobs globally and per domain. The defaults are 15 and 3. I don't recommend you increase them unless you know what you are doing, but if you want a slower/more cautious client, please do set them lower.
The new advanced downloader ui has a bunch of quality of life improvements, mostly related to the handling of example parseable data.
full list
- downloaders:
- after adding some small new parser tools, wrote a new pixiv downloader that should work with their new dynamic gallery's api. it fetches all an artist's work in one page. some existing pixiv download components will be renamed and detached from your existing subs and downloaders. your existing subs may switch over to the correct pixiv downloader automatically, or you may need to manually set them (you'll get a popup to remind you).
- wrote a twitter username lookup downloader. it should skip retweets. it is a bit hacky, so it may collapse if they change something small with their internal javascript api. it fetches 19-20 tweets per 'page', so if the account has 20 rts in a row, it'll likely stop searching there. also, afaik, twitter browsing only works back 3200 tweets or so. I recommend proceeding slowly.
- added a simple gelbooru 0.1.11 file page parser to the defaults. it won't link to anything by default, but it is there if you want to put together some booru.org stuff
- you can now set your default/favourite download source under options->downloading
- .
- misc:
- the 'do idle work on shutdown' system will now only ask/run once per x time units (including if you say no to the ask dialog). x is one day by default, but can be set in 'maintenance and processing'
- added 'max jobs' and 'max jobs per domain' to options->connection. defaults remain 15 and 3
- the colour selection buttons across the program now have a right-click menu to import/export #FF0000 hex codes from/to the clipboard
- tag namespace colours and namespace rendering options are moved from 'colours' and 'tags' options pages to 'tag summaries', which is renamed to 'tag presentation'
- the Lain import dropper now supports pngs with single gugs, url classes, or parsers–not just fully packaged downloaders
- fixed an issue where trying to remove a selection of files from the duplicate system (through the advanced duplicates menu) would only apply to the first pair of files
- improved some error reporting related to too-long filenames on import
- improved error handling for the folder-scanning stage in import folders–now, when it runs into an error, it will preserve its details better, notify the user better, and safely auto-pause the import folder
- png export auto-filenames will now be sanitized of \, /, :, *-type OS-path-invalid characters as appropriate as the dialog loads
- the 'loading subs' popup message should appear more reliably (after 1s delay) if the first subs are big and loading slow
- fixed the 'fullscreen switch' hover window button for the duplicate filter
- deleted some old hydrus session management code and db table
- some other things that I lost track of. I think it was mostly some little dialog fixes :/
- .
- advanced downloader stuff:
- the test panel on pageparser edit panels now has a 'post pre-parsing conversion' notebook page that shows the given example data after the pre-parsing conversion has occurred, including error information if it failed. it has a summary size/guessed type description and copy and refresh buttons.
- the 'raw data' copy/fetch/paste buttons and description are moved down to the raw data page
- the pageparser now passes up this post-conversion example data to sub-objects, so they now start with the correctly converted example data
- the subsidiarypageparser edit panel now also has a notebook page, also with brief description and copy/refresh buttons, that summarises the raw separated data
- the subsidiary page parser now passes up the first post to its sub-objects, so they now start with a single post's example data
- content parsers can now sort the strings their formulae get back. you can sort strict lexicographic or the new human-friendly sort that does numbers properly, and of course you can go ascending or descending–if you can get the ids of what you want but they are in the wrong order, you can now easily fix it!
- some json dict parsing code now iterates through dict keys lexicographically ascending by default. unfortunately, due to how the python json parser I use works, there isn't a way to process dict items in the original order
- the json parsing formula now uses a string match when searching for dictionary keys, so you can now match multiple keys here (as in the pixiv illusts|manga fix). existing dictionary key look-ups will be converted to 'fixed' string matches
- the json parsing formula can now get the content type 'dictionary keys', which will fetch all the text keys in the dictionary/Object, if the api designer happens to have put useful data in there, wew
- formulae now remove newlines from their parsed texts before they are sent to the StringMatch! so, if you are grabbing some multi-line html and want to test for 'Posted: ' somewhere in that mess, it is now easy.
next week
After slaughtering my downloader overhaul megajob of redundant and completed issues (bringing my total todo from 1568 down to 1471!), I only have 15 jobs left to go. It is mostly some quality of life stuff and refreshing some out of date help. I should be able to clear most of them out next week, and the last few can be folded into normal work.
So I am now planning the login manager. After talking with several users over the past few weeks, I think it will be fundamentally very simple, supporting any basic user/pass web form, and will relegate complicated situations to some kind of improved browser cookies.txt import workflow. I suspect it will take 3-4 weeks to hash out, and then I will be taking four weeks to update to python 3, and then I am a free agent again. So, absent any big problems, please expect the 'next big thing to work on poll' to go up around the end of October, and for me to get going on that next big thing at the end of November. I don't want to finalise what goes on the poll yet, but I'll open up a full discussion as the login manager finishes.
What kind of dedication and work does it take to create something like this?
I bet you thought this was a thread made to suck up to Hydrus Dev. HAHA, WRONG!
There's so many things about the Hydrus network that irritate the crap out of me, and it's clear that, as they're deliberate design considerations, they're not going to ever change. Therefore, I want to make my own. Problem is, I haven't really developed shit before, so I just want to know what I'm in for. Maybe I'm mistaken, but I'm under the impression that Hydrus Dev was only starting out when he began creating his as well, so maybe he could offer a few pointers, being already so far into his own project.
Anyway, here's some notes I've been making on how I might do things differently. I imagine Hydrus Dev has already considered a lot of these ideas before, so being more world-weary, and having lost that old shine in his eyes, maybe he could tell me why they're fucking stupid and wouldn't work at all.
What kind of dedication and work does it take to create something like this?
I bet you thought this was a thread made to suck on Hydrus Dev's cock. HAHA, WRONG!
There's so many things about the Hydrus network that irritate the crap out of me, and it's clear that, as they're deliberate design considerations, they're not going to ever change. Therefore, I want to make my own. Problem is, I haven't really developed shit before, so I just want to know what I'm in for. Maybe I'm mistaken, but I'm under the impression that Hydrus Dev was only starting out when he began creating his as well, so maybe he could offer a few pointers, being already so far into his own project.
Anyway, here's some notes I've been making on how I might do things differently. I imagine Hydrus Dev has already considered a lot of these ideas before, so being more world-weary, and having lost that old shine in his eyes, maybe he could tell me why they're fucking stupid and wouldn't work at all.
Pyrus Database
- Store tags and other filesystem-related metadata in two separate databases for full reversibility. Since I consider reversibility so important, I think it deserves this level of security, to be stored in a more stable database of its own which gets written to less frequently and is therefore less prone to breakage. This secondary database will contain all forms of metadata the source filesystems could possibly support; not just the basics like filenames or creation dates, but also comments music tags, or even multi-purpose "description" fields. Users could also create custom keys such as "Likes" for archived YouTube or Twitter posts. Naturally, this level of data retention may seem redundant to many users, so it wouldn't be mandatory.
- Also store the full source paths of programs, or, optionally, just the final directory name. When certain files are "marked", they may all be stored (or at least symlinked) in a single internal directory with all of their original filenames. This would allow certain flash files, and even games or programs to function normally within the database. Naturally, these directories should be excluded when you start the server up. I'm thinking of a hierarchy like this.
db/.
db/everything else
db/symlinks
db/symlinks/(Games/(Visual Novels/))euphoria
db/symlinks/(Programs/)Hydrus Network
* "symlinks" subdirectories up to user discretion, even if they don't seem necessary to me when programs must work with relative paths for this level of abstraction to work anyway.
- No silly client-side case conversions. Come on, I mean we have the full fucking range of the UTF-8 character set here, not 7-bit ASCII. If users really feel like it, then maybe include an OPTION to use this kind of filter. With this and the previous point, some measure of reverse compatability with the Hydrus PTR might be possible, but that sounds like too much of a hassle.
- Since community spirit is important for a federated user participation-based booru project like this, maybe even throw in a third database for user ratings and comments.
Pyrus Client
- Develop a GUI and TUI in tandem, or if it has to be a GUI, at least don't use a terrible library that forces you to look at the hideous default GTK3 theme.
- Don't do anything absolutely fucking stupid like develop a garbage "Media Viewer" implement and integrate it so deeply into the operation of the program that other features must get held back for it.
- Allow users to create custom execs for each filetype, or even for entirely different database variables. If a user doesn't want to define another set of execs for their client, then they can simply write "rifle"/"xdg-open"/"whatever" into each field. Since we've abandoned the Media Viewer, which is used to untangle the messy jungle of a filesystem behind the curtains, we can go even more abstract and create variables for several client-specific factors to be passed as arguments for the exec.
%s might mean all selected files
%a might mean all files (this is dumb and dangerous)
%v might mean all files in the active view
and so on.
For example, and that is if that's the way the program works, an exec binding might look like this:
Bind Control-E:
exec 'Image Viewer.exe' – %v
Which, though it'd make a ridiculously long command, I don't think would have to be particularly RAM or CPU intensive. After all, this is the kind of thing ordinary file managers like Ranger do.
You could get a whole lot more complicated in the long-term by introducing a whole scripting language which can bring up windows in the client itself, or use if statements and the like, but maybe it'd be best to adhere to the KISS philosopy.
So about how stupid are my ideas?
Release Tomorrow!
I had a difficult week, but I got some great work done. The downloader overhaul is entirely complete but for some help. Tomorrow's release has a variety of unusual new features, such as subscriptions without popups, subscription 'quality' review, improved easy-import download pngs, more download logic controls, improved file system healing, and more.
The release should be as normal tomorrow.
Version 316
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v316/Hydrus.Network.316.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v316.tar.gz
I had an excellent week. I was able to cleanly 'do' the big gallery overhaul, resulting in many changes. This is an important release for anyone who does a lot of downloading.
gallery update
So, the meat of the gallery overhaul is complete! Where available, subscriptions, the gallery downloader, and the url downloader will now use the new parsing system to parse 'thumbnail pages'. The new pipeline has numerous improvements behind-the-scenes and is another important step in allowing users to create and share their own downloaders for new sites.
I won't go into all the details here, but anyone who has poked around the new parsing system's advanced dialogs will now see that gallery pages are now featured.
The important change for most users is that the main gallery downloader page is now a 'multi-gallery' page. It looks much like the multi-watcher for thread-watching, where the one page manages multiple queues simultaneously. Gallery download pages now list every query you enter separately and work on them simultaneously, reducing overall ui load significantly and making it much easier to pause one query without affecting others, spot queries that return 0 results, avoid big chokes and delays caused by larger queues, and manage the results you finally get as different pools. The old single-download page can no longer be created–your existing gallery pages will be updated to multi-pages on update (your existing queues will survive as generic 'updated from' entries, but everything will start paused, so you'll want to go through them, verify it all looks good, and then unpause when you are ready).
And like the multi-watcher, gallery pages can have a 'highlighted query'. Double-clicking on a list entry will show its current thumbnails in the media panel and show its full details in a panel below the main list. This new panel is very similar to the old gallery page's ui and lets you see downloads as they happen and inspect the file import queue and gallery log and so on in more detail.
Gallery download pages are now also downloader-agnostic. You open them just as 'gallery' from the F9 new page popup (there's also a new shortcut for them, under the 'main_gui' set), and they have a new 'gallery selector' button that lets you change the source for the next queries you enter. The gallery selector means that one gallery page can handle queries for multiple sites simultaneously! This new button will always default as 'deviant art' for now, but when I do the last stage of this downloader overhaul–the 'searcher' step–I will be able to add some reasonable 'favourite' options here.
Subscriptions can also use the new gallery parser, and they have the neater 'gallery selector' as well. You shouldn't notice any difference with how they operate–they should just run as before. But after some thought, I have reduced subscriptions' maximum 'first time' and 'regular' file limits to 1000 and written some help on the edit panel to explain this change. tl;dr is: use download pages to do big one-time syncs, use subs to keep up with new stuff. If this 1000 change affects you, please check out the help and feel free to ask me about it.
URL downloader pages also support parsable gallery urls. They only do the single page you add (i.e. they don't start a whole new multi-page search, but this capability will come), but they'll suck up any drag and drop for any working parser. For this week, I've written gallery parsers for danbooru, gelbooru (0.2.0 and 0.2.5), and e621 to get us started. Like with 'file pages', I will roll out more gallery parsers over the coming weeks until all default downloaders are using the new system.
A lot has changed. I have done a whole bunch of testing here and I am overall happy with it, but if you encounter problems with an unusual query or site or anything else, please let me know. If you are the sort of user with 200k items waiting to download in your current session, I strongly recommend you ensure you have a good backup before you update. This multi-gallery page also makes it easier than ever to create a session with lots and lots of files, but I will reiterate that the client tends to get laggy at 50k+ items, so take it easy and make a schedule for big stuff if you can!
There is a little more work to do here to support some unusual situations like tumblr, but the big hump of the downloader overhaul is now done. The 'searcher' change will be much simpler (it is basically just converting samus_aran -> http://safebooru.org/index.php?page=post&s=list&tags=samus_aran and some ui to manage it) and then it'll just be ui cleanup and figuring out an easy-mode share object, and redoing some help, and then we can talk about what we want out of a login manager (including whether it should be put off for something else).
multi-watcher
The multi-watcher also has a new 'highlight' panel just below its list. In a similar way, it has similar ui to the old single-page watcher and lets you inspect the file list and gallery (thread) log. Single watcher pages are removed this week, so all existing single watchers will be updated to multi-watchers. A variety of related single-watcher 'page naming' options and so on are also removed. Rather than be redundantly complicated, hydrus now just has 'url downloader', 'watcher', 'gallery', and 'simple downloader'.
full list
- gallery:
- gallery url classes can now be linked to parsers!
- if parsers are linked, gallery pagewalk can now work on the new parsing system. gallery import pipeline has been significantly updated to reflect this
- gallery import objects are now 'multiple' gallery imports, much like the multi-watcher, with each separate query having its own entry in a list (they also run in parallel!)
- the multi-gallery list will show file/gallery pause status in slender columns, and will show a 'stop' character when gallery parsing is done
- wrote a 'gallery selector' button and added it to the new multi-gallery page, so you can spawn queries for ~different sites~ on the same import page! it always defaults to 'deviant art' for now, but when the next 'searcher' overhaul step is done, this will be customisable
- the new page selector and related 'pages' menu is now simpler–with the new selector, you just select 'gallery'
- added 'new_gallery_downloader_page' shortcut action to the 'main gui' set to allow quick opening of this new page type
- wrote a 'gallery import panel', which reviews a single gallery import stream, and added it to the multi-gallery page to show the current highlighted query
- as all gallery imports now run in parallel and work on the new system, the now almost-useless 'cancel' gallery pagewalk button is now removed
- with the wider availability of the new gallery log for file count and error reports, shifted around and smoothed out some gallery status text presentation
- improved the auto url_class->parser linking 'try to fill in gaps' logic to work with gallery urls (this was surprisingly complicated)
- fixed a misc stupid waste of time in auto url_class->parser linking
- many misc updates to gallery pipeline
- .
- subscriptions:
- wrote a new gallery pagewalk pipeline for subscriptions, which still does oldest-to-newest url addition and obeys file limits and so on
- numerous subscription pipeline and error handling tweaks and improvements, particularly in regards to the new code
- subscriptions now have max initial and periodic file limits of 1000. existing subs with >1000 or infinite will be cut to 1000. there is a help button on the edit sub panel to explain why you should do large syncs with the manual downloader and not subs
- the ugly and dangerous-if-you-scroll-in-the-wrong-place gallery selector mismash control in the edit subscription panel is now replaced with the new gallery selector button
- fixed an issue where the edit subs panels could sometimes say '48 years ago' (i.e. displaying a literal time delta since 0, 1970) on initial timestamps
- juggled some 'periodic limit' reporting logic to skip an unusual false positive that affect hentai foundry subs for now and more in future
- .
- urls:
- the url downloader now accepts gallery urls and will receive drag-and-dropped gallery urls. at the moment, it only parses the one page (i.e. it doesn't start a new 'searching' pagewalk) and sends the parsed links to its file queue
- .
- watcher:
- finished 'watcher panel', which reviews a watcher, and added it to the multi-watcher page to show the current highlighted watcher
- the single watcher page is completely removed–it is only the multiple watcher now. all singles will be converted to multiples on update
- some single-watcher options (like watchers naming their own page tabs and the [404]-style page name prefixes) are removed
- multiple watcher panel now lists file/checking pause status and has separate buttons to control these paused statuses
- fixed some misc watcher highlight code–highlighted watchers should correctly publish to the page from the start of session load now
- improved some 'repage' logic in how highlighted threads get removed
- .
- misc:
- discovered a scroll-setup parameter that stops janky scroll-to-click-focus behaviour on all the new scrolling panels, thank the LORD
- improved some 'can't parse' error handling for post url parsing
- reworked how all importers present their network jobs to the ui, including fast response if the switch happens during a job
- in prep for searcher switchover where all downloader sources will be harmonised into one system, booru identifiers now present in several ui locations as 'name', not 'booru: name'
- updated the danbooru parser to deal with the new 'next page' markup they use
- wrote a gelbooru gallery parser that works with 0.2.0 and 0.2.5 gelb sites–an ancillary issue where gelb-related downloaders could sometimes not accurately figure out the magic '42' next-page offset is hence now fixed
- wrote an e621 gallery page parser
- these sites hence now support single-page gallery drag-and-drop
- added url classes for artstation gallery url and its api counterpart, but didn't go further yet–we aren't quite there with api pagewalking just yet
- updated deviant art gallery url classes
- added an e621 'search initialisation' gallery url class to improve some future drag-and-drop stuff
- url normalisation no longer cuts off 'www.'-style prefixes
- url comparison is more careful to test 'www.'-style prefixes, so a file import cache should recognise that 'http://www.blah.com/blah' is the same as 'https://blah.com/blah'
- did a bunch of refactoring to further split up the bloated ClientImporting.py
- fixed some misc downloader layout that may have been hiding some texts previously
- some multi-watcher and multi-gallery events like add/pause query should be a bit snappier
- in the parsing ui, url and title priorities are now 50 by default
- prepped a little subscription unit test code for when searcher object is done
- misc downloader layout improvements
- misc listctrl refactoring
next week
A lot of stuff I overconfidently said "I will check this this week" about has been left in the dust of this unexpectedly large job. I want to spend a week focusing on this smaller stuff and catch up back to regular schedule.
Version 323
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v323/Hydrus.Network.323.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v323.tar.gz
I had a good week. I finished a decent first version of the downloader easy-import and did some other stuff.
easy import
There is now a simple new dialog under network->downloaders->import downloaders that lets you import new downloaders just by dragging and dropping an encoded png onto it. It tries to do all the semi-complicated comparison and association work to actually build a new downloader clientside automatically.
Another dialog, under network->downloader definitions->export downloaders, is for advanced users who are comfortable with the new downloader system to put these new pngs together. These pngs should be shareable in any way any other png image would be–you can post one in a thread, or in a program like discord, or to a longer-term storage.
I've written some basic help here for both sides:
https://hydrusnetwork.github.io/hydrus/help/adding_new_downloaders.html
https://hydrusnetwork.github.io/hydrus/help/downloader_sharing.html
In the coming weeks, I suspect the advanced users will start to post these easy pngs in one of these locations:
https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System
But maybe we'll start an 8ch thread for them as well. I am not sure what the details of the new system's overall workflow will actually be–e.g. will we need to update some parsers every few months, or will almost all of them last ten years? are there a hundred sites people want parseable, or ten thousand?–so please let me know your feedback. I could see a future iteration of this system having version control and an 'rss feed'-like way of automatically checking for and updating downloader data, but this first one is scrappy pngs on imageboards.
I'll post an example importable png in the 8chan release thread so you can try the import out right now.
While I am interested in helping users learn how to create new downloaders, and I'll fold some good new defaults into the client, I do not plan to write a significant number of these new downloaders. Now the system is completely user-editable, I am free to worry less about the minutiae. Please let me know how the system falls short, and I can put my time more into that.
misc
The deadlocked network issue users with extremely busy clients (200+ simultaneous downloads in the queue) were seeing should be fixed. I used a semi-hacky solution, basically culling max active jobs to 50 or so with a new check, so if you are one of these heavy users, please let me know how you get on. If you find a new bandwidth-based deadlock or your CPU usage spikes, I can revisit this.
The watcher and gallery downloader pages now support right-click on their lists! I've thrown in some simple stuff, like pause/play and copy url/query text, but let me know what else you'd be interested in doing here en masse. I expect I'll add a 'retry failed' next week.
Derpibooru gets a new 'no filter' search entry this week, which should search for all content (rather than just that allowed by their default filter, as a user helpfully let me know this week). The solution I am rolling out today appears to work, but if you find it gets confused and falls back to the default in some situations, let me know and I'll do a more technical solution that we have more confidence in.
Gfycat should also support file page drag and drop import! Just drag from your browser's address bar to hydrus, and it should grab an mp4.
The issues with unusual tag sorting in the media viewer/manage tags dialog should all be fixed! All tag lists are now sorted with the same code, and incidence-based default sort will still propagate up to the media viewer correctly.
full list
- wrote first version of the new downloader easy-import drop-panel. you drop downloader-encoded pngs on it, and it maybe asks you a question and jumbles its way through auto-importing all the required data to the client
- extended this file import to do some cleverer 'example url merging' when parsers are otherwise dupes, rather than spamming similar dupes on import
- wrote first version of the new downloader export panel. it takes gugs, url classes and parsers, and predicts sensible sub-objects to include to make functional downloaders, and bundles it into one png
- fleshed out help for the new easy import/export system
- the client now slows down gallery and watcher processing when the network engine is under heavy load, aiming for no more than 50 jobs in system at once. the solution is a bit hacky for now, but it should alleviate the deadlock issue when there are ~180+ simultaneous gallery/watcher network jobs pending
- the multi-watcher panel's list of watchers now supports right-click menu to copy/open urls and pause/play files/checking
- the multi-downloader panel's list of downloaders now supports right-click menu to copy query texts and pause/play files/searching
- added a 'derpibooru tag search - no filter' GUG that disables the default derpi no-explicit-files rule
- added basic gfycat support to default client–drag and drop any typical video page, and it should import ok
- fixed the canvas/hover window tag sorting discrepancy–all tags are now sorted with the same code, and the media view sort order should be the same as your default sort order (although in this case incidence has no effect as there are no tag counts)
- rewrote the network job control's cog menu to be a bit more dynamic, and added 'override gallery slot requirements for this job' if appropriate
- fixed a stupid typo bug in the shutdown maintenance jobs test code that was causing pending repository work to not report right
- fixed gallery searches that include unicode characters that end up in the path of the url (rather than the query parameters)
- fixed an issue where highlighting a watcher would unpause its checking
- generalised the way the new listctrl class can produce right-click menus
- fixed some api link calculation that was over-prescribing api link display pairs (this affected the artstation file page url class by default). these pairs are now also sorted in the links dialog
- misc png-export improvements to present better with the new easy import/export stuff
- the summary texts in the tag filter panel now ellipsize (…), so if the tag filter is complicated, it won't try to boot a superwide edit panel!
- the manage subscriptions panel now correctly initially sorts in a case-insensitive way (previously, it was usually sorting A-Za-z, which is different to regular aA-zZ resorting behaviour, so it always sort-flickered after the first edit)
- the status bar has a new segment for reporting when the client is 'busy' with different jobs. for most typical usage, it'll just stay blank. let's see how it goes.
- fixed mr. bones's wild review when the client currently has no files
- punched up the new file report mode to specify full paths where available
- improved some misc downloader code
next week
If the downloader easy-import fails in certain common ways, I'll give it a second pass. Otherwise, I'll be focusing on triaging the final downloader overhaul jobs.
Release Tomorrow!
I had a great week. As well as some misc work, I churned through about half of the remaining downloader overhaul jobs (things like better ui and options for max jobs per domain), fixed the pixiv downloader, and wrote a prototype twitter username downloader.
The release should be as normal tomorrow.
Version 322
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v322/Hydrus.Network.322.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v322.tar.gz
I had a good week. I took it a little easy after pushing to get over the last big overhaul hump last week. There's some fixes, new help and interesting misc work.
downloader help and fixes
I am thankful and relieved that last week's big shift seems to have basically gone well. There were a couple of bugs–artstation and tumblr were only searching the first page of their galleries, and the new GUG object had some unclean ui in places–but these are fixed today. Working with that code without having to deal with the old legacy system has been great.
I've also significantly extended and polished the help here:
https://hydrusnetwork.github.io/hydrus/help/downloader_intro.html
If you are interested in writing your own downloader for hydrus, please check it out. It is not short or trivial–it is a powerful system!–but I've tried to make it fairly comprehensive. Let me know if anything is too confusing or worded too complicated. The only bit left to do is 'how to easy-share a downloader' because I haven't written that system yet!
And I've added some right-click->try again/skip to the gallery logs for Gallery and URL Downloaders. This is semi-experimental, so if you are an advanced downloader and I've been talking to you about 'resuming' a failed download, please give it a go and let me know how it works IRL. Subscriptions and Watchers work on different gallery sync logic, so these downloaders' gallery logs remain 'read-only' for now.
misc
You can now put '\' (or '/' for Linux) in file export phrases! It is now easy to export well-tagged files to complicated folder structures, including those based on namespace/tag!
File drag and drops started from thumbnails in hydrus now begin with significantly less lag, particularly for very large drops.
The 'would you like to do maintenance on shutdown?' dialog now lists a brief summary of the expected jobs. I could improve the descriptions further, so let me know what you think.
Tags across the program now (lexicographically) sort according to the new and improved human number sorting method.
Thanks to the same user who added the 'cookies.txt' import the other day, advanced users can now mass-set collected thumbnail groups as alternates in the duplicate system. Please be careful with this–the duplicate system is still not good at grouping together larger numbers of files. There's also a duplicate_media_set_alternate_collections media shortcut action to go with it.
I've added new debug tools at help->debug->report modes->file report mode and ->data actions->review threads. The review current network jobs under networking should also present some statuses better. If I've been talking with you about these tools, please try them r.e. your problems and we'll see if we can figure anything new out.
full list
- wrote gugs help
- gave url classes and parsers help a pass
- wrote e621 html gallery page example help
- wrote gelbooru html file page example help
- wrote artstation json file page example help
- wrote url class links help
- gallery logs for the gallery downloader and url downloader now support 'try again' and 'skip' right-click menu for gallery log entries. the try again allows just the one page or also allowing search to continue) so, if a gallery query fails for some reason, you can now try again/continue where it broke. subs/watcher/simple downloader work on more complicated gallery search logic, so their gallery logs will remain read-only for now
- all gallery log buttons now support right-click menu to mass-export urls to png or clipboard. non read-only also support import
- fixed an issue with gallery searches that rely on both api url conversions and url class next gallery page urls (I think just artstation and tumblr by default) not generating the next page url correctly
- improved some misc gallery url processing logic
- fixed some issues with gallery url generators with invalid example urls causing problems opening the edit gug and gallery selector panels
- fixed an issue where you could only delete a gug if it was in an ngug, ha ha
- thanks to a different submission by prkc on the discord, collections now have a _right-click->set collections as groups of alternates_ duplicate action (note the duplicate menu only appears in advanced mode). the related shortcut action duplicate_media_set_alternate_collections is also added
- export phrases now support '\' ('/' in linux) in the path export phrase in order to create folders. you should also be able to do \[series]\ to create optional namespace folders. slashes in tags will still be replaced with _
- to stop the client sometimes doing laggy vacuum checks every maintenance cycle, vacuums that cannot occur due to limited disk space now will still count as 'done' for the purposes of rescheduling
- added 'file report mode' to the help debug menu. This will spam popups as file and thumbnail actions are requested
- tightened up some network job status setting to help us debug the 'there are a ton of jobs in network engine, but the three active on this domain seem stalled' issue
- wrote a simple 'review threads' panel under help->debug->data actions->review threads. I knocked it together in about ten minutes, and it's likely unstable as hell, but it's pretty neat!
- some instances where many file paths are copied quickly (exporting paths to clipboard and drag and drop) no longer do a safety check for file existence, so should be much faster to go. this particularly reduces startup lag for large file drag and drops!
- the 'would you like to do maintenance work in this shutdown?' dialog now lists a summary of what it thinks it'll be working on. I _could_ make this more detailed, so let me know how it works for you
- tags with numbers should now sort according to the new improved human sorting method–it now shouldn't matter where the numbers are in the tag–as long as the text-and-number-breaks lines up with another tag, they'll be compared each part in turn correctly
- fixed some human sorting code for unusual number characters like ⑨. they will be treated as text, not a number, for now
- misc fixes
next week
This easy-share system for the downloader is probably top priority, and some last misc downloader work–cleaning up old code, adding tests, workflow and ui improvements, that sort of thing.
With the downloader winding down, I am thinking about what to do next. I would like to get the python 3 update in before the end of the year, so at present, my current plan is to spend a few weeks finishing downloader, spending x weeks on a simple login manager, taking a month off for python 3 attempt, and then working on The Next Big Thing™. I am pretty sick of the download engine and don't want to get bogged down in another quest to write an amazing login engine that can cook you breakfast, so I would like to know what sites you would like to log into with the new download engine so I can draw an appropriate line on how complicated I should go. I know FurAffinity is highly requested, and I'll be converting the existing Pixiv and Hentai Foundry login stuff to something less hacky, but what else would you like? If the login is strange, how is it strange? If you know, what kind of cookies/headers/login process are involved? Would you rather I just make it easier to import cookies.txt from your browser? I don't use sad panda, so can someone who does summarise precisely what cookies/domains/login order/whatever is going on there, and what you suggest would be the best way to automate that? All input on this whole subject would be appreciated.
Release Tomorrow!
I had a good week. I fixed several bugs from network problems to the tag sorting issue, added some small new features, added simple gfycat support and a derpibooru no-filter downloader, and finished the first version of the downloader easy-import system.
I have a very full day tomorrow, so the release may be a bit late.
Version 321
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v321/Hydrus.Network.321.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v321.tar.gz
I had a great week. An important milestone in the downloader overhaul is done. Users who use 'custom' boorus will want to read this post and may like to put off updating a week or two.
downloader overhaul
So, this week I have switched over the old hardcoded method of converting search text into an initialising gallery url to the new user-customisable and -shareable GUG (Gallery URL Generator) objects. This was the last hardcoded component I needed to replace, and so I was finally able to disconnect and delete all the old downloader code. We are now entirely running on the new customisable and shareable system!
Regular users should not see much different today. Your subs and galleries should switch over to the new 'gallery selector' values automatically. Nothing is too functionally different, but a bunch of the behind the scenes is completely new and much neater all around. The manage boorus dialog, which was the old way of cobbling together some user-customisation, is also gone–all sites now run on the same new network submenus.
I've added new downloaders for artstation and derpibooru and brought back the newgrounds artist lookup (which works better than you might think!). Now we are completely on the new system, I expect to roll out more new searchable sites in future updates myself and other users to start creating and sharing complete downloaders themselves. In the coming weeks, I will write an easy one-action way for regular users to import these new downloaders (at the moment, it is still a little ugly and technical to do and set up).
Needless to say, while my tests have gone well, this is a big update, so make sure you refresh any backup before you update, and let me know if you have any problems.
Now the new downloader is 'out' and its functions and terms are pretty much fixed, I will also be finishing the 'how to write a downloader' help, which will be a comprehensive guide on the whole new system.
custom boorus
Any 'custom' boorus (i.e. ones you created yourself or imported via the manage boorus dialog) will unfortunately not be completely automatically updated. There is not quite enough information in the old booru object to generate all the parts I need in the new system (I was able to generate GUGs and gallery/post parsers), so you will have to add/do some things to get them working again.
If you are comfortable in the new system, you will need to:
-Get a gallery and post url class for the booru.
-Double-check (and rename, if you want) the auto-generated GUG and parsers.
-Link the url classes to the parsers.
-Run a test search on the new downloader.
-Update the downloader selection in the downloader or subscription's gallery selector button.
If you are not comfortable creating url classes but are ok with importing and linking them, I recommend you check the github site here:
https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/NEW%20Download%20System
–for url class updates, and if what you want is missing, drop requests here or in the discord, and I or another advanced user will see what we can do.
If you are completely lost but you still want to use your custom boorus, I recommend you put off updating this week. You can either find a convenient time to learn how to do the above steps (I am happy to help if you run into trouble), or you can wait a few weeks for me to bring out this 'downloader easy-import', which will make it as simple as opening one dialog and dropping a png file from the above site on it.
Note that if you used a custom derpibooru booru, since I've added explicit functionality this week, all you have to do is the final step above: go to your derpi gallery downloaders or subscriptions and change the selected gallery from the broken entry, which will be something like 'derpibooru: not found' to the new one 'derpibooru tag search' and you should be going again. You might still like to run a test download if you use more than one tag–the GUG I made uses their search format, which preserves spaces but requires a comma to separate multiple tags, like 'rainbow dash,straight'.
misc
I put some time into the native video renderer this week, cleaning up the pipeline and how the different components talk to each other. It has cleared out a bunch of jitter and scanbar calculation jank. I think I will put some more time into it, but overall I am happy with the changes. Please let me know if you run into any particular trouble with certain files.
Also, before doing client processing, repository services will now check for a little additional free space (now half the size of your client.mappings.db file, rather than a flat 1GB). If they lack the space, they'll throw up a popup and safely pause the repo. Since a PTR client.mappings.db is now about 8.5GB, please make sure your hydrus drive (and system drive, for its temp folder!) have about 5GB free at least!
full list
- downloader overhaul:
- the basic downloader overhaul is complete! at this point, any user can create and share the objects required for a completely new downloader! it is still rough in some places, so a round of EZ-import is coming to make adding new downloaders a single easy drag and drop action
- rounded out the ngug (nested gugs, which contain multiple gugs) code
- updated the edit gug panel to deal with gugs and ngugs on different notebook pages
- added a bunch of logic to this panel and backend data handling to deal with missing gugs in ngugs
- if an ngug cannot find a gug by its internal identifier key, it will now attempt to fallback to its simple name, and will silently fail if no gug can be found. all gug tracking now uses this 'key first, name later' id method, so downloaders and subs should generally survive gug renames and same-name overwrites
- the gallery selector now works in gugs. it has two 'pages', depending on which gugs are set to 'display', and will note if the chosen gug is cannot be found in the current definitions. the gugs have slightly more specific names ('gelbooru tag search', 'hentai foundry artist lookup', etc…) than before
- the gallery selector also puts 'non-functional' gugs (i.e. those with no parsable gallery url class) to a third page
- moved the gallery downloader gallery and file pipeline completely over to the new system
- the gallery downloader will now bundle nested gugs (like hentai foundry artist, which searches both works and scraps) into a single downloader
- moved the subs gallery and file pipeline comppletely over to the new system
- the subs gallery sync now handles nested gugs (like hentai foundry artist, which searches both works and scraps) in an interleaved manner and make behind-the-scenes checking decisions in a clearer and more logical way
- subs should now make correct 'hit limit' stop reason reports and not generate new gallery pages when the current page has exactly enough results to hit the current file limit
- artstation artist lookup is now available as a default downloader
- newgrounds artist lookup makes a triumphant return. it works pretty well, given how flash and NG has changed since
- derpibooru tag lookup is now available as a default downloader. due to unusual search syntax on derpibooru, please enter queries exactly as you would on derpi, using ',' or ' AND ' to separate tags (such as 'rainbow dash,straight')
- pixiv now has multiple artist lookup options–either images, manga, ugoira (doesn't work yet!), or everything
- the old downloader code is deleted!
- the old manage booru dialog is deleted!
- 'custom' boorus (i.e. new ones you created or imported to 'manage boorus'), cannot be completely automatically updated to the new system. I've figured out a way to generate new gugs and gallery&post parsers, but they will miss url classes to get working again. your custom-booru subs will notice this and safely pause until the issue is fixed. if you rely on custom boorus, please check the release post for info on this–you might like to put off updating
- many misc changes and fixes to gugs and overall gallery url handling pipeline
- some misc refactoring and concept-renaming in gallery pipeline r.e. gugs
- when the downloader tries to import what looks like a raw html file, its error notes will specify this and suggest a parser may be needed
- moved the 'media viewer url display' options panel from the manage url match links dialog to the new network->downloaders->manage downloader and url display
- this new dialog also hosts a list for managing which downloaders to show in the first list of the downloader selector
- .
- misc:
- gave the video rendering pipeline communication logic a quick pass, cleaning up a bunch of bad code and other decisions. the video renderer should be quicker to respond to various changes in scanbar position, and incidences of the frame buffer suddenly sperging out (usually inexplicably falling behind the current frame position or deciding to regen for no apparent reason) should be greatly reduced if not completely eliminated
- the test that stops repository processing if there is not enough disk space now uses half the current size of client.mappings.db for its estimate (previously 1GB) and also tests temp folder location free space (just as the vacuum test does) and reports this nature of the error along with pausing the repo, stopping further attempts
- might have fixed another out-of-order dialog close/open event combination during manage tags close->advanced content update open
- fixed gallery queries that include '/' (or some other unusual characters) that end up in the 'path' of the url (as opposed to the query). this fixes 'male/female' on e621, for instance
- 'advanced mode' users now have a 'nudge subs awake' menu entry below 'manage subs'. this simply wakes the subs daemon (which usually only checks once every four hours), in case any subs are due
- 'db report mode' now reports every db job as it comes in (formerly, it only reported some optimisation esoterica). this makes it a more lightweight version of 'db profile mode' for several debugging tasks
- fixed a tiny issue in fetching the 'how boned am I?' stats when the user had zero inbox/everything count
- fixed a typo in the default new url class object that was breaking the edit ui panel
- highlighted the quiet filename tagging options on edit import folder panel
next week
I'd like a simple week. I'll deal with anything that comes up due to this week's big changes, but otherwise I'd like to catch up on help docs and just do something easy and interesting.
Release Tomorrow!
I had a good week. I fixed some bugs in the new download pipeline (including the artstation/tumblr short-search issue), caught up on a bunch of unusual small jobs, and finished the bulk of the 'how to write a downloader' help docs for those users interested in trying it out.
The release should be as normal tomorrow.
Tag parser script
Here's something I did to make copying individual booru tags easier. hydrus is good with booru browser but its page downloader doesn't read tags. And sometimes you just wanna quickly add tags to an individual image.
So this tamper/greasemonkey script will add a button and a key bind to booru posts and the Illustration2Vec demo site to copy all tags on the page including their schemas and rating. Then you can just right click your image in hydrus > manage file tags > paste tags > apply. The whole process takes about 5 seconds.
github.com/JetBoom/boorutagparser
Important Release Tomorrow!
I had a great week. A big milestone in the downloader overhaul is done: all downloads, from start to finish, now happen on the new system's code. Any advanced user can create and share a completely custom downloader for any site the new parser can figure out. I am really pleased with it.
There are some final important objects to update. Most users will hopefully experience no big changes, but anyone who uses a custom booru (that they created or imported through the manage boorus dialog) will want to read the release post before they update. There will be nice new downloaders for Artstation and Derpibooru, and the Newgrounds downloader makes a return.
I've got a bit more to test, but the release should be as normal tomorrow.
Version 320
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v320/Hydrus.Network.320.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v320.tar.gz
I had a great week. The downloader overhaul is in its last act, and I've fixed and added some other neat stuff. There's also a neat hydrus-related project for advanced users to try out.
Late breaking edit: Looks like I have broken e621 queries that include the '/' character this week, like 'male/female'! Hold off on updating if you have these, or pause them and wait a week for me to fix it!
misc
I fixed an issue introduced in last week's new pipeline with new subs sometimes not parsing the first page of results properly. If you missed files you wanted in the first sync, please reset the affected subs' caches.
Due to an oversight, a mappings cache that I now take advantage of to speed up tag searches was missing an index that would speed it up even further. I've now added these indices–and your clients will spend a minute generating them on update–and most tag searches are now superfast! My IRL client was taking 1.6s to do the first step of finding 5000-file tag results, and now it does it in under 5ms! Indices!
The hyperlinks on the media viewer now use any custom browser launch path in options->files and trash.
downloader overhaul (easy)
I have now added gallery parsers for all the default sites hydrus supports out the box. Any regular download now entirely parses in the new system. With luck, you won't notice any difference, but let me know if you get any searches that terminate early or any other problems.
I have also written the new Gallery URL Generator (GUG) objects for everything, but I have not yet plugged these in. I am now on the precipice of switching this final legacy step over to the new system. This will be a big shift that will finally allow us to have new gallery 'seachers' for all kinds of new sites. I expect to do this next week.
When I do the GUG switch, anything that is supported by default in the client should switch over silently and automatically, but if you have added any new custom boorus, a small amount of additional work will be required on your end to get them working again. I will work with the other parser-creators in the community to make this as painless as possible, and there will be instructions in next week's release post. In any case, I expect to roll out nicer downloaders for the popular desired boorus (derpibooru, FA, etc…) as part of the normal upcoming update process, along with some other new additions like artstation and hopefully twitter username lookup.
In any case, watch this space! It's almost happening!
downloader overhaul (advanced)
So, all the GUGs are in place, and the dialog now saves. If you are interested in making some of your own, check what I've done. I'm going to swap out the legacy 'gallery identifier' object with GUGs this coming week, and fingers-crossed, it will mostly all just swap out no prob. I can update existing gallery identifiers to my new GUGs, which will automatically inherit the url classes and parsers I've already got in place, but custom boorus are too complicated for me to update completely automatically. I will try to auto-generate gallery and post url parsers, but users will need GUGs and url classes to get working again. I think the best solution is if we direct medium-level users to the parser github and have them link things together manually, and then follow-up with whatever 'easy import' object I come up with to bundle downloader-capability into a single object. And as I say above, I'll also fold in the more popular downloaders into some regular updates. I am open to discuss this more if you have ideas!
Furthermore, I've extended url classes this week to allow 'default' values for path components and query parameters. If that component or parameter is missing from a given URL, it will still be recognised as the URL class, but it will gain the default value during import normalisation. e.g. The kind of URL safebooru gives your browser when you type in a query:
https://safebooru.org/index.php?page=post&s=list&tags=contrapposto
Will be automatically populated with an initialising pid=0 parameter:
https://safebooru.org/index.php?page=post&pid=0&s=list&tags=contrapposto
This helps us with several "the site gives a blank page/index value for the first page, which I can't match to a paged URL that will then increment via the url class"-kind of problems. It will particularly help when I add drag-and-drop search–we want it so a user can type in a query in their browser, check it is good, and then DnD the URL the site gave them straight into hydrus and the page stuff will all get sorted behind the scenes without them having to think about it.
I've updated a bunch of the gallery url classes this week with these new defaults, so again, if you are interested, please check them out. The Hentai Foundry ones are interesting.
I've also improved some of the logic behind download sites' 'source url' pre-import file status checking. Now, if URL X at Site A provides a Source URL Y to Site B, and the file Y is mapped to also has a URL Z that fits the same url class as X, Y is now distrusted as a source (wew). This stops false positive source url recognition when the booru gives the same 'original' source url for multiple files (including alternate/edited files). e621 has particularly had several of these issues, and I am sure several others do as well. I've been tracking this issue with several people, so if you have been hit by this, please let me if and know this change fixes anything, particularly for new files going forward, which have yet to be 'tainted' by multiple incorrect known url mappings. I'll also be adding some 'just download the damned file' checkboxes to file import options as I have previously discussed.
A user on the discord helpfully submitted some code that adds an 'import cookies.txt' button to the review session cookies panels. This could be a real neat way to effect fake logins, where you just copy your browser's cookies, so please play with this and let me know how you get on. I had mixed success getting different styles of cookies.txt to import, so I would be interested in more information, and to know which sites work great at logging in this way, and which are bad, and which cookies.txt browser add-ons are best!
a web interface to the server
I have been talking for a bit with a user who has written a web interface to the hydrus server. He is a clever dude who has done some neat work, and his project is now ready for people to try out. If you are fairly experienced in hydrus and would like to experiment with a nice-looking computer- and phone-compatible web interface to the general file/tag mapping system hydrus uses, please check this out:
https://github.com/mserajnik/hydrusrvue
https://github.com/mserajnik/hydrusrv
https://github.com/mserajnik/hydrusrv-docker
In particular, check out the live demo and screenshots here:
https://github.com/mserajnik/hydrusrvue/#demo
Let him know how you like it! I expect to write proper, easier APIs in the coming years, which will allow projects like this to do all sorts of new and neat things.
full list
- clients should now have objects for all default downloaders. everything should be prepped for the big switchover:
- wrote gallery url generators for all the default downloaders and a couple more as well
- wrote a gallery parser for deviant art–it also comes with an update to the DA url class because the meta 'next page' link on DA gallery pages is invalid wew!
- wrote a gallery parser for hentai foundry, inkbunny, rule34hentai, moebooru (konachan, sakugabooru, yande.re), artstation, newgrounds, and pixiv artist galleries (static html)
- added a gallery parser for sankaku
- the artstation post url parser no longer fetches cover images
- url classes can now support 'default' values for path components and query parameters! so, if your url might be missing a page=1 initialsation value due to user drag-and-drop, you can auto-add it in the normalisation step!
- if the entered default does not match the rules of the component or parameter, it will be cleared back to none!
- all appropriate default gallery url classes (which is most) now have these default values. all default gallery url classes will be overwritten on db update
- three test 'search initialisation' url classes that attempted to fix this problem a different way will be deleted on update, if present
- updated some other url classes
- when checking source urls during the pre-download import status check, the client will now distrust parsed source urls if the files they seem to refer to also have other urls of the same url class as the file import object being actioned (basically, this is some logic that tries to detect bad source url attribution, where multiple files on a booru (typically including alternate edits) are all source-url'd back to a single original)
- gallery page parsing now discounts parsed 'next page' urls that are the same as the page that fetched them (some gallery end-points link themselves as the next page, wew)
- json parsing formulae that are set to parse all 'list' items will now also parse all dictionary entries if faced with a dict instead!
- added new stop-gap 'stop checking' logic in subscription syncing for certain low-gallery-count edge-cases
- fixed an issue where (typically new) subscriptions were bugging out trying to figure a default stop_reason on certain page results
- fixed an unusual listctrl delete item index-tracking error that would sometimes cause exceptions on the 'try to link url stuff together' button press and maybe some other places
- thanks to a submission from user prkc on the discord, we now have 'import cookies.txt' buttons on the review sessions panels! if you are interested in 'manual' logins through browser-cookie-copying, please give this a go and let me know which kinds of cookies.txt do and do not work, and how your different site cookie-copy-login tests work in hydrus.
- the mappings cache tables now have some new indices that speed up certain kinds of tag search significantly. db update will spend a minute or two generating these indices for existing users
- advanced mode users will discover a fun new entry on the help menu
- the hyperlinks on the media viewer hover window and a couple of other places are now a custom control that uses any custom browser launch path in options->files and trash
- fixed an issue where certain canvas edge-case media clearing events could be caught incorrectly by the manage tags dialog and its subsidiary panels
- think I fixed an issue where a client left with a dialog open could sometimes run into trouble later trying to show an idle time maintenance modal popup and give a 'C++ assertion IsRunning()' exception and end up locking the client's ui
- manage parsers dialog will now autosort after an add event
- the gug panels now normalise example urls
- improved some misc service error handling
- rewrote some url parsing to stop forcing '+'->' ' in our urls' query texts
- fixed some bad error handling for matplotlib import
- misc fixes
next week
The big GUG overhaul is the main thing. The button where you select which site to download from will seem only to get some slightly different labels, but in truth a whole big pipeline behind that button needs to be shifted over to the new system. GUGs are actually pretty simple, so I hope this will only take one week, but we'll see!
Version 319
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v319/Hydrus.Network.319.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v319.tar.gz
I had a mixed week, but I got some good work done. It is mostly improvements to the downloaders, and then some stuff for advanced users to play with.
downloaders and urls
Following the success of the new gallery page 'delay' system, where gallery page hits across the program are spread out over time, I have made several improvements: the system's logic is more reliable, it occurs on a per-domain basis, the waiting period and status updates now occur in the network engine itself (so will appear in the network job control ui–I expect to add some kind of 'override wait' action to the cog menu soon), and the thread watcher now also has a delay similar system, also under options->downloading. Also, the network engine will now not permit more than three simultaneous jobs per domain as a global bottleneck, and the overall total num connections is boosted as a result to 15 (from 10). In general, the network code should be just a bit more pleasant and less CPU-spiky while still serving a decent throughput when there is work to be done.
The right-click->share->copy->urls menu on thumbnails and the media viewer is now much more powerful. For the focused file, it can now copy (all/all recognised) URLs, and for a thumbnail selection, it can copy (all/all of a specific URL Class) URLs. So, if you want to select a hundred files and get all their Deviant Art URLs in your clipboard real quick, it is now easy!
Also, the client now does some additional normalisation of URLs (it alphabetises query strings on all URLs, not just those matched in the new system). It will take a minute or two to convert and merge your existing DB URLs on update.
Due to some of the phases of network engine update and normalisation, some subscriptions have been hitting their 'periodic' limits by accident. If you have seen some of these, or get more this week, please do not freak out unless you get additional errors–it should all blow over in another sub cycle.
searchers, or gugs as I like to call them
I have finished the first version of the 'searcher' object. I have also decided to call it the Gallery URL Generator (GUG), as this more precisely describes what it does. Advanced users may like to check the new not-plugged-in dialog under network->downloader definitions->manage gallery url generators to see what I am going for. Let me know if there is anything you think this object needs that I have forgotten!
Assuming this goes well, I'll plug this in next week and we'll start experimenting for real.
full list
- started the new convert-query-text-to-gallery-urls object. these objects, which I was thinking of calling 'Searchers', will be called the more specific and practical 'Gallery URL Generators', or GUGs for short
- the first version of GUGs is done, and I've written some test ui for advanced users under network->downloader definitions->manage gugs. this ui doesn't save anything yet, but lets you mess around with different values. if we don't think of anything else needed in the next week, I will fix this code for v320 and start filling in defaults
- watchers now have a global checking slot, much like the recent change to galleries and subs. it safely throttles dozens of threads so they don't rudely hammer your (or the destination server's) CPU if they all happen to want to go at once (like just after your computer wakes up). the option is similarly under options->downloading, and is global for the moment
- moved the new gallery delay/token management code to the better-fit bandwidth manager (it was in domain manager before)
- the gallery delay/token code now works per-domain!
- moved the gallery delay/token checking code into the network job proper, simplifying a bunch of import-level code and making the text display now appear in the network job control. token consumption now occurs after bandwidth (it is now the last hoop to jump through, which reduces the chance of a pileup in unusual situations) I expect to soon add some kind of 'force-go' action to the cog menu
- the network engine will now not permit more than three jobs active per domain, and the overall limit has been raised from ten to fifteen
- the media right-click menu now supports copying: all of a files recognised urls; all of a files urls; all selected files' urls of a specific url class; and all selected files urls
- reworked and harmonised a bunch of urlparsing and generation code–all urls should now appear as full unicode across the program, generally without %20-type encoding characters unless explicitly entered by the user. character encoding now all happens on the backend in requests
- non-url-class-matched urls now have their query parameters alphabetised as part of the normalisation process
- all urls in the db will have their query params alphabetised on update, and any file relationships merged to the new/existing normalised url
- the manage urls dialog will now normalise newly added urls (but should also still permit the removal of non-normalised urls)
- reworked how gallery hits update file import object caches, particularly for subscriptions
- fixed an issue in subscriptions gallery logging where the gallery log would always state it had found the max number of files and typically redundantly generate an 'ignored' stub–it should now say something like 'found 7 files - saw 5 previously seen urls, so assuming we caught up' as originally intended
- simplified some gallery->file import object creation
- galleries now compact until 100 entries (was 25)
- watchers now gallery-compact after a successful check
- watchers now show the 'just added'/'already watching' status for 15s, up from 5s
- network report mode now reports three time–once each for job addition, start, and successful completion
- fixed an issue with the new 'max width' popup sizing calculation that was sometimes not fitting for new height requirements correctly
- fixed an issue with the new url class next page generation code
- fixed an issue where TIOs with data regarding since-deleted services were failing to initialise at the ui level
- misc status text cleanup
next week
I want to finish all the remaining 'default' gallery parsers. If I can get them all out and get GUGs actually working ok, I might be able to finish off the rest of the GUGs for v321, auto-convert all the legacy boorus people have to the new system, and completely delete the old downloader code! Fingers crossed, we are absolutely in the end-game of the downloader overhaul now.
Release Tomorrow!
I had a great week. All the 'default' galleries are now parsed on the new system, and I am completely prepped for the last big switchover. I also fixed some bugs, including issues with new subscriptions and source url checking, and sped up some tag searches significantly. There is also some cookies.txt import stuff for advanced users to play with.
The release should be as normal tomorrow.
Version 318
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v318/Hydrus.Network.318.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v318.tar.gz
I had a great week. I caught up on more small stuff and put some more work into the new gallery downloaders.
downloaders
Unfortunately, tumblr decided to kill their 'raw' url access this week. There is not a lot of firm word from tumblr on the issue, but there is some scattered conversation around, and it seems it is dead and not to return. Whatever the ultimate reason for this, it broke our tumblr downloader, so v318 has an updated tumblr parser to fetch the 1280px versions of urls. I have also taken the opportunity to switch the tumblr 'gallery' parser over to the new system, so the tumblr downloader now fetches and associates neater 'post urls' for its file import objects rather than raw file urls and adds 'creator' tags. However, because of the URL changes, your tumblr subscriptions will hence hit their 'periodic' file limits and likely redownload some 1280px versions of files you already have in raw–if this affects you heavily, you might want to pause your tumblr subs before you update and carefully experiment and curate what happens after you are working again in v318. Note that some artists (like lmsketch) attach a lot of images to their posts, so if your periodic limit were 100–and that 100 now means 'posts' instead of file urls–you are potentially talking a lot of files that are going to be redownloaded! Again, I recommend heavy tumblr subscribers pause before update and maybe consider recreating their tumblr subs from scratch with an initial file limit of 10 or so.
The multi-gallery download page has some more improvements this week. I've fixed an issue where the sub-downloaders on the same page were unintentionally all sharing some bandwidth rules with each other. Multi-galleries also now have an 'added' column. And the 'pause' ⏸ and 'stop' ⏹ characters used in the lists on the multi- pages are now editable, also under options->downloading, for those who have trouble viewing this unicode.
I have also made the 'only get a gallery page every x seconds' option global to the whole program (it was previously per-downloader). Being able to create twenty new whateverbooru queries at once with a single click of the paste button is powerful and great, but it also meant spamming servers with many heavy gallery requests all at once, so now all downloaders share the same slot that comes up every x seconds. The delay option is under options->downloading. I recommend 15s for downloaders and 5s for subscriptions. Let's see how 'global' works, and if it is an annoying bottleneck, I will see about making it per-domain.
Subscriptions now auto-compact whenever they sync. This means they delete old fully processed URLs they no longer need to calculate file velocity just to keep them running snappy. You shouldn't notice any change except maybe a faster-loading 'manage subscriptions' dialog.
A couple of unusual data problems meant that xbooru and gelbooru were not searching well in the new system. I have fixed these, so if you got affected by this, please rerun your queries and let me know if you still have any problems. I also added gallery parsers for rule34.paheal and mishimmie (the paheal update should finally fix the 'paheal gets gallery urls in file results' issue!). Advanced users might like to refer to the gelbooru situation (and tumblr and artstation api gallery url classes) to look at url classes's new 'next gallery page' component, which lets you define an easy logical way to predict the next gallery page from a recognised gallery url and now acts as a fallback if the gallery parser cannot find one (as is usually the case with api results!).
full list
- downloaders:
- extended url classes to support 'next gallery page' generation–a fallback that predicts next gallery page url if the parser cannot provide it (as is often the case with APIs and unreliable next-page-url galleries such as gelbooru)
- integrated this new next page generation into new gallery processing pipeline
- updated gelbooru, tumblr api and artstation gallery api url classes to support the new next gallery page business
- fixed the url class for xbooru, which wasn't recognising gallery urls correctly
- wrote new gallery parsers for rule34.paheal and mishimmie (which are both shimmie but have slightly different gallery layout). this should finally solve the 'one paheal gallery url is being parsed into the file list per page' problem
- 'fixed' the tumblr parser to fetch the 1280px url (tumblr killed the raw url trick this past week)
- misc text/status fixes
- wrote a gallery parser for tumblr that fetches the actual tumblr post urls and hence uses the new tumblr post parser naturally! (tumblr post urls are now more neatly associated as 'known urls' on files!)
- note that as the tumblr downloader now produces different kinds of urls, your tumblr subs will hit your periodic limits the next time they run. they will also re-download any 1280px files that are different to the previously fetched raws due to the above raw change (protip: keep your subscription periodic file limits low)
- cut the 'periodic limit' subscription warning popup down to a much simpler statement and moved the accompanying help to a new help button on the edit sub panel
- multi-gallery pages now have an 'added' column like multi-watchers
- the new 'pause' ⏸ and 'stop' ⏹ characters shown in the multi-downloader pages are now customisable under options->downloading (some users had trouble with the unicode)
- the watcher now shows the 'stop character' if checking is 404/DEAD
- fixed an issue where the new gallery imports on the same multi-page were all sharing the same identifier for their ephemeral 'downloader instance' bandwidth tracker, which meant they were all sharing the same '100rqs per 5mins' etc… rules
- the page and subscription downloader 'gallery page delay' is now program-wide (since both these things can run in mass parallel). let's see how it goes, maybe we'll move it to per-site
- subscription queries now auto-compact on sync! this means that surplus old urls will be removed from their caches, keeping the whole object lean and quick to load/save
- gallery logs now also compact! they will remove anything older than twice the current death velocity, but always keep the newest 25 regardless of age
- .
- misc:
- the top-right hover window will now always appear–previously, it would only pop up if the client had some ratings services, but this window now handles urls
- harmonised 'known urls' view/copy menu to a single code location and added sorted url class labels to entries (which should reduce direct-file-url misclicks)
- greatly sped up manage tags dialogs initial calculation of possible actions on a tag alteration event, particularly when the dialog holds 10k+ tags
- greatly sped up the second half of this process, when the action choice is applied to the manage tag dialog's current media list
- the buttons on the manage tags dialog action popup dialog will now only show a max of 25 rows on their tooltips
- some larger->smaller selection events on large pages with many tags should be significantly faster
- subscription popups should now 'blank' their network job controls when not working (rather than leaving them on the old job, and without flickery-ly removing the job control completely)
- the file cache and gallery log summary controls now have … ellipsized texts to reduce their max width
- fixed an issue where larger 'overriding bandwidth' status wait times would sometimes show instead of legit regular smaller bandwidth wait times
- removed a now-superfluous layer of buffering in the thumbnail grid drawing pipeline–it seems to have removed some slight lag/flicker
- I may have fixed the issue where a handful of thumbs will sometimes remain undrawn after several fast scrolling events
- gave the some-linux-flavours infinitely-expanding popup message problem another pass. there _should_ be an explicit reasonable max width on the thing now
- added a 'html5lib not found!' notification to the network->downloaders menu if this library is missing (mostly for users running from source)
- help->about now states if lz4 is present
- gave 'running from source' help page another pass, including info on running a virtual environment
- in file lookup scripts, the full file content now supports string transformations–if this is set to occur, the file will be sent as an addition POST parameter and the content-type set to 'application/x-www-form-urlencoded'. this is a temp fix to see if we can get whatanime.ga working, and may see some more work
- if the free space on the db dir partition is < 500MB, the program will not boot
- if the free space on the db dir partition is < 1GB, the client will not sync repositories
- on boot the client can now attempt to auto-heal a missing local_hashes table. it will give an appropriate error message
- misc post-importing-cleanup refactoring
next week
I still have a few more small things I want to catch up on, but it isn't so urgent now, so I'd like to get started on the new 'searcher' object, which will be the final component of the downloader overhaul (it will convert the initial 'samus_aran' search phrase into an initialised search url). I feel good about it and may have some test ui for advanced users to play with by v319.
What stops you from using hydrus more?
I think that suggestions make the program better, improvements here and there also go a long way, but really there are hard aspects of the program that cause people to stop everything they are doing till they are improved/reworked.
let me give an example. I have an image db that is currently 2 million images large.
There was no way in fuck that I was keeping tabs on this shit on its own, so the program is very good in that regard, at least my shit is sorted and can be brought up if I tag/rate something.
now, here comes the stop gap.
I have 2 million files, and before my db corrupted, I had something around the area of 200k potential matches for duplicates
so I need to go through the duplicates first
my database broke through so I have only 827k images in the dup finder with 73k images.
Now thinking ok, while things get sorted out, why not do them? dup finder only works on 2 images at a time, regardless of if the rest of my db in or not there is work that can be done
well here is a random pair
im on a 4k monitor, a 55 inch 4k, so when an image is small and one is big, you can generally see one is better easily, however the smaller one is not worse by a lot, note the bigger ones embers, on the yellow shit at the bottom right, zooming into 100% on it makes it clear its not just an upscale, and that it actually is a higher resolution. but if you put these in the program, and find these in dup detector, note that due to the slightly different resolution the program won't keep zoom or where its zoomed in between the two images.
also note that trying to move where its zoomed into fast with the mouse results in making a selection rather then moving the image.
I also noted at least with ratings a while back making a 'this one is better/prefer this one' selection did not transfer the ratings or the tags for me, I could have set this shit up horribly and its my fault they didn't work but still.
but that also hinges on a 'there can be only one' system, where the loser gets deleted,
I did a test a while back when per image notes were something you can do, I wanted to see what would happen if something got deleted, would a note show up.
The image I posted for this shows its use, it would either be a user note or a general note, that would say what you deleted it, im not sure about any of you, but I want to know why I deleted something and if it was a mistake get it back, there was a point where I was deleting stuff and All I can think of is what if one of those was a mistake.
and thats what this comes down two
I can't work on db tag/rate in general due to duplicates and the thought of doing several hundred thousand uselessly
\/
I cant work on the whole archive because of a borked archive
\/
the archive I want to work on is a small subset
\/
the way to work on the subset is not ideal
\/
even if I worked on the subset what would be done? I can't delete anything yet.
and that's where it ends for me.
I still use the program because its better then what I was doing
but the moment a note can be set for what something is being deleted and show up in ui, perfect,
If a quick menus for common reasons something is being removed happens, perfect.
If notes can take a dup image selection and put either 'have higher quality/prefer alternate' on it automatically, even better
at that point im able to work through the dups, and then I would work toward getting all my files in dup and dealt with,
if dup detector would have arbitrary zooms and try to follow images regardless slight ratio differences, even better,
and once that's done I would be able to dive into rating and tagging everything I have
Im honestly wondering what other peoples hangups are on why they don't use the program more/fully
I think a list like this would be more useful to hdev then small suggestions that end up pushing bigger things further away.
Release Tomorrow!
I had a mixed week, but I got some good stuff done. The new downloader systems have a variety of improvements and I finished a first version of the new 'searcher' object for advanced users to play with.
I have a bit of additional testing to do, but the release should be as normal tomorrow.
Version 317
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v317/Hydrus.Network.317.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v317.tar.gz
I had a great week. I polished some ongoing download stuff and caught up on a bunch of smaller jobs.
tag filter and tag import options
A while ago, I wrote a new 'tag filter' that applies blacklist and whitelist rules to a list of tags. It is more powerful than some of the old systems I was using at the time, but the ui was pretty ugly, so I didn't roll it out everywhere it could be. This week, it gets new responsibilities and completely reworked ui and workflow.
The filter's edit panel (which you can get to under tag import options from the blacklist button or beside the 'get tags' checkboxes) now has simpler 'whitelist' and 'blacklist' sub-pages that let you just say 'just give me namespaces x, y, and z' or 'do not get "tagme" tag' or 'do not download if you see "vore" or "diaper"' and so on while still letting you make a very complicated filter if you need to. These simple pages are the default. They also offer namespace checkboxes (as compiled from all your current parsers) to make it simple to set up a character/creator/series whitelist in just a few clicks. All the surrounding help has been given a pass as well, so it should all be just a better all-around workflow from now on!
The old list of namespace checkboxes on tag import options is now gone. If you have some tag import options that use these, they will be automatically converted to the new 'get tags' and a tag filter with the same namespaces checked, so you shouldn't experience any change–your options are now just more powerful if you ever want to change them.
With this change, tag import options are now completely decoupled from the old downloader, which makes a bunch of things easier on my end. Other than some little tweaks here and there, I think I am done with tag import options for now, so if you have been waiting to see how it now works, please check out the new defaults options under network->downloaders->manage default tag import options.
multi-downloader improvements
I made some quality-of-life improvements to the new multi-gallery and -watcher download pages. Loading a highlight should now be much much faster (>3s for some down to typically <50ms) and filter out deleted files. And adding or removing a query/thread should update the list immediately rather than after an annoying half-second delay. Also, if you prefer a new query or watcher to be immediately highlighted if there is not an existing highlight, there are now options for this under options->downloading.
Also, an issue where watchers were often still checking after 404 is fixed. While I regret the problem, I am glad the new gallery log and related improvements are making it easier to identify and diagnose these 'gallery-level' problems when they happen.
And I have added some parsers for imgur and derpibooru, thanks chiefly to efforts from the community. You should now be able to drag-and-drop pretty much any imgur link onto the client and get everything, including for large galleries, and for mp4 videos as well. Derpibooru doesn't have gallery support yet, but you can drag-and-drop single file page URLs and you should get tags and everything. Let me know if you discover any problems with these!
slower thumbnail scrolling
I've added an experimental new option to options->gui to change the rate of thumbnail scrolling. It starts at 1.0 for 1 thumbnail height per scroll tick (which for most OSes means 3 thumbs scrolled per mouse wheel click), but if you want to change it to 0.5 or 0.37 or whatever, it should work ok. I had to improve a bunch of the thumbnail drawing pipeline to get this to work, and I am glad I did as some of it was real creaky. I expect to do a little mork work here in the coming weeks to reduce some scroll jitter (and maybe some redundant CPU) I noticed while working on this.
full list
- completely overhauled the tag filter panel:
- the tag filter panel now has 'whitelist' and 'blacklist' pages beside the old 'advanced' sub-panel. these new simple pages are much more human friendly for common workflows and provide easy-select checkboxes for namespace classes (which are compiled from all the namespaces your parsers can currently do)
- the tag filter rule entering workflow now stops you from creating overcomplicated rulesets: when adding a blacklist rule, it will now only add an explicit entry if it is not already blocked by a higher rule (otherwise it will just discard from whitelist, if there)–and when adding a whitelist rule, it will now only add an explicit entry if it is already blocked by a higher blacklist rule (otherwise it will just discard from blacklist, if there)
- tag filters now provide more human-friendly summary statements
- misc improvements to tag filter ui logic
- the various help texts surrounding the tag filter panel all got passes
- the tag filter panel now uses text-and-paste controls for mass-adding of tags
- namespace checkboxes have been completely removed from the tag import options panel and various other related places. any existing TIO with checked namespaces will be automatically updated to 'get tags' with an appropriate filter. this is an important step in the rewrite–everything is now handled in the new tag filter panel
- simplified and sped up the actual tag filtering code
- .
- numerous multi-importer improvements:
- the gallery and watcher page lists will now ~dynamically~ resize in height based on number of entries, from roughly four columns to twenty four. this relayout code somehow seems to work on all platforms
- sped up the 'results loading' step of gallery/watcher highlighting immensely–on a typical list of a couple hundred files, it should now be about 50ms total (before, depending on presentation rules, it could be 0.8-3s)
- added an additional db-skipping optimisation for calculating presentation status
- watcher and gallery highlights will now filter out trash and completely deleted files (the ones that appear with a dark default 'hydrus' icon) on reloads
- added two checkboxes to options->downloading for 'if nothing is highlighted when I add a new X, highlight that new X' for watchers and galleries
- adding or removing a query or watcher from the new multi-lists should now be reflected in the list ui instantly, rather than after a <=1s delay
- added url classes and parsers for imgur single and multiple urls–thanks to the community for providing some examples
- added url class and parser for derpibooru single file pages–again thanks to the community. derpibooru hence now supports basic drag and drop import
- fixed an issue where the watcher was often still checking despite 404 status
- watchers and galleries use a little less CPU to update some of their ui
- added simple subsidiary page parsing support to file import objects (previously, this only worked in the gallery log)
- .
- gave the thumbnail scrolling code a pass–it is now a bit cleverer about drawing and uses a larger number of smaller 'tile' bmps rather than pages
- added an 'EXPERIMENTAL' option to options->gui to change the number of thumbnails each scroll tick scrolls. it defaults to 1.0, but you _should_ be able to set 0.5, 0.37, whatever. please report any bugs!
- added a thumbnail debug mode to help see the new thumbnail layout boundaries
- .
- misc:
- the max subscription file limits are now 10,000 for users in advanced mode
- the default subs initial/periodic limit is now 100/100 (bumped up from 100/50)
- the file import dialog now has a little cog icon to change whether human sort is applied on path addition events (for e.g. if you want to add in some date order from an explorer window)
- humansort now sorts case-insensitive
- by default, unmatched urls will no longer display in the top-right of the media viewer. see how you like this and let me know if you would like an option to put them back
- the speed text on the right-side of the network job control now dynamically resizes to its min size, which gives the text on the left side (where it is often cut off, saying 'overriding bandwidth …') more space when available
- I think I fixed an issue where the popup frame could spam-resize in odd ways (such as growing a pixel wider every update tick)
- watchers will no longer include the '* ' highlight prefix in subject-based sort comparisons
- in prep for an eventual major code refactoring, the thumbnails' underlying media object now stores a faster db-based numeric file identifier
- 'duplicate' calls on the new listctrl will now insert the dupes in the current correct sort location, rather than tacking them on the end
- drag and drop imports to the new listctrl will also now insert like this
- caught up edit subscriptions panel to the finalised common listctrl panel code, including the import/export/duplicate buttons
- the multiple checkboxlist selection dialog now sorts by label
- converted all old checkboxlist dialogs to the new panel system
- massively sped up certain kinds of parsing that were wasting time hitting a cache test way too often
- fixed an old hash filtering system
- moved to a simpler and more stable way of calculating certain text extents
- fixed an issue where the include directory (which has the original source, which isn't a big deal but is nice to have) wasn't being correctly copied into the linux build
- the os x .tar.gz build now has the include directory
- refactored some client tags code around
- misc cleanup
next week
I enjoyed catching up on things this week, and there is more to do. I'd also like to get some more gallery parsers done and fixe some unusual problems in the new system (gelbooru is sometimes cutting queries off at page two, and some similar issues with other boorus), and start thinking about the final 'searcher' object for the downloader overhaul.
Release Tomorrow!
I had a great week. I caught up on a bunch of smaller issues and bugs and worked on a variety of gallery stuff–gelbooru and xbooru gallery searches are fixed, the tumblr parser is (regrettably) 'fixed' for the 'raw url' issue, and there's plenty of behind-the-scenes downloader improvements as well.
This release should be as normal tomorrow.
Version 315
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v315/Hydrus.Network.315.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v315.tar.gz
I had a mixed week. The gallery rewrite proved more complicated than I expected, so I put it off for a week.
all misc stuff this week
I updated to the new version of wxPython. It has a bunch of bug fixes that I think will help some menu regeneration problems some users have had, and I hope improve stability for the platforms/flavours where crashing has been a persistent issue. Please let me know if you notice your jank-situation improves or declines.
The old default tag import options ui under options->importing is now completely removed. Everything is now under network->downloaders->manage tag import options, which now also has some new copy/paste buttons to help mass-apply TIOs across multiple specific url classes in its list.
You can now set default 'checker options' for subscriptions under options->downloading. The various checker options buttons across the program also have tooltips for quick review.
I added some human-friendly number sorting this week and applied it to the paths that get entered in the file import paths dialog (when you drop a folder on the client). It makes files named like 'Favourites - 10.jpg' sort [10, 11, …, 99, 100] rather than the purely lexicographic [10, 100, 11, …, 99]. If you have had issues here, let me know how it works and where else it could apply!
The migrate database dialog got a pass. It presents some information better and its workflow is simpler. I'll update the help soon to match it.
The moebooru parser now fetches the original 'png' of files where available, and I added a tumblr post parser that also fetches the post tags–if you want this, hit network->downloader definitions->manage url class links and edit->set it for the 'tumblr file page api' entry.
full list
- got started on the big gallery update, but decided not to pull the trigger just yet. I hope to do it next week, switching the whole thing over to a two-object multi-watcher kind of deal
- updated to wxPython 4.0.3 for all platforms
- cleaned up some menubar replacement code, and the update to the new wxPython should also fix a "event for a menu without associated window" bug some gtk2 users were seeing on quick menubar changes
- manage default tag import options panel now has copy/paste buttons that work on the listctrl
- added some 'paste tag import options' safety code to make sure no one accidentally pastes a subscription or something in there, wew
- added default checker options for subscriptions to options->downloading
- unified how checker options are edited from their button, much like how file and tag import options work. it also has a summary tooltip on the button
- the checker options under options->downloading are now these slimmer buttons
- in the manual import dialog (which pops up when you drop a folder/files on the client), the files will now be added in 'human friendly' number sorting, so files of the sort 'Favourites - 10.jpg' will sort [10, 11, …, 99, 100] rather than the purely lexicographic [10, 100, 11, …, 99]
- gave the migrate database dialog a pass–a bunch of misc presentation changes and a general simplification of workflow, now based more on just increase/decrease location weight
- a bunch of texts on page management (left-hand) panels that share horizontal space with buttons should now ellipsize ("downlo…") when they get too long for the width instead of drawing in an ugly way over the buttonspace
- moved the manage import folders dialog to the new listctrl and added a 'paused' and better 'check period' column
- if a user tries to run a 'paused' import folder specifically from the menu, the import folder will now unpause (I will probably remove this old paused variable in the future–it isn't of much use any more)
- tightened up some repository reset code that wasn't deleting all service tables and hence recovering from some service id malformation errors correctly
- wrote a 'clear orphan tables' db maintenance routine that kills some spare tables some users who have previously deleted/reset repositories may have floating around
- fixed an issue with parsing folders after hitting cancel button on the import files pre-dialog
- if watchers encounter non-404 network errors during check, they should now just delay checking for four hours (before, they were also pausing checking completely)
- if watchers are in 'delay' mode, they'll also not work on files.
- file and gallery downloads that hit a 403 (Forbidden) will now present a simpler error status, like they do for 404
- the new post downloader will no longer fail if one of the parsed source urls is not a url. the borked string will also not be associated as a url
- regular gallery downloads now override bandwidth for the file download step, which is almost always the second half of a pair of post_url/file downloads, just to keep things in sync in edge cases
- cleaned up some timestamp generation and 'overriding in x seconds' strings to be more human friendly
- improved some serverside file parse error handling to propagate the actual error description up to the client a bit better
- fixed typo causing incorrect num_ignored count in file import status button right-click menu
- parseexceptions will now present more data about which page and content parser caused the problem. I am not totally happy about how this solution works and may revisit it
- the lz4 import error catching is now more broad to catch some odd problem I discovered in new Linux build environment
- the moebooru parser now fetches the original png of an image, if available
- added a new tumblr parser that also gets post tags–it _shouldn't_ be the default
- the new login pipeline now kicks in for the legacy logins–pixiv and hentai foundry–on a per-url basis, so adding pixiv/hf urls to the url downloader will trigger a login even if needed (previously, this was tied to legacy gallery initialisation, which explains some pixiv 'missing' login stuff some users and I were having trouble with)
- if the legacy login system fails in the new pipeline, it now sets a flag and won't try again that client boot
- the old 'default tag import options' panel is now completely removed from options->importing. please check 'network->downloaders->manage default tag import options' for the new url-based settings
- misc fixes
next week
I'll see about this gallery stuff. I meant to sneakily swap out the basic gallery parsing with the new code this week, but it was way too awkward on its own, and I realised I had to do a bigger 'search' improvement along with it. I expect to convert regular download pages to something like the multi-watcher–something that lists and works on each individual search separately. I've got most of the data-side code written here, so I really just have to make some new ui and then test it a bunch.
Release Tomorrow!
I had an excellent week. A good first version of the new gallery parser is done for gallery pages, subscriptions, and simple drag-and-drop. Furthermore, all gallery pages are now 'multiple' (much like the thread watcher, which is now also multiple-only) and support multiple queries for different sites running on the same page at the same time!
This will be a big release. I still want to do some more testing, but I think I will have it out at the normal time. I recommend heavy gallery users wait for the release post to review the changes and make sure you update well.
No Release Tomorrow!
This week went well, with plenty of work going into the gallery update, but I am afraid I felt more and more ill throughout the week and no longer feel confident I can put out a good release. I could tie off the work I have done and rush something out tomorrow, but I am not thinking completely straight and this is a big change, so I am putting v316 off for a week so it can all be tested properly and isn't in buggy limbo.
I'll just restart next week as usual schedule, so v316 should now be on the 1st August. Thanks everyone, and sorry for the delay!
Version 314
windows
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v314/Hydrus.Network.314.-.OS.X.-.App.dmg
linux
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v314.tar.gz
I had a great week. Tag import options are now easy to mass-manage, and I have fleshed out the new downloader system's file page parsers.
default tag import options
Almost done with tag import options, I promise!
This is an important but slightly complicated update. Nothing huge will change, and you'll see what is going on as you use it. tl;dr is: go to network->downloaders->manage default tag import options and set something good for default for file posts if you haven't already.
This week has an important update to how tag import options (TIO) typically work. First, the old TIO options under options->importing are now deprecated–they are no longer referred to by any downloader, and I will be removing the ui entirely in a couple of weeks. All downloaders now inherit from the newer network->downloaders->manage default tag import options, which operates in the new download system. Please make sure your default tag import options are set up how you want, as any new downloader or sub now inherits from this new location! You will get a popup about this on update as a reminder.
Second, the TIOs for new downloaders and subscriptions now start in a 'default' mode, meaning 'I do not have any specific options, just use whatever is set in the defaults at the time of import'. In the TIO edit ui, this is a new checkbox that greys out the rest of the panel. This means that if you change the new defaults under network->downloaders->manage default tag import options, every 'default' TIO across the program will inherit those changes!
This new 'default' mode makes the job of managing TIOs for dozens or hundreds of downloaders and subscriptions much easier, particularly now that TIO have become more powerful and complicated. Rather than, let's say, adding the tag 'vore' to twenty different subscriptions' blacklists, you can now just add it to the defaults and all your 'default'-set subscriptions will work with that updated default when they next run. But if you have a particular subscription or downloader that needs a special blacklist or namespace filtering, you can still set a specific TIO and make it happen.
Your existing downloaders and subscriptions will keep their specific TIOs on update. They will keep parsing tags like they did before.
But when you are comfortable with this new system, you may want to set many if not all of your subscriptions to default TIO, so I have added a button to manage subscriptions that sets TIO to all selected subs! Furthermore, any normal 'tag import options' button now has a right-click menu that lets you copy/paste the TIO to/from clipboard, so if you do have some clever tag filter or blacklist you want to spam to several places, you can now do it with just a few clicks.
Also, the 'url downloader' now has its own TIO. This also starts 'default', which is basically what it was doing before–referring to network->downloaders->manage default tag import options based on import URL right when it imported–but now you can overrule this 'refer to defaults' mode and set a particular set of rules if you are doing some clever one-time job with the URLs you are giving it.
more parsers
In the new page downloader, if a site does not yet have a new parser, it falls back on the old system. This left me the job of filling in the blanks for all the default sites that come with a fresh install of hydrus. I have been adding new ones here and there, but this week I pulled out the stops and finished them all off. There are new parsers for furry.booru.org, hentai foundry, moebooru (konachan, yande.ro, sakugabooru), shimmie (rule34.paheal, rule34hentai, mishimmie), newgrounds, sankaku (thanks to some users for contributing this one), and tumblr. All these sites now support Post URL drag-and-drop import (and sakugabooru doesn't actually have a search by default, so this is likely the only way you can take advantage of it for now) and have improved file parsing support and can sometimes grab a source time and so on.
So, by default, a new hydrus client now deals with file pages entirely on the new download system. This frees me up to delete more and more of the old legacy system in the coming weeks, shifting everything over to the newer and neater code. I expect I'll throw in some other new parsers for imgur and so on as well!
There remains the issue of custom boorus. Any user who added a custom booru in the old system either by downloading a yaml file from somewhere or making it themselves has something that is a bit tricky to convert to the new system automatically. I made this new system in part so I wouldn't be needed to make new downloaders, so I don't want to go down a rabbit hole of writing a hundred new parsers, but I think I'll figure out a simple way to update the old custom booru objects into barebones new parsers in a coming update. I will probably do it all in one go, when the gallery and 'searcher' object work are done, but it may need a bit of extra work on your end to connect things together. If you care about this stuff, please watch this space. I will be working with the users who are interested in making new parsers, and we'll figure out a variety of 'make it work now in a difficult technical way' to 'I can wait a couple months for the single drag-and-drop solution' methods so you can get derpibooru and the other unusual but desired boorus working properly.
Btw, the new tumblr parser does not work with the in-client tumblr downloader yet, but it will in a couple of weeks when the gallery stuff is finishing up. But if you want to try it out, just try drag-and-dropping some tumblr post URLs on the client in v314. As well as doing the difficult 'raw' URL translation correctly (and not doing it for the pre-2013 URLs for which the translation is invalid), it now figures out a creator: tag, including swapping out for the original root creator on reblog posts! It is almost as complicated as the pixiv parser, and I am really pleased that the new download system is flexible enough to have supported what I needed it to do.
misc
There are some other quality-of-life options and improvements this week:
options->speed and memory now has 'cache timeout' options for thumbnails and rendered images.
options->downloading now has 'hide/show the 'D' on short import summaries', the default being hide.
options->gui now has a 'only autosave last session during idle time', which is useful if you run a really heavy client (like 200k+ files/import objects) that you leave on all the time and are getting regular 10s+ lag from the session autosaving. I still recommend you try if you can to keep your client below 50k at once, but if you can't, try this out and let me know how it goes.
The network menu is now categorised into something better than 'a giant single mess'.
Edit subscription panels now has 'copy queries', which will copy all the selected subscription queries' search texts to your clipboard. This is basically the opposite of the 'paste queries' button on the same panel, and makes it easy to copy twenty queries to a new subscription or chat window or whatever.
full list
- tag import options can now be set to 'default', meaning 'use whatever the default is at the time of import', which will be an easier way of managing TIOs for many subs that you'd prefer all share the same TIO settings anyway
- updated tag import options ui to enable this default setting where appropriate
- updated the newer import pipeline to work with 'default'-set tag import options
- new downloaders, subscriptions, watchers, and multi-watchers now start with 'default' tag import options
- deleted the old default tag import options management code and put some text up on options->importing making notice about the impending shift. a popup message will also say this on update
- tag import options buttons now have a right-click menu with copy/paste/default options for quick assignment and duplication!
- added 'overwrite tag import options' to manage subscriptions–once you are comfortable with the new 'default' TIO mode, and after some small tests, you might want to switch all your subs over to 'default'
- the 'urls' downloader now has a tag import options–it initialises as 'default'
- added furry.booru.org to gelbooru 0.2.0 parser
- wrote a hentai foundry file page parser
- wrote a moebooru file page parser (this works for konachan, yande.re, and sakugabooru)
- wrote a shimmie parser (this works for rule34.paheal, rule34hentai, and mishimmie)
- wrote a newgrounds parser
- integrated the user-created sankaku parser
- wrote a tumblr parser that handles photo, photoset, and video posts, auto-converts to 'raw' urls for those post-2012 urls that can handle it, figures out a creator tag (reverting to the reblog root if it is a reblog post!), and cooks you breakfast
- (hence all these above sites now support drag and drop!)
- rolled out some new tumblr url classes to handle all this.
- added sakugabooru url classes
- fixed an issue where url classes were not normalising api urls in all cases, meaning some url classes would not api-link correctly in 'manage url class links' panel
- fixed an issue with deviant art legacy gallery parser pulling some funky 'creator:' tags
- some misc new downloader error handling improvements
- the watcher now uses the new gallery object to parse and generate file import objects
- the downloaders with gallery logs should now report non-success gallery fetches, along with error tracebacks (this will include some no-worry 404s the legacy downloader sometimes uses to terminate searches)
- added image and thumbnail cache timeout time delta buttons to options->speed and memory
- added a 'show the D on short file import summaries' checkbox to options->downloading–it defaults to off
- the 'I' on short file import summaries is now 'Ig' to clear up 1/I confusion
- added 'copy queries' to the edit subscription panel, which lets you copy all the selected queries' search texts to clipboard, newline separated
- added a checbox to options->gui that commands 'last session' only be autosaved during idle time. this is useful if you usually have a huge (200k+ file) session and your client is always on
- fixed file import status button right-click, which I messed up somehow last week with the 'retry ignored' add
- shook up and collapsed the network menu into neater categories
- tightened-up the rarely used pre-parsing conversion panel on the edit page parser panel to just a button with a bit of explaining text
- if database errors include the word 'malformed', the client now throws a little extra error text pointing people to the help.txt in the db dir
- cleared out some legacy download code
- cleared out legacy hard drive import error handling, moving it all to the new file import object
- misc refactoring and cleanup
next week
I moved the new gallery watcher forward as well, this week. The new gallery pipeline is mostly in place, and thread watchers now use it, so now I have to re-jigger the regular 'walk through gallery page' downloader code around a bit and then I can blitz a bunch of new gallery parsers (which are thankfully very simple) like I've just done the file page parsers today. So, I'd like to have gallery url drag-and-drop (i.e. you drag and drop a danbooru gallery search url onto the client, and it starts a new hydrus page for that search, either getting just that page's thumbnails, or starting a new search from page index 0, or starting a new search from the dropped page) in for next week.
Client maitenance error
After SQlite run out of disk space on after exit maitenance, with following errors:
DBException: OperationalError: database or disk is full
and then
OperationalError: no such savepoint: hydrus_savepoint
(log https://pastebin.com/8JWzsrwj)
Hydrus client does not offer to do maitenance anymore after exit and does none while idle. Encountered on 310, after updating to 313 Hydrus asked to do maitenance once, but no maitenance was actually done, and it wasn't asking anymore again.