windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v313/Hydrus.Network.313.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v313/Hydrus.Network.313.-.Windows.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v313/Hydrus.Network.313.-.OS.X.-.App.dmg
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v313/Hydrus.Network.313.-.OS.X.-.Extract.only.tar.gz
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v313/Hydrus.Network.313.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v313.tar.gz
I had a great week. There are some more downloader and tag import improvements, and my gallery overhaul is going very well.
gallery log
Other than some final parsers, tweaks, and bells and whistles, the 'page download' part of the downloader overhaul is done, so I have moved onto the gallery part–where pages of thumbnails are converted into Page URLs. This work is going very well–a lot of the data side of it was already done due to the page download stuff–and I am rolling a bit of it out today.
All the downloaders will now have a new 'gallery log' box attached, which is a little bit of text and a second 'table' icon button, like the one for file imports. This button launches a window with a similar table of data, this time listing all the Gallery URLs visited and the result (including any errors). Until now, this information has been discarded, but having this logged will prove very useful for a variety of reasons–I already used it on my dev machine this week to debug some pixiv stuff.
The gallery log is read-only for now (it only reports what has happened), but I will spend the coming weeks adding in a new gallery parser and start offloading the work from the old system when available, just as has been happening with the file downloader, and then I'll be able to add some kind of 'retry' and 'continue from here' kind of commands to its right-click menu and add Gallery URL drag-and-drop and so on. Thankfully, gallery parsing is significantly simpler than post page parsing, so I expect to roll this out pretty quick. I am really pleased with how this has come together.
For those users who are interested in editing in the new parsing system, please check out the new 'danbooru gallery page parser' under network->manage parsers. It doesn't do anything yet, but it is a good idea of the basics of what you'll be creating here.
tag import options filters
Tag import options is marching on. The new 'get all tags' checkbox now has a 'Tag Filter' object attached which lets you control which tags you want much more finely than the old namespace checkbox system–such as 'get all tags except "species:" tags' and 'get only "creator:", "character:", and "series:" tags, but (these ten unnamespaced tags) are also fine'.
Also, the new and advanced 'only add tags that already exist' option has a similarly-advanced filter to go along with it, so if you are feeling clever, you can now say 'get all namespaced tags, but only get existing unnamespaced tags'.
I've written a bit of help into both the new tag filter editing dialogs, but please let me know how it could be improved. This stuff is complicated, and I could see adding an 'easy mode' checkbox/panel after some feedback.
I expect to phase out the old namespace checkbox system in the coming weeks. Any existing checked namespaces will be converted to an appropriate tag filter beside the 'get tags' checkbox. It will still be useful to know what namespaces a parser expects to generate, so I'll see about putting that somewhere in the ui as well.
I will be adding more ways to quickly copy/paste and mass-set/inherit tag import options, so if you want to take advantage of these new filters, please do a couple of small tests to make sure you know what you want, and then wait for these new features. Don't waste time editing a hundred subscription TIOs' filters individually!
downloaders
In multi-watchers and subscriptions, the summary of import progress now has a shorthand for number of unsuccessful results, like '222/245 - 2I4F', which means "222 items out of 245 processed, with 2 'ignored' and 4 'failed'". I like the overall change, but looking at my updated IRL client now I think I'll pull the 'D' for 'deleted' out, as this is a bit spammy not so useful. The 'I' and 'F' are great for quickly finding problems, however.
Just as I rolled out the Deviant Art parser last week, they switched up how their Post URLs work, which meant DA subs will have hit their periodic limits (thinking they were seeing new URLs) and also meant my new parser wasn't kicking in. I've fixed the underlying URL-matching in the update this week, so while you will still get some subscriptions moaning about hitting their periodic file limits, it should all settle down in a sync or two and the new parser will kick in properly from now on. You might like to scan your DA subs for any with 'I' in their progress texts as above, as the new parser may be able to work on these URLs (typically these are images with high-res download disabled, which the old parser could not handle, but there are also some 'here's a pack of adobe brushes in a strange file format' that will just fail harmlessly).
I discovered that the Pixiv downloader was missing explicitly 'manga' works this week, so that is also corrected. The multi-file works it was previously fetching were only 'multi-file illustrations', which are under a slightly different section of their site. I expect I removed the manga search (technically, I was searching for 'type=illust' instead of 'type=all') in an old version when manga results were more of a pain than not. In any case, manga is now included in searches going forward. If you have some subscriptions that you would like to retroactively fetch old manga links for, instead of resetting the subs or running a full manual download again, I recommend you wait for the gallery overhaul (and the subsequent 'Searcher' work) so I can roll out an explicitly 'manga-only' Pixiv downloader, at which point you can just run manual downloads for just those artists' manga (due to Pixiv being odd technically, re-running entire searches can eat a bunch of bandwidth, as it is difficult to tell if the client has seen a pixiv file/URL before and so skip the download).
full list
- fleshed out the new gallery log and its constituent log entry objects
- added gallery logs to gallery downloaders, subscriptions, url downloaders, simple downloaders and watchers
- added very simple gallery log reporting to these downloaders
- added first, read-only version of gallery log ui to these downloaders
- fleshed out some new gallery/file-object pipeline stuff
- wrote a simple danbooru gallery page parser and added it on update. it doesn't do anything yet, but if you are into the new parsing system, please check it out as an example
- the url downloader now has a full file import status control with status text
- fixed a url count issue on completely fresh gallery downloads that was stopping gallery searches one file (like 199 vs 200) before the file limit
- the pixiv downloader now fetches 'type=all' gallery pages, which include specifically manga file pages (as opposed to merely multi-file 'illustrations')
- added 'retry ignored' to the file import status button's right-click menu
- fixed the deviant art url class and parser to use the new file page format. also added an '(old format)' class to match the old way for legacy purposes (this legacy class also uses an api conversion to connect to the new parser–we'll also figure out a way to convert all these over at the db level en masse later!)
- updated some similar deviant art gallery stuff as well
- tag import options now has a tag filter to go along with the 'get all tags' checkbox! ('get all tags' is now renamed to 'get all' as a result). this filter lets you make more complicated tag filtering decisions like 'get all tags except "species:" tags'.
- the new 'only get tags if they already exist' checkbox now also has a filter, if you want to only apply this test to a subset of tags (like the unwashed mess of unnamespaced tags many boorus and sites provide)
- generalised a 'tag filter' button class to make it simpler to edit tag filters across the program, and cleaned up some related status code
- fixed a problem with deriving tag import options for specific url classes when that url class was part of an api-url-class chain
- if the domain manager cannot now find a url match for a pending download, it now assigns the file post default tag import options to that import
- added a new 'duplicates' options page that has a hacky way to edit the weighted scores used to determine which of the pair of files to present file in the duplicates filter
- unifed how some file import status generation works, adding a new 'simple status' string to briefly summarise progress in multi-watcher and edit subscriptions columns
- cleared out some old redundant status caching in the urls downloader
- simplified how almost all timestamp strings are generated
- simplified how time delta strings are generated
- brushed up some simple common ways to present timestamps as 'human pretty' strings
- all places where timestamps would be presented as a mix of '5 days ago' and complete datetime strings will now present as '5 days ago' unless you set the new options->gui 'always show iso' checkbox. going back to simple to clear up confusion in workflow and code. I may revisit this, as turning on ISO mode now spams it all over the place
- cleaned up the 'looks like the computer just woke from sleep' check and reduced its grace period to fifteen seconds. foreground daemons (like the subscription daemon) and the network engine will now also obey it
- added a 'simulate wake from sleep' debug action to better test the sleep-wake detection code
- improved my custom statictext class to auto-wrap text without flickering
- used this new autowrapping to improve wrapping and layout of popup message texts
- replaced all other st wrapping with this new code
- wrote a little helper function to better dedupe lists in future
- did a bunch of refactoring to neaten some long common func names
- deleted some old unused code
next week
More gallery work. I'd like to flesh out the new gallery log pipeline and have it step in when a parser is available, and then I can start rolling out new gallery parsers. I have also recently had a good run at clearing out small stuff, so I'd like to go through some of my older todo–just some small weird jobs that have been on the back-burner for longer than I wanted.