windows
zip: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.Windows.-.Extract.only.zip
exe: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.Windows.-.Installer.exe
os x
app: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.OS.X.-.App.dmg
linux
tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.Linux.-.Executable.tar.gz
source
tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v356.tar.gz
I had an excellent and busy week catching up after E3. Duplicate better/worse/same relationships are now stored more logically and efficiently, and a variety of bugs are fixed. Maintenance is also easier to control.
duplicates
The second phase of the big duplicates rewrite is now done! Rather than the old system of trying to store better/worse relationships for every possible member of a duplicate group, which is very complicated, requires many human decisions, and is not very useful in practise, this new system instead lumps all duplicate files into a single group with a single 'king' representing the best file of that group.
The groups are transitive, so putting a file into a group with a better/worse/same command–or even merging a group into another–is now just a single action. Rather than comparing a file with every member of a group, it is now just merged as a simple member or the new king. The overall workflow of dealing with large duplicate groups is now much much faster. Furthermore, alternate relationships are stored between groups of duplicates, not individual files, so again the number of choices you need to make is automatically reduced as you process your normal duplicate filter queue.
Searching and managing duplicate file groups is also easier. When searching for system:num duplicate relationships, you simply search for how many 'duplicates' a file has, rather than how many better/worse/same. Right-clicking on a thumbnail to open up the 'file relationships' menu–which is now available to all users–now presents the same simple 'duplicates' count-and-show-in-a-new-page entry and will say whether the file is the best file of its group. If the file is not the best of its group, you can view that king or set the current one as king with one click.
Future updates will show king/not king status in the thumbnail UI. I will also write a new system search predicate to filter for king/not king.
Your existing better/worse/same data will be converted to the new system on update, with appropriate kings chosen. If in recent weeks you noticed some files that seemed to be both alternates and duplicates, this will now auto-resolve as the duplicates are merged together.
The new jpeg quality estimator, which will sometimes present 'high quality vs low quality'-style statements in the duplicate filter, now appears for all users. It isn't perfect, but it works ok.
The last phase of the rewrite will be to update potential pairs. It will reduce the potential pair queue count even further and improve the pair-elimination efficiency of the decisions presented in the filter (essentially by forcing king vs king comparisons). I want to take a week to do some smaller duplicates work (like commands to undo/dissolve existing relationships intelligently), so with luck I expect to have that done for 358. I am overall very pleased with the work here so far, although there is still a good amount to do, and some proper new help to write up.
the rest
I fixed the popup spam that the new clipboard watcher was falsely making on many non-matching URLs. It was some test code that was misfiring–even though the URLs were not being added since they were not matched, a higher part of the system thought they were not being added due to a problem with pages. I am sorry for the confusion–it is now fixed.
Also the unusual OS X shortcuts issue, where shortcuts were not working on some OS X installs until a menu was opened! This was due to a weird OS X specific problem in the recent main gui menubar detection code. If you are on OS X and notice your menubar stops updating, please let me know.
I put a bit of time into the maintenance code this week. Most maintenance jobs, like repository processing and thumbnail regeneration, now use the same 'clock' to figure out if they should be working. They now know if they are running in idle time or shutdown time or forced by the user, and will cancel out faster depending on the situation as needed. The other upshot of this is that the file menu now has a 'exit and force shutdown maintenance' option, and the shutdown splash finally has a 'stop shutdown maintenance' button! I regret that this took so long to come in–but now it is, and it works well!
You can now also force file maintenance from database->maintenance->file maintenance. It tells you if no work needs to be done.
full list
- duplicates:
- moved better/worse/same quality duplicates relationships to the new 'king' group-based model. rather than tracking every relationship, duplicates are now stored in groups with a single 'best' file
- as a result, duplicate relationships are now transitive! saying that one king is duplicate to another will merge groups. the 'better' king is the new king, and 'same quality' kings choose one of the kings pseudorandomly. advanced exceptions: saying that a king is better than a basic member of another group or saying that two members are same quality is still valid but will simply 'poach' the non-king member from the other group in order to ensure the wrong king doesn't end up on top in the eventual merge. saying KingA is same quality as MemberB will merge the groups with KingB as the new king (since it is presumably same/better quality to all members of A)
- the thumbnail right-click 'duplicates' entry is now renamed to 'file relationships' and is no longer advanced mode only. the 'find similar files' entry is folded into this
- the thumbnail 'file relationships' menu now shows a simple 'duplicates' count rather than the old messy better/worse/equal. it will show all the members of a duplicates group when clicked. the menu also notes if the focused file is the best file of its group. if it is not, you will get the option to show the best file or make the focused file the best
- as a result, it is now much simpler to view a group of duplicates and overrule a 'best quality' member as needed
- added the 'media' shortcut 'duplicate_media_set_focused_king' to shortcut setting a 'best quality' file
- the system:num duplicate relationships now has the simpler 'duplicates' entry, to search on size of the entire group. searching for kings/not kings will come soon
- due to the new duplicate transitivity rules, potential pairs are now eliminated at a much faster rate!
- setting duplicate relationships will overrule false positive or alternate relationships already in place
- manually setting alternate relationships to more than two thumbnails at once will now set each file as alternate to every other file in the selection, completely eliminating potential pairs within the group. if you try to do this to large groups of files you will get a longer yes/no confirmation message just to make sure you aren't overwriting some potential dupes by accident
- all existing better/worse/same relationships will be converted to the new group storage in this update, with appropriate kings determined. potential pair queue counts will be reduce accordingly, and the temporary alternate/duplicate confusion from the alternates update will be auto-resolved by merging truly duplicate 'alternates' together
- fleshed out the duplicate test code significantly to handle the new dupe groups and their interactions with the recent false positive and alternates changes
- refactored some db test code into separate client/server/duplicates files and cleaned up dupe tests readability
- potential pairs are now the only component of the new system still on the old pairs system. the duplicate filter will still serve up some inefficient (i.e. non-king) comparisons
- .
- the rest:
- fixed the issue where many clipboard-watcher-caught URLs that did not match were producing false-positive 'could not generate new page for that URL' error popups
- the clipboard text-fetcher now tests against incompatible clipboard types (like a screenshot) better, and all instances of text fetching now report errors more gracefully and with more information
- fixed the unusual OS X issue where many shortcuts were not being processed after client boot until the top menubar was opened and closed. a variety of other blocking-while-menubar-is-open issues that were false-positive misfiring are now fixed as well, please let me know if you still have trouble here
- the file menu now has an 'exit and force shutdown maintenance' option to force-run outstanding maintenance jobs
- when shutdown maintenance work is going on, the shutdown splash screen now has a 'stop shutdown maintenance' button!
- cleaned up some file maintenance manager maintenance locking and shutdown cancel logic
- moved all the idle-mode maintenance checks to a new system that explicitly defines idle/shutdown/forced maintenance work and tests those states in a unified manner, checking idle mode and the new splash cancel button status and so on more reliably. a lot of maintenance should cancel out quicker when appropriate
- misc shutdown logic cleanup
- added a 'file maintenance' option to the database->maintenance menu that forces the new file maintenance manager to run its queue. it'll make a little popup as it works, or a note that no work is due
- the 'regenerate' thumbnail menu is also available to all users
- jpeg quality estimates are now available for all users in the duplicate filter. they only display when the two jpegs' quality have different labels
- the jpeg quality estimator now handles some unusual jpegs that load with empty quantization table arrays
- the duplicate filter now handles bad jpeg quality estimations gracefully
- cleaned up some ffmpeg communication code
- the ffmpeg debug text that spawns on a help->about call that fails to discover ffmpeg version information now prints stderr output as well. if you have been hit by this, please give it another go and let me know what you get
- the same ffmpeg 'no response' error on file parse now popups and prints some debug info and returns a better error
- dialogs and windows on the new panel system now support a new pre-close tidying system
- the manage tags dialog and window will now cancel any pending large tag autocomplete queries on close
- regular gui pages now support a new pre-close tidying system
- search pages will now cancel any pending search results loading or tag autocomplete queries on close
- improved reliability of the popup message manager chasing the main gui when it is sent to another screen by a keyboard shortcut (such as shift+win+arrow on Windows). it should work now if the mouse cursor is in either window. please let me know if this causes trouble for virtual display navigation
- the network engine now waits significantly longer–60s–on connection errors before trying again, and with every failed attempt will wait n times longer again. when in this waiting state, a manual user cancel command cancels it out faster
- I believe I have fixed/improved a situation where media viewer hover windows would sometimes disappear immediately after appearing on some Linux window managers
- improved hover window report mode to state more focus info in case the above is insufficient
- to better link the two requests and consume bandwidth under strict rules more precisely, the override bandwidth rule that kicks in when a file page has a single file is now 3 seconds instead of 30
- updated options->connection page to specify that 'socks4a'/'socks5h' is needed to force remote dns resolution
- sped up tag parents initialisation
- repositories now group tag sibling and parent petitions by the parent/better tag's namespace
- removed some old network 'death time' code that is no longer useful and was interfering with heavy petition processing
- the log now flushes itself to disk every 60s rather than 300s
- misc fixes and cleanup
next week
This week was a crush, so I focused on the duplicates work and bug fixes and cleanup. I have a variety of small 'new stuff' jobs to catch up on next week. Some more shortcuts to add, some client api work, some quality of life like pasting tags into siblings/parents. And some smaller auxiliary duplicates work.
I enjoyed taking a week off, but I am glad to be back to work.