[ / / / / / / / / / / / / / ] [ dir / random / abcu / erp / htg / kind / loomis / mai / miku / x ]

Catalog (/hydrus/)

Learn more about the EARN IT Act, the latest attempt to gut Section 230
/fringe/ has been migrated.
[Create a thread]
Sort by: Image size: [Show all] Archive
R: 8 / I: 2 / P: 1 [R] [G] [-]

Version 398

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v398/Hydrus.Network.398.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v398/Hydrus.Network.398.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v398/Hydrus.Network.398.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v398/Hydrus.Network.398.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v398.tar.gz

I had a good work week. Tag autocomplete gets some new search options, and advanced users who make downloaders get some new text processing tools.

tag autocomplete

When I recently overhauled the tag autocomplete pipeline, I eliminated some unusual logical hoops where you could accidentally fire off expensive searches that would fetch all tags. Now the code is clean, I am adding them back in as real options.

The main thing here is that services->tag display is now services->tag display and search. It has several new options to change search based on what the autocomplete's current 'tag domain' is (i.e. what the button on the dropdown says, "all known tags" or "my tags" or whatever else). The options are available for every specific tag domain and the "all known tags" domain, and only apply there.

There are three new search options: You can have full namespace lookup, so an input of 'ser' also finds 'series:metroid' and all other series tags; you can have an explicit input of 'series:*' show all 'series' tags; and you can have '*' show all tags. These queries are extremely expensive for a large service like the public tag repository (they could take minutes to complete, and eat a ton of memory and CPU), but they may be appropriate for a smaller domain like "my tags". Please feel free to play with them.

There are also a couple of clever options setting how 'write' autocompletes (the ones that add tags, like in the manage tags dialog) start up, based on the tag service of the page they are on. You can set them to start with a different file or tag domain. Most users will be happy with the defaults, which is to stick with the current tag domain and "all known files", but if you want to change that (e.g. some users like to get suggestions for "my tags" from the PTR, or they don't want tag counts from files not in "my files"), you now can. The old option under options->tags that did the "all known files" replacement for all write autocompletes is now removed.

I have optimised the database autocomplete search code to work better with '*' 'get everything' queries. In the right situation, these searches can be very fast. This logic is new, the first time I have supported it properly, so let me know if you discover any bugs.

string processing

This is only important for advanced users who write downloaders atm. It will come to the filename tagging panel in future.

I am plugging the new String Processor today into all parsing formulae. Instead of the old double-buttons of String Match and String Converter, these are now merged into one button that can have any combination of ordered Matches and Converters, so if you want to filter after you convert, this is now easy. There is new UI to manage this and test string processing at every step.

The String Processor also provides the new String Splitter object, which takes a single string like '1,2,3' and lets you split it by something like ',' to create three strings [ '1', '2', '3' ]. So, if your HTML or JSON parsing provides you with a line with multiple things to parse, you should now be able to split, convert, and match it all, even if it is awkward, without voodoo regex hackery.

I also did some background work on improving how the parsing example/test data is propagated to different panels, and several bugs and missed connections are fixed. I will keep working here, with the ideal being that every test panel shows multiple test data, so if you are parsing fifty URLs, a String Processor working on them will show how all fifty are being converted, rather than the current system of typically just showing the first. After that, I will get to work on supporting proper multiline parsing so we can parse notes.

the rest

Double-clicking a page tab now lets you rename it!

system:time imported has some quick buttons for 'since 1/7/30 days ago'.

I cleaned out the last of the behind-the-scenes mouse shortcut hackery from the media viewer. Everything there now works on the new shortcuts system. There aren't many front-end changes here, but a neat thing is that clicking to focus an unfocused media window no longer activates the shortcut for that click! So, if you have an archive/delete filter, feel free to left-click it to activate it–it won't 'keep and move on' on that first click any more. I will continue to push on shortcuts in normal weekly work, adding mouse support to more things and adding more command types.

You can now enter percent-encoded characters into downloader queries. A couple of sites out there have tags with spaces, like '#simple background', which would normally be broken in hydrus into two tags [ '#simple', 'background' ]. You can now search for this with '#simple%20background' or '%23simple%20background'. Generally, if you are copy/pasting any percent-encoded query, it should now work in hydrus. The only proviso here is %25, which actually is %. If you paste this, it may work or not, all bets are off.

I am rolling out updated Gelbooru and Newgrounds parsers this week. Gelbooru searching should work again, and Newgrounds should now get static image art.

full list

- new tag search options:

- there are several new options for tag autocomplete under the newly renamed _services->tag display and search_:

- for 'manage tags'-style 'write' autocompletes, you can now set which file service and tag service each tag service page's autocomplete starts with (e.g. some users have wanted to say 'start my "my tags" service looking at "all known files" and "ptr"' to get more suggestions for "my tags" typing). the default is 'all known files' and the same tag service

- the old blanket 'show "all known files" in write autocompletes' option under _options->tags_ is removed

- you now can enable the following potentially very slow and expensive searches on a per-tag-domain basis:

- - you can permit namespace-autocompleting searches, so 'ser' also matches 'ser*:*', i.e. 'series:metroid' and every other series tag

- - you can permit 'namespace:*', fetching all tags for a namespace

- - you can permit '*', fetching all tags (╬ಠ益ಠ)

- '*' and 'namespace:*' wildcard searches are now significantly faster on smaller specific tag domains (i.e. not "all known tags")

- short explicit wildcard searches like "s*" now fire off that actual search, regardless of the 'exact match' character threshold

- queries in the form "*:xxx" are now replaced with "xxx" in logic and display

- improved the reliability of various search text definition logic to account for wildcard situations properly when doing quick-enter tag broadcast and so on

- fixed up autocomplete db search code for wildcard namespaces with "*" subtags

- simplified some autocomplete database search code

- .

- string processing:

- the new string processor is now live. all parsing formulae now use a string processor instead of the string match/transformer pair, with existing matches and transformers that do work being integrated into the new processor

- thus, all formulae parsing now supports the new string splitter object, which allows you to split '1,2,3' into ['1','2','3']

- all formulae panels now have the combined 'string processing' button, which launches a new edit panel and will grow in height to list all current processing steps

- the stringmatch panel now hides its controls when they are not relevent to the current match type. also, setting fixed match type (or, typically, mouse-scrolling past it), no longer resets min/max/example fields)

- the string conversion step edit panel now clearly separates the controls vs the test results

- improved button and summary labelling for string tools across the program

- some differences in labelling between string 'conversion' and 'transformation' are unified to 'conversion' across the program

- moved the test data used in parsing edit panels to its own object, and updated some of the handling to support passing up of multiple example texts

- the separation formula of a subsidiary page parser now loads with current test data

- the string processing panel loads with the current test data, and passes the first example string of the appropriate processing step to its sub-panels. this will be expanded in future to multiple example testing for each panel, and subsequently for note parsing, multiline testing

- added safety code and unit tests to test string processing for hex/base64 bytes outcomes. as a reminder, I expect to eliminate the bytes issue in future and just eat hashes as hex

- cleaned up a variety of string processing code

- misc improvements to string processing controls

- .

- the rest:

- double-clicking a page tab now opens up the rename dialog

- system:time imported now has quick buttons for 'since 1/7/30 days ago'

- all hydrus downloaders now accept percent-encoded characters in the query field, so if you are on a site that has tags with spaces, you can now enter a query like "simple%20background red%20hair" to get the input you want. you can also generally now paste encoded queries from your address bar into hydrus and they should work, with the only proviso being "%25", which is "%", when all bets are off

- duplicates shut down work (both tree rebalancing and dupe searching) now quickly obeys the 'cancel shutdown work' splash button

- fixed a signal cleanup bug that meant some media windows in the preview viewer were hanging on to and multiplying a 'launch media' signal and a shortcut handler, which meant double-clicking on the preview viewer successively on a page would result in multiple media window launches

- fixed an issue opening the manage parsers dialog for users with certain unusual parsers

- fixed the 'hide the preview window' setting for the new page layout method

- updated the default gelbooru gallery page parser to fix gelb gallery parsing

- updated the newgrounds parser to the latest on the github. it should support static image art now

- if automatic vacuum is disabled in the client, forced vacuum is no longer prohibited

- updated cloudscraper for all builds to 1.2.38

- .

- boring code cleanup:

- all final mouse event processing hackey is removed from the media viewers, and the shortcut system is now fully responsible. left click (now with no or any modifier) is still hardcoded to do drag but does not interfere with other mapped left-click actions

- the duplicates filter no longer hardcodes mouse wheel to navigate–whatever is set for the normal browser, it now obeys

- cleaned up some mouse move tracking code

- clicking to focus an unfocused media viewer window will now not trigger the associated click action, so you can now click on archive/delete filters without moving on!

- the red/green on/off buttons on the autocomplete dropdown are updated from the old wx pubsub to Qt signalling

- updated wx hacks to proper Qt event processing for splash window, mouse move events in the media viewer and the animation scanbar

- cleaned up how some event filtering and other processing propagates in the media viewer

- deleted some old unused mouse show/hide media viewer code

- did some more python imports cleanup

- cleaned up some unit test selection code

- refactored the media code to a new directory module

- refactored the media result and media result cache code to their own files

- refactored some qt colour functions from core to gui module

- misc code cleanup

next week

I will be taking my week vacation after next week, and I don't want to accidentally create any big problems for the break, so I will try to mostly do small cleanup work and bug fixes.

R: 586 / I: 93 / P: 1 [R] [G] [-]

Q&A Thread: For simple questions that don't need their own thread

Here you can ask questions so that the board is not clogged with small threads.

Old thread >>9327

R: 377 / I: 89 / P: 1 [R] [G] [-]

Bugs Thread

BUGS THREAD

R: 2 / I: 0 / P: 1 [R] [G] [-]

Site Request

I'm a new user to Hydrus and I'm kind of confused about how you create the images to add a new downloader. If they can, can someone either tell me how to make it or even better post an image for it. The website I'm requesting is "https://booru.allthefallen.moe/". Call me what you want.

R: 721 / I: 204 / P: 1 [R] [G] [-]

Suggestions


Drag and drop windows with tag rules. Show two windows side by side and one window can be programmed with the rule "ADD tag foo" and the other one has the rule "REMOVE tag foo, ADD tag bar" and you can drag and drop files to them.

Deriving tags from regex of other tags/namespace tags. A file has the tag "filename:big_ugly_name" and we could regex that namespace for another tag.

Tag sets with hotkeys: save a set of tags under a hotkey so it's quick to add them to a file while filtering

Opaque window behind tag list in the corner so it doesn't get hidden by picture background

Option to default certain mime types to be excluded from slideshow and only open externally, will help with videos with odd codecs that don't preview in the slideshow correctly

Option to specify hamming distance in "find similar images", you can't change the option once it's in the filter window and you have to enter the hash manually in the "system:similar to" option
R: 74 / I: 23 / P: 1 [R] [G] [-]

Parsing scripts

How about a thread for discussing/creating/sharing parsing scripts?

I made one for md5 lookup on e621.net (actually I just modified Hydrus_dev's danbooru script). Let me know if I did anything wrong with it, I'm pretty clueless… but it seems to work fine.


[32, "e621 md5", 1, ["http://e621.net/post/show", 0, 1, 1, "md5", {}, [[30, 1, ["we got sent back to main gallery page -- title test", 8, [27, 1, [[["head", {}, 0], ["title", {}, 0]], null]], [true, true, "Image List"]]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-general"}, null], ["a", {}, 1]], null]], ""]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-copyright"}, null], ["a", {}, 1]], null]], "series"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-artist"}, null], ["a", {}, 1]], null]], "creator"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-character"}, null], ["a", {}, 1]], null]], "character"]], [30, 1, ["", 0, [27, 1, [[["li", {"class": "tag-type-species"}, null], ["a", {}, 1]], null]], "species"]], [30, 1, ["we got sent back to main gallery page -- page links exist", 8, [27, 1, [[["div", {}, null]], "class"]], [true, true, "pagination"]]]]]]

R: 10 / I: 2 / P: 1 [R] [G] [-]

Version 397

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v397/Hydrus.Network.397.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v397/Hydrus.Network.397.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v397/Hydrus.Network.397.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v397/Hydrus.Network.397.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v397.tar.gz

I had an ok week. I mostly did small fixes and improvements.

highlights

I added 'system:has/has no note with name xxx' to search for files with specific note names. This is bundled into the same dialog as last week's 'number of notes', for the new generic 'system:notes' entry in the normal system predicates list.

When you have many page tabs in a row, they will now middle…elide their name text to try to fit better. You can turn this off if you like under options->gui pages.

When files in the duplicate filter are pixel duplicates, the file size and import time comparison statements now have no 'score' and colour blue. Pixel duplicates are often a different judgment call, so until I can get more flexible scoring options here, I am hardcoding it to be less visually green/red.

If you need to run a gallery download with a search term that includes '#', this should now work.

I updated the IPFS code to work with the new IPFS 5.0 daemon. I think I have things mostly correct, but the new daemon has some new CORS restrictions that may need configuration on the IPFS side, depending on your situation, to give hydrus permission to talk to it.

The left page panel's position should set itself more reliably for users who start the program minimised to system tray. If you have had trouble here, please let me know how you get on. I believe the changes here will also reduce some UI lag for many other page adding and moving operations, particularly for large sessions.

full list

- regular changelog:

- added 'system:has/has no note with name xxx' to search for specific note names

- in the normal system predicate list, the notes pred is now the generic 'system:notes' to launch a combined dialog for both num notes and named notes

- favourite tag suggestions are now sorted in manage tags dialog according to the default tag sort

- page names will now middle…elide when there are too many to fit into a row (and normally left/right buttons would be added). if the elided tabs still do not fit, the buttons will pop up as before. added a checkbox to options->gui pages to turn this text eliding off

- pulled the 'page name' options on that panel into their own box and added some text regarding the 'my big row of import page tabs keeps scrolling weird' issue

- when files are pixel duplicates, the filesize and age comparison statements will now have 0 score and thus be coloured neutral blue

- the standard text entry dialog now always selects any default text it starts with, so you can now type to immediately overwrite. see how you like it and if there are some places where you think an exception should be made

- updated the IPFS interface to work with the new IPFS 5.0. all api requests are now POST so it doesn't 405, and the User-Agent is overridden to one that IPFS will not 403 at, and I fixed a typo the new api is more strict about

- a hack to get page splitters to lay out correctly on session load is rewritten from a hammer to a scalpel. pages now set their splitter positions on their first individual visible selection. this both reduces some minor ui lag on session/page load and improves splitter positions for clients that open minimised to the system tray

- a long-time odd issue where loaded sessions would initially select the top-left-most non-page of pages is fixed. now the bottom-left-most page of any kind is selected

- fixed tag autocomplete selecting the bottom-most pre-loading result. it now correctly selects at the top

- fixed an issue setting certain values (typically loading a default) to a tag import options panel

- the client is now more aggressive about clearing subscriptions from memory when they are finished running

- in windows, the main method that copies files now checks for modified time of the source file. if it is before 1980-01-01 UTC, it does not copy the file metadata, as some Windows has trouble with this lmaoooo

- cleaned up how some thumbnail 'current focus' media determination code works. should have fixed some weird errors when hitting certain shortcuts on collections

- cleaned up basic list/sort code across the program

- the 'queue' and add/edit/delete listboxes now emit change signals when new items are added or imported

- pyparsing, a helper for cloudscraper, is now correctly bundled in the built releases. a new line in help->about displays this

- help->about now lists cloudscraper version

- updated the discord link to the new https://discord.gg/wPHPCUZ

- .

- upcoming string processing changes for advanced users:

- I extended string parsing code this week, but I am not yet ready to turn it on. when it does come on, it will change all formulae from the fixed string match/converter pair a combined general string processing 'script' of n steps

- wrote a new 'string splitter' object that takes one strings and splits it into up to n strings based on a separator phrase (such as ' ,')

- wrote an edit panel for string splitters

- wrote a new 'string processor' object that holds n ordered string match/converter/splitter objects and filters/converts/splits x strings into y strings based on those steps

- wrote an edit panel for string processors. it has a notebook that live updates with test results for each step on every update

- wrote unit tests for string match

- wrote unit tests for string converter

- wrote unit tests for string splitter

- wrote unit tests for string processor

- refactored string conversion edit panels to their own file

- refactored string conversion controls to their own file

- misc string processing cleanup and labelling improvements

- .

- technical url parsing stuff:

- urls are now stripped of leading and trailing whitespace during normalisation, just in case a paste contains some extra whitespace. previously, it would sometimes throw a 'doesn't start with http' error

- the hydrus url normalisation process now normalises the hostname according to the NKFC unicode format, meaning unusual characters like ?and e◌ are now replaced with their normalised visual equivalent ? and é, and hence these urls will no longer throw errors when they are added

- if '?' or '#' end up in a hostname (which are invalid characters), it is now converted to _, just to stop complete parse mangling when weird urls are submitted. this character replacement may become more sophisticated in future

- the hydrus downloader should now support search terms that include '#'

- download query parameters that contain '%23' ('#', encoded) are now not unquoted in url normalisation

next week

Next week is a cleanup week. I will push on more wx->Qt code cleanup. I would also like to try to turn on some of the background text/notes work I have been doing for the past couple of weeks.

I will be taking my week vacation in two weeks, in the 399->400 gap from the 27th to the 3rd.

R: 0 / I: 0 / P: 1 [R] [G] [-]

can we talk about this???

why i'am prolife for babies children

please watch this video link video:https://www.youtube.com/watch?v=8nhXQS5UUGQ

R: 16 / I: 4 / P: 1 [R] [G] [-]

Version 396

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v396/Hydrus.Network.396.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v396/Hydrus.Network.396.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v396/Hydrus.Network.396.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v396/Hydrus.Network.396.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v396.tar.gz

I had a good week. The client now has better file 'notes' support.

There are several database updates this week. Users with large databases can expect the first boot to take a few minutes.

notes

A long time ago, I added prototype 'notes' to files. You can access it under a media right-click->manage->notes. It was a bit hacky, functioning just like a plain .txt attached to files, but some users really picked it up. This week it becomes more mature.

Firstly, files can now have multiple notes, each with their own name. Existing notes will get the simple default name of 'notes'. Also, files that have notes now show a notes icon on their thumbnails and in the media viewer top-right hover window. Clicking the hover window icon quickly opens the edit notes dialog.

And to find files with notes, there is now 'system:number of notes'. Its panel has a couple of quick buttons for finding files with any notes or no notes. Notes are stored in a faster and more space-efficient way, so this should work fairly quickly for most situations.

The next step here is to add note parsing to the downloader system. The main thought is to get artist post text and similar imported with nice 'blah gallery artist comment' style names as you download files. I had hoped to figure something out for this this week, but I didn't have time to do it properly, so I focused on polishing the above. I would like to fold parsing in in upcoming normal weekly work.

The new storage system for notes also builds a fast search cache for their text. It will not be extremely difficult to add some sort of 'system:note has text xxxxx' in future as well, if you want to search for parsed mega links or whatever else. Beyond that, I'd love some note 'preview' on the media viewer and comprehensive import/export/Client API support for notes.

unusual tag searches

I am really happy with the new tag autocomplete system. It is so much nicer to work on now it is all unified. This week I fixed a bunch of unusual searches.

A good example is '/f/', which according to hydrus 'special character' rules also matches 'f'. Short text queries like 'f' and '/f/' that do not do full lookups will now match all special character results, including if they have a namespace, so all combinations of 'board:/f/' and '/f/' and 'board:f' and so on will now match 'board:/f/'.

Also, wildcards are now unified. 'Complex' wildcards that have inserted *, like 's*s ara*', are now treated logically the same way as 'simple' ones that only have an * on the end like 'sam*', and should more reliably return all the correct results. They ~also~ support the special character rules as above, so 'int*fies' should match '[intensifies]'.

And wildcard file search predicates, where you might search for 'character:samus* (wildcard search)' to return all files that have any tag that matches that wildcard, now support the special character rules in all cases. When you include special characters in the wildcard text, such as 'e*mple-tag*', that '-' character is now preserved (previously, it was being replaced with space, which is behind-the-scenes hydrus technical trickery).

A new tag search cache supports and speeds up these logical improvements. Users with large databases can expect the first boot into 396 to take a few minutes.

the rest

I fixed the sort in the tag suggestion boxes!

You can now click the 'inbox' icon in a media viewer top-right hover window to archive a file.

The file->shortcuts dialog now has descriptive, sorted names for the reserved shortcut sets.

'system:dimensions' now has some quick-click buttions for common ratios and resolutions.

'system:known url' is better about fetching both www and non-www versions of URLs.

full list

- notes:

- the file notes system is more mature. files now store multiple named notes

- the edit notes ui is now a tabbed window with add/edit_name/delete buttons

- media results now load with their notes, so note access is instant

- thumbnails now show a notes icon when they have notes

- the media viewer top-right area shows a notes icon when the current file has notes

- clicking the media viewer top-right notes icon opens edit notes

- the edit notes menu entry now lists the number of current notes if there are notes

- added a 'system:number of notes' predicate. it has easy 'has/no notes' buttons for quick filtering

- the file notes database table will be updated on update, it shouldn't take long. existing notes will get the default 'notes' name

- duplicate notes now share the same storage space in the database

- in prep for a future search expansion, notes are now cached in the database for fast text search

- in prep for note parsing, wrote a 'note import options' object. it doesn't do anything in the program yet, but it supports multiple note conflict resolutions, note extension detection, and global and specific note renaming

- wrote unit tests for the new note import options

- .

- some tag search stuff:

- hydrus now maintains an internal mapping of direct 'searchable' versions of tags to the tags themselves, which allows it to now do fast exact-match (short search) and complicated wildcard lookups of tags with unusual characters. 'f' and '/f/' will now return '/f/' and 'board:/f/' quickly, 'board:f' and 'board:/f/' will return 'board:/f/' quickly, and 'te*a*' will correctly return 'test-tag'

- it will take a few minutes to regenerate this new cache on update

- complex wildcards like 's*m*' are now treated the same as simple ones like 'sam*' and should match unusual subtag characters in all cases

- wildcard tag file search predicates are now plugged into the new cache, so the search preds '/f/*', 'board:/f/', 'board:/f/ast;', 'b*d:/f/' and 'b*d:/f/*' now all match files with 'board:/f/', as do wildcards that include replacement characters, so the same should be true above for 'f' instead of /f/' in all cases

- new wildcard search preds do not collapse their characters for their presentation string, so 'date:2*-01-01' now renders like that, not 'date:2* 01 01'

- wildcard file search predicates are now faster for simple (just an asterisk on the end) subtag wildcards

- the fts search cache is moved from 'master' to 'caches' db this week, it will take a few moments on update

- the 'repopulate tag search cache' db regen job now repopulates the fts cache, the new 'searchable' cache, and the integer tag cache

- the database repair code now checks for the fts cache and new 'searchable' cache on boot and, if they are missing, warns the user and creates empty tables

- .

- improvements:

- fixed the unsorted tags in tag suggestion boxes

- clicking the inbox icon in the top-right hover window now archives the file

- system:dimensions now has quick buttons for 16:9, 9:16, 4:3, 1:1, 1080p, 720p, and 4k

- system:known url searches are now better about fetching www and non-www urls for the domain or url class

- the edit shortcut sets panel now has nicer english names for reserved shortcut sets, and also sorts them in a more logical way

- you no longer have to be in advanced mode to copy file hashes from thumbnails

- users in advanced mode can copy the internal file_id of files from the thumbnail/viewer copy menus (this is most useful for the client api)

- system num_frames, num_words, and num_notes now display alternate 'has/no xxx' labels when they search for =0 or >0

- you can now search for 0 with system:num_frames

- .

- fixes:

- users who could restore from system tray using the menu but had trouble with clicking _should_ now have better luck with clicking

- fixed some instances where fps could be calculated as 0, which would lead to other problems down the line. now a missing or 0 fps is remapped to 1

- fixed system:framerate for '<' queries

- the status bar cells now get expanded tooltips to describe what they do

- fixed some media result caching code that could in rare cases cause an error in content update processing when the result disappeared from the cache during processing

- the 'hard-replace siblings and parents' button on 'manage tags' now makes a submenu so its actions' long labels show better

- fixed a handful of tables that were not starting sorted

- a variety of credential parse and other server failures that were formerly returning 403 now properly return 400 and 409

- in order to improve default 'open externally' behaviour on Linux/macOS, if the environment variable XDG_DATA_DIRS is not preserved through a hydrus build launch env, hydrus now sets a simple 'default' value for this before running xdg-open

- if the client is booted from a windows shortcut to a built release, the program restart command is slightly more reliable

- .

- misc:

- cleaned up some db update error reporting code, it should now more reliably make an english-friendly popup text box before splurging technical info

- refactored some media object code, cleaned some class definitions, and added typing hints

- misc code cleanup

- the 'getting started' help files now have anchor definitions, so their sections can now be #linked to

- added several links in the 'getting started' help to the user-created video guides here: https://github.com/CuddleBear92/Hydrus-guides thank you for making these!

- added a link to the help for the user-made 'other archiving software' guide here: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/wiki/0-Alternative-Programs-and-Resources#software thank you for making this!

- fixed link to AUR package in the help

- updated cloudscraper in all builds to 1.2.36

- updated windows mpv to a significantly newer dll, it now reports api version 1.108

- included libgpg-error.so.0 in Linux build, which will improve some Linux situations (more reports from Ubuntu 20.04 or others about missing/conflicting .so files are welcome)

next week

Next week is a small jobs week. I want to catch up on bugs, see if I can move notes parsing forward, and also (as part of that) improve our text match/transform tools so complicated texts and filenames are a bit easier to parse. The new IPFS daemon is also broken for hydrus, so I will also be looking at that.

R: 13 / I: 5 / P: 1 [R] [G] [-]

Paying someone to tag all of my files

I'm dead serious. $10 for 10 Gb worth of files (Price is non negotiable). Will send them to you via MEGA.

Post your discord ID here if you're interested.

I'll give you more instructions on how to tag them in DMs, like which tags to prioritize etc…

R: 1 / I: 0 / P: 1 [R] [G] [-]

Your discord server link isn't working.

R: 4 / I: 0 / P: 1 [R] [G] [-]

Version 395

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v395/Hydrus.Network.395.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v395/Hydrus.Network.395.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v395/Hydrus.Network.395.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v395/Hydrus.Network.395.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v395.tar.gz

I had a good work week. I fixed a variety of bugs, improved the feel of the suggested tags columns, and made zooming in the media viewer nicer.

suggested tags quality of life

I recently did some work on the different 'suggested tags' panels the manage tags dialog can support. This week I worked on some of the feedback from that pass, hopefully making it a bit easier to get what you want done. As a reminder, you can set these columns up under options->tag suggestions.

The columns are now 'synced' with the current media. They all refresh/repopulate reliably when you move to another media in the media viewer. Also, all columns now hide tags that all the selected media already have, and they update this ~as you edit~, so if both 'favourites' and 'related' suggest 'character:samus aran', and you add that through any means, both lists will remove that tag! Furthermore, if you remove a tag that one of those lists would show, they now re-show that tag instantly, without having to do any new lookup.

Furthermore, the tag lists now try to retain the selection location when you add a tag through the keyboard enter key, so even though a tag you hit enter on will disappear, the focus should stay on the tag above or below. I hope this makes it easier to work with these columns with just the keyboard. As a second remind to keyboard users, you can currently set up some slightly mickey-mouse shortcuts to move focus around here under the 'main_gui' shortcuts, the actions starting 'show_and_focus_manage_tags_…'.

zoom center

When you zoom a file in or out, there is a 'centerpoint' about which the zoom happens. This position 'stays still', while everything else grows or shrinks around it. Until now, this has been hardcoded on the media center, which was sometimes unhelpful when the media had been panned. This centerpoint is now customisable under options->media. It now defaults to the media window centerpoint, which means the image will always grow and shrink around the center of what you are looking at. This makes it much nicer to pan and zoom in the duplicate filter. You can change it back to the media centerpoint, the media top-left corner, or even the current mouse cursor position, for RTS-vidya-like zooming.

fixes

'character:aran' matches 'character:samus aran' again in the Windows build (or anything else running python 3.7). It was a regex library version issue.

The hover windows in the media viewer are now aware of the new shortcuts system in Qt and should pass uncaught regular keyboard shortcuts up to the media viewer. So, if you click the top-right hover to set a rating or the center-right duplicates hover to do a skip, you should now be able to hit custom keyboard shortcuts to add tags or set ratings and so on without having to click back on the media viewer. Where this will not happen are simple things like the taglist on the left eating up/down arrow keys to do list navigation and a handful of Qt-hardcoded shortcuts like tab and space, which can intercept and do panel navigation and currently-focused-button pressing.

If your mouse has back and forward buttons, these should now be caught by the shortcuts system, for those shortcut sets that can currently do mouse events. I assume there is no such thing as a double back-click, but let's see what happens!

The client now tries to avoid large analyze and vacuum database maintenance jobs in the normal maintenance cycle. Too many users in unusual situations have been hit by unreasonably big jobs here, and the benefit is not worth it.

Dialog messages that intercept ok and cancel events, the 'hey, are you sure you want to cancel, there is some uncommitted stuff, yes/no' sort of thing, now have improved logic. They fire off in better order and shouldn't ever bother you twice in one action.

deleted tags overwrite

This is an important bug fix, but it is technical and not a priority for non-advanced users for now.

Some users recently reported some unusual deleted mappings counts on the PTR, not matching that of other users. I also had a job to check that deleted tags were being filtered correctly out of local tag domains, so I went into it this week. To my mild horror, I realised that deleted tags were not being filtered out of tag parsing at all, so any time files were being re-parsed, any previously deleted tags were being overwritten. In the case of tag repositories, deleted tags were still being discarded at the server level (and hence not overwritten for other users), but the overwrite was happening to the local client that did the 'late' reparsing.

I regret this oversight, and I am sorry for the transactions of inconvenience and difficulty this has likely caused, even without us ever realising directly what was going on. The issue is unusual, with numerous technical caveats that make it apply to different users in different ways, but I believe it has not hit most users too much. It is also retroactively fixable for tag repositories.

The parsing is now fixed. Tag import options will now filter out the currently deleted tags from their importee file's parsed tags. If you wish, you can force an overwrite using the tag import options cog menus. Hard drive imports and tag migrations continue to overwrite deleted tags, as these are still considered 'human, manual' actions that can take responsibility for overwriting a deleted tag.

For tag repositories, particularly the PTR, a 'reprocess content' run will fix the historical record, re-deleting things that should be deleted. Advanced users can do this now from the review services panel if they wish. This is an expensive job, so I have not queued it up automatically in today's update. I think I will write a separate targeted maintenance routine for it in the coming weeks.

full list

- some more suggested tags fixes/qol:

- favourite tags now correctly refreshes on new media

- the tag suggestion lists in manage tags now discard current and pending tags that _all_ the current media already have, and all tag suggestion lists update this filter any time the media gets a tag content update! they _should_ update live now

- all tag and predicate taglists now try to move the selection to a 'nice' neigbour when a keyboard enter activation results in the current selection being removed (e.g. as in these tag suggestion lists). the nice selection should be the tag after, before, or at the top of the list, and should make it nicer to keep navigating the list and add tags with your keyboard

- all tag and predicate taglists now try to preserve selection on simple clear-and-set data refreshes

- .

- deleted tags overwrite update:

- due to an unfortunate oversight, until now tag parsing has not filtered out previously deleted tags from the tags it parses and sends to the local database

- as the majority of downloaded files are parsed once per site per user and in a similar time window before manual editing ever occurs, and most non-tag-sibling-eligible bad tags are site specific or not parsed to begin with, and as these undesired tags were not broadcast up to the tag repository, this problem has not been very obvious and I believe has not affected most users too much. this is however a reason why some users who have more recently downloaded many older files are seeing smaller 'deleted mappings' counts on their ptr review panel (and some low quality tags in their db), as they have been re-adding previously deleted tags to their local store

- this has been fixed. tag import options now load the pending importee file's metadata before tags are filtered and discard currently deleted tags from those to be added or pended. this applies to parsed tags, additional tags, and those tags added through special other means, such as from a parent gallery page.

- if you do wish to allow parsed or additional tags to overwrite currently deleted tags for a particular job, the cog icons on the edit tag import options panel now allow you to permit overwrite for either

- tags added via hard drive imports or the migrate tags tool still overwrite deleted tags as before

- as this is a local-only problem, there is thankfully a retroactive fix for this issue for tag repository domains, involving a content reprocess run to re-apply deleted tags. I am not activating this automatically this week as this is a heavy job for the ptr and I need to study the true fallout of the problem more, but I may in future, likely as a smaller and more targeted maintenance job. advanced users can do it now under the ptr's review services panel

- I regret missing this, and I am sorry for any inconvenience. I only discovered it through the serendipity of some users recently reporting unusual deleted counts and a personal item in my todo to check the reliability of deleted mapping filtering for local tag domains–turns out it never got added, and we never specifically noticed, fugg

- there are now unit tests for the improved tag filtering pipeline and both of these new overwrite options

- .

- the rest:

- hydrus can now use several different zoom 'centerpoints' about which to expand and shrink a zooming file. this was previously hardcoded to the center of the media. under options->media, you can now set it to be the media window center (the new default, which feels much nicer after a pan), the mouse cursor, the old media center, or the media top-left corner

- cleaned up the related zoom positioning code, and removed the jarring old re-centering off-screen rescue hack when zooming out to canvas zoom

- added a warning about big zooms to the media options page

- fixed tag autocomplete filtering in python 3.7 so 'character:aran' matches 'character:samus aran' again

- when the hover windows on a media viewer have focus, they _should_ now pass up all options->shortcuts shortcuts to the media viewer

- mouse back/forward buttons _should_ now be supported in the shortcuts system, as much as your OS allows them to work like regular clicks

- fixed a rare crash with the 'clear trash' button

- the client will now not re-analyze tables that have been previously scanned with at least 100k rows in the normal 'soft' maintenance cycle, as this is an expensive operation with limited benefit

- the client will now not vacuum database files greater than 1GB in the normal 'soft' maintenance cycle, as this is an expensive operation with limited benefit

- the new 'cannot vacuum because xxxx' log entry is now only ever printed once per boot. however due to the above change, it likely won't appear in the normal maintenance cycle anyway now

- cleaned up some vacuum code

- reworked the panel system to better test data validity vs 'woah, you sure you want to do this?' tests and generally cleaned and simplified the canok/cancancel/isvalid testing logic for all panels. panels like manage siblings will now not produce two message boxes if you try to ok them on an uncommited pair and then back out of the ok

- refactored the top level window code and improved scrollable panel code typing

- more standalone gui function code refactoring

- fixed a click-selection-test bug when clicking on certain whitespace in certain predicate lists

- the text of the cloudflare-specific error when encountering a captcha page is improved

- cleaned up some tag list menu copy and select code, both the menu labels and the copy action, for unusual tags. the 'copyable tags' fetching code is now flexible and unified for menu and action

- cleaned up the taglist sibling copy code, eliminating the chance of dupes

- fixed a _little_ of the wording on the discard/exclude tag list menu labels for negated predicates, it still feels a bit awkward and I will keep working here

- cleaned up some old media metadata fetching code

- misc import code typing

- misc list/iterable typing improvements

- added some misc media-tag tool code

- unified the tag import options tag filtering pipeline somewhat to deal with the deleted overwrite situation

- improved a debug ui test to no longer need window focus

- misc help cleanup

next week

Next week is a 'medium-sized' jobs week. I would like to finally get to an expansion of the prototype 'file notes' system. I would ideally like multiple named notes, note parsing support (so you can pull an artist comment from a web page), note preview on the media viewer background, and note import/export. This is a ton of work, so I don't expect to get it all done in one week.

R: 3 / I: 0 / P: 1 [R] [G] [-]

Version 394

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v394/Hydrus.Network.394.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v394/Hydrus.Network.394.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v394/Hydrus.Network.394.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v394/Hydrus.Network.394.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v394.tar.gz

I had an ok week mostly cleaning code.

The bug from last week where subscriptions that had incomplete file queues were unable to start is fixed. I apologise, this was a typo that was not caught by my code checker or my testing routine.

autocomplete cleanup

The tag autocomplete code has seen many updates and expansions over time, and the code had become a mess of hardcoded hooks and logical duct-tape. This week I went into it and pulled all the text parsing and results caching code out to single separate dedicated objects. Essentially most of the big changes this week are all behind the scenes, and you can check the changelog for more detailed info, but multiple small bugs and odd situations with autocomplete entry are now fixed.

A user recently contributed code that parses system predicates from text input. Now I have this nice new parsing system, I hope to integrate this new feature soon.

If you encounter new bugs or certain complicated wildcard lookups that do not work, please report them–it is now much easier for me to fix this stuff and support new search types.

environment changes

If you patch my source code, please note that the source structure has changed. The single flat directory of py files is now split into some basic subdirectories. This is one more step forward in getting the structure more manageable, and I expect to do more in future cleanup weeks.

full list

- autocomplete cleanup:

- the text you type into tag autocomplete is now parsed in a unified object. all the variants of empty text, invalid text, valid text, namespace text, and wildcard text are all tested and fetched in one simple location with better code

- autocomplete results caching is now a unified object that tracks and filters results in one location. wildcard searches are now never cached by accident, and switching from tag cache to system predicate cache and to non-initialised cache is instant and more reliable

- when an autocomplete, either in a search page or a context that manages tags, has results include multiple sibling variants of the typed text, they are now all elevated to the top of the list. the ideal is at the top, the entered text is next, and any known siblings follow

- the search character 'collapse' that ensures quote marks and hyphens and other odd characters are unified across tags now applies uniformly to all non-complicated-wildcard search tags, with namespace not collapsed and subtag always collapsed

- when entering an explicit wildcard search, both strict and autocomplete versions (whether they end with an asterisk) are now displayed

- the way tag results are filtered is now more accurate for some unusual wildcards

- it is now more difficult to slip cpu-killer search tags (weird asterisk combinations) through

- the quick-broadcast that happens when the user hits enter before any results have started loading now uses the unified object and chooses a safer and more reliable broadcast value. the test whether to do the quick-broadcast is also more reliable, particularly in unusual situations where a recent search was cancelled or delayed. note that for many users, the cache and search tech is fast enough that this very rarely triggers

- searching with a wildcard below the autocomplete threshold can no longer trigger a full search, nor an invalid exact-text search

- namespace count merging is now unified across db tag fetches and media fetches

- include current/pending buttons now filter down to media-based tag autocomplete counts

- namespace tag autocomplete queries will no longer show up some unusual siblings below the 'anything' tag

- deleted a whole bunch of old a/c and caching code

- added comprehensive unit tests for the new parsed autocomplete text object

- added comprehensive unit tests for the new predicate results cache object

- .

- the rest:

- fixed a stupid typo bug in the new domain checking code that was stopping subscriptions with incomplete file queues from starting. I apologise for this

- network error responses 502 (Bad Gateway) and 503 (Service Unavailable) are now treated as a retryable. the 503 is assuming it is not a CF challenge page. if they fail all retries, they are considered a network infrastructure error

- all other misc 5xx http responses are now treated as instant network infrastructure errors and will be logged in the new domain health tracker

- the exit splash screen now opens a bit earlier, so you now shouldn't have any momentary uncertainty where no windows are open

- clients that start minimised to system tray _should_ be better about restoring splitter positions on first show

- the various 'management panels', the panels on the left of main gui pages, now have smaller minimum width where available. the gallery and watcher panels are still the widest, which is a limitation of the current list tech. when it gets better column sizing code and selection memory, this will improve

- fixed an issue loading gifs with some OpenCV versions

- brushed up some running from source help

- deleted the Py2To3 script that attempts to detect a legacy python 2 install

- improved all the gui files' import order

- cleaned up and refactored some subscription code

- added a bunch of type hints to edit panel code

- misc code cleanup

- .

- environment updates:

- did second step of hydrus project structure improvement–now the project is split into subdirectories for core/client/server/misc and some client subdirs. work here will continue

- linux build gets some new libraries, cv is up to 4.2.0

- it isn't important, but hydrus is now built in python rather than directly from command line. my build scripts now include cloudscraper and the new hydrus source code tree in the build as they are, rather than hardcoded copying

next week

I would like to catch up on small jobs and bug fixes. I feel pretty exhausted due to IRL stuff, so I want to take my foot off the throttle a little. I hope you are hanging in there and staying healthy.

R: 10 / I: 2 / P: 1 [R] [G] [-]

Version 393

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v393/Hydrus.Network.393.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v393/Hydrus.Network.393.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v393/Hydrus.Network.393.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v393/Hydrus.Network.393.-.Linux.-.Executable.tar.gz

EDIT: This release had a hotfix two hours after initial release to fix the CloudFlare code. If you got it early, please redownload. The links are the same, the build is new.

I had a good week catching up on messages and small jobs. There is also a (hopefully) neat prototype solution for some cloudflare issues.

cloudflare and network

CloudFlare hosts content for many sites online. They have a variety of anti-DDoS tech, normally a variation on a 'is this a web browser?' test, that sometimes stops hydrus from downloading. If you have seen unexplainable 503 errors on a site that works ok in your browser, this may have been it.

This week, I am adding a library, cloudscraper, to my network engine to try to solve this. Now hydrus will attempt to detect CF challenge pages when they are downloaded and pass them off to this new library, which attempts to solve the javascript challenge (the part where a CF site sometimes says 'this process may take five seconds…', just as a browser would, and then copies the solved cookies back into the hydrus network session.

This first version can only solve the simple javascript challenges. It cannot do the more serious captcha tests (which usually appear when a site is under active attack) yet, but this is a possible future expansion. I would appreciate any feedback from users who have had bad CF problems. If a page cannot be solved, or if your entire IP range had been flat-out blocked, hydrus should now attach new CF-appropriate error messages.

For users who run from source, this library is optional. You can get it through pip, 'cloudscraper'.

Additionally, I am rolling in the very first basic version of 'this domain is having trouble' tech to my network engine. Now, if you keep having connection errors or CF issues with a site, hydrus recognises this and slows down access for a bit. The default is that no new network job will start as long as its site has had three serious errors in the past ten minutes, but you can edit this rule, including turning it off completely, under options->connection. Subscriptions will also try to wave off sites having trouble, just as if you had hit bandwidth limits. I expect to extend this system significantly in future, particularly by adding UI to see and manage current domain status.

the rest

The Windows build of hydrus is now on a newer version of python, 3.7 instead of 3.6. This makes for a variety of small background improvements all over the place, and some updated libraries, but may also introduce an odd bug here or there in more rarely used hydrus systems. Let me know if you run into trouble!

I did another round of file search optimisations on searches that do not have tag search predicates, such as bare system:rating or system:inbox queries.

Thanks to some excellent work by a user, there is another new DA parser. This one gets full-size images, video, flash, and even pdf! The only proviso is that it needs to be logged in to DA to get most content, otherwise it 404s. I believe the hydrus DA login script works, but if you have trouble with it, Hydrus Companion is always a good fallback.

The blacklist in tag import options gets a bit of work this week. It now tests unnamespaced rules against namespaced tags, so if you have a rule for 'metroid', it will also block 'series:metroid'. This is a special rule just for the tag import options blacklist (it also tests all siblings of the test tag), and to reflect it, the tag filter/blacklist edit panel now has a second 'test' text to show how the test input applies to a TIO blacklist, as opposed to a regular tag filter.

This is a small thing, but if you are mostly a keyboard user, the mouse now hides on the media viewer without having to be moved!

If you have had crashes trying to open a file or directory picker dialog, try the new BUGFIX checkbox under options->gui. It tries to use a different style of dialog that may be better for you.

full list

- cloudflare and network:

- the hydrus client now has an experimental hook to the cloudscraper module, which is now an optional pip module for source users and included in all built releases. if a CF challenge page is downloaded, hydrus attempts to detect and solve it with cloudscraper and save the CF cookies back to the session before reattempting the request. all feedback on this working/breaking irl would be welcome. current expectation for this prototype is it can pass the basic 'wait five seconds' javascript challenge, but only a handful of the more complicated captcha ones

- if a CF challenge page is not solvable, the respective fail reason for that URL will be labelled appropriately about CloudFlare and have more technical information

- .

- the hydrus network engine now has the capability to remember recent serious network infrastructure errors (no connection, unsolvable cloudflare problem, etc..) on a per domain basis. if many serious errors have happened on a domain, new jobs will now wait until they are clear. this defaults to three or more such errors in the past ten minutes, and is configurable (and disableable) under options->connection. this will be built out to a flexible system in future, with per-domain options+status ui to see what's going on and actions to scrub delays

- basically, if a server or your internet connection goes down, hydrus now throttles down to limit the damage

- subscriptions now test if a domain is ok in order to decide whether they can start or continue file work, just like with bandwidth

- serverside bandwidth alerts (429 or 509) are now classified as network infrastructure errors

- I expect this system will need more tuning

- .

- the hydrus downloader system now recognises when an expected parseable document is actually an importable file. when this is true, the file is imported. this hopefully solves the situation where a site may deliver a post url or a file

- .

- the rest:

- the windows build of hydrus is now in python 3.7.6, up from 3.6. this rolls in a host of small improvements, including to network stability and security (e.g. TLS 1.3), and possibly a couple of new bugs in more unusual hydrus systems

- similarly, all the windows libraries are now their latest versions. opencv is now 4.2

- greatly sped up several file searches that include no tags such as bare system:rating, most system file metadata predicates, or bare system:inbox, when the result size is much smaller than the total number of files in the file domain

- thanks to some excellent work by a user, the Deviant Art downloader gets another pass–it can now get high res versions of images where they are available, and video, and flash, and pdf! the only proviso is that you need to be logged in to DA to get most content, otherwise you get 404. the current hydrus DA login script _seems_ to work ok

- tag import options blacklists now test unnamespaced rules against namespaced tags. so if you blacklist 'metroid', a 'series:metroid' will be caught and the blacklist veto signal sent. this can be escaped with the 'advanced' exception panel, which now permits you to add 'redundant' rules

- the edit tag filter panel now explains the blacklist rules explicitly and has a second 'test' green/red text to display test results for a tag import options blacklist, with the new sibling and namespace check

- added some unit tests to test the new tag import options blacklist namespace rule

- when 'default' tag import options are set, the edit panel now hides the per-service options, rather the the previous disable

- the system tray icon now destroys itself when no longer needed, rather than hiding itself. it should now be more reliable in OSes that do not support system tray icon hide/show. if your OS still doesn't get rid of them, and you get a whole row of them, I recommend just leaving it always on

- the system tray now has a tooltip with the main hydrus title and pause statuses

- the timer that hides the mouse on the media viewer is now fired off when the window first opens (previously it would only initiate on the first mouse move over the window), so users who navigate mostly by keyboard should now see their cursors nicely hide on their own

- added some semi-hacky import/export/duplicate buttons to edit shortcuts. I'll keep working on this, it'd be nice to have import/export for whole shortcut sets

- added a semi-hacky duplicate button to the 'manage http headers' dialog

- the 'clear' recent tag suggestions button is now wrapped in a yes/no dialog

- a new checkbox under options->gui now lets you set it so when new cookies are sent from the API, or cookies are cleared, a popup message summarises the change. the popup dismisses itself after five seconds

- the client api now also returns 'ext' on /get_files/file_metadata calls, just as a simpler alternative if the 'mime' is a pain

- fixed a bug when petitioning tags through the client api, with or without reasons

- fixed an error where subscriptions that somehow held invalid URLs would not be able to predict some bandwidth stuff, which would not allow the edit subs dialog to open

- the string transformation dialog's step subdialog is now ok with example strings that are bytes. even then, this str/bytes dichotomy is an old artifact of python 2 and I will likely clean it up sometime so string transformers (and downloaders) only ever work utf-8 and hashes just work off utf-8 hex

- added a BUGFIX checkbox to options->gui that tells the UI to use Qt file/directory picker dialogs, instead of the native OS one. users who have crashes on file selection are encouraged to try this out

- updated running from source help with cloudscraper, a new pip masterline, and some windows venv info

- the 'import with tags' button on 'import files' dialog gets another rename for new users, this time to 'add tags before the import >>'. it also gets a tooltip

- handled an unusual rare error that could occur when switching out a media player inside a media viewer, perhaps during media viewer shutdown

next week

Next week is a cleanup week. I'd like to go into the tag autocomplete code and clean up how the input text is parsed and how results are cached and filtered. Otherwise, I'll do more wx->Qt cleanup and catch up on some small jobs.

R: 2 / I: 0 / P: 2 [R] [G] [-]

I just discovered Hydrus - what are some of the best / most active tags to follow?

R: 1 / I: 0 / P: 2 [R] [G] [-]

Version 392

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v392/Hydrus.Network.392.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v392/Hydrus.Network.392.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v392/Hydrus.Network.392.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v392/Hydrus.Network.392.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v392.tar.gz

I had a mixed week, but I am happy with my work. Some large database jobs are faster and tag autocomplete results have better sibling support.

tag sibling database cache

I was not able to get as much done here as I had hoped, but I was able to finish the first step and fix some sibling bugs. The client now keeps a cache for fast sibling lookups, which will be a great tool going forward.

The tag autocomplete system now uses this new cache. When you type in the start of a tag, possible matches' siblings are added to the results list, even if they do not strictly match what you typed. These siblings used to be computed on the fly, but now it is looked up using fast pre-generated data. I have also improved the accuracy of the search and added 'pending' siblings in.

As I was working here, I also fixed some sibling presentation and search bugs. If you discovered that a 'series:blah' search was delivering some unexpected 'cosplay:blah' results, this should now be fixed. There is still more to do here, which I hope to naturally fix as I replace more old systems with nice new caches.

There are two next steps here: first to write a 'what file tags should look like with siblings collapsed' cache, which will speed up tag search and file load and also eliminate many of the uncertain counts you get with autocomplete results like 'character:samus aran (500-700)'. Second is to write a full-featured 'which siblings apply to this service?' options system, so you can say 'ignore the ptr's siblings, use mine instead' and other clever combinations, rather than the current 'always apply all to all, or one to one'.

the rest

e621 and sankaku get revised default downloaders, fixed for getting 'rating' and 'genre' tags respectively.

The system tray icon got some bug fixes. It is also more stable under non-Windows, but doesn't seem to hide reliably yet.

The hydrus server and the client's API and booru now try to host their services on IPv6 as well as IPv4. This is new stuff to me, so let me know if you come across any problems!

full list

- db-level tag sibling cache:

- the hydrus client db now maintains a fast cache of current+pending tag-to-ideal-tag sibling relationships. it works for specific services and 'all known tags'. this is a nice tool and the first step in having a proper hard-baked siblings mappings cache

- the new sibling cache can be regenerated under _database->regenerate_. the 'autocomplete cache' entry under that menu is also renamed to the now more appropriate 'tag mappings cache'

- the db repair system can regenerate this new cache if any part is missing on boot

- the lookup that finds tag sibling matches for autocomplete uses this and is now faster, specific to the searched service, more accurate about status, and now includes pending siblings

- wrote a new unified object to manage a collection of tag siblings, it is now in use at the db level

- as I continue to develop this new fast tech, the old 'apply all sibs to all services' option, which was always buggy, may sometimes not apply in it. I will ultimately replace it with a fuller per-service choice system that will work quickly and properly and in the same unified way

- fixed a bug where only one local tag service's siblings would be matched at the ui level when looking at 'all known tags'

- fixed a bug in the file search code where searching for a tag that had an unnamespaced sibling going to it would result in searching for all possible namespaces of that sibling (e.g. searching for 'character:samus aran' when 'samus aran'->'character:samus aran' sibling existed would result in effectively 'anything:samus aran')

- when tag services are deleted, they are now better about cleaning up their siblings and parents quickly

- optimised some tag and hash id->value database cache population routines to improve performance for large queries (e.g. when fetching all the tag parents/siblings on boot). also these caches are now larger, 100k instead of 25k

- all cache regen code now forces an immediate analyze of the new tables to speed up imminent access

- .

- the rest:

- updated the default e621 file page parser to get rating tags again (looks like their markup just changed again)

- updated the default sankaku file page parser to get their recently redefined 'genre' tags

- in edit subscriptions, the 'overwrite tag import options/check options' actions now initialise their dialogs with the current value for the first subscription, rather than the global program default

- in the edit subscription panel, the checker options button is moved down to the file/tag import options

- when not in advanced mode, the edit tag import options panel now has some red-text at the top to reinforce to new users that they should generally use the defaults

- the tag import options blacklist now secondarily checks against all known siblings of the parsed tags, rather than just the 'collapsed' ideal siblings

- subscriptions are now more aggressive about clearing out old urls from their file import caches–instead of clearing the 251st url after it has aged twice the death period, now they use just one DP. also, checkers with static checker timings will use five times that check period as DP if that is smaller. static checkers, or those that never die, will use a flat value of six months as DP if that is smaller

- moved a bunch of the debug 'data actions' to a new 'memory actions' menu

- significantly reduced how often the system tray regenerates its menu, which seems to improve stability

- fixed an issue where guis that were maximised before a minimise were restoring from a system tray icon click to normal view

- double-clicking the system tray when the ui is hidden should no longer do a fast show/hide

- fixed an issue where if the gui was minimised, the main animation timer would not run for other windows (e.g. a separate media viewer)

- improved ui shown/hidden tracking logic for the new system tray icon for different OSes

- fixed the 'refresh_page_of_pages_pages' shortcut action, which had faulty old wx code in it

- fixed a wx->Qt bug where modal popups that cannot be cancelled, and thus pop up a 'sorry, you can't dismiss this' text when you try to close them, were nonetheless still closing afterwards

- the hydrus client and server now attempt to listen their servers on both IPv4 and IPv6, failing gracefully if IPv6 is not available

- the 'is this a localhost request?' check now understands IPv6 localhost (::1 or ::ffff:127.0.0.1)

- may have solved a 100% cpu repaint issue with the a/c dropdown in some qt environments

- added info to installing help about Windows N and clean installs

- misc media viewer wx->Qt code cleanup

- misc code cleanup

- .

- experimental hellzone, be wary ye scabs:

- added an experimental 'sub-gallery url' url content type to the parsing and downloading system. this url is queued into the gallery log even if the primary gallery page found no file/post urls, and is meant for galleries that link to galleries. not yet ready for primetime, but feedback would be appreciated

- added an experimental ui-hang relief mode, activated under _help->debug->data actions->db ui-hang relief mode_, which _should_ stop the ui hanging in unusual long-time ui-synchronous db jobs. it may cause other problems, so it is default off. it also prints begin/end statements to log for additional info. users who experience ui hang due to db job processing time are invited to play with this mode and report back results

next week

Next week is a small jobs week. I fell behind on messages this week, so my main job is to catch up on those and then to clear out my bugs queue.

R: 7 / I: 0 / P: 2 [R] [G] [-]

Version 391

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v391/Hydrus.Network.391.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v391/Hydrus.Network.391.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v391/Hydrus.Network.391.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v391/Hydrus.Network.391.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v391.tar.gz

I had a great week. The client now has 'minimise to system tray' support, you can search by framerate and 'number of frames', and a bunch of bugs are fixed.

system tray

This first version is buggy/crashy under non-Windows, so I have disabled it for non-advanced mode non-Windows users for now. The icon menu doesn't work well for non-Windows, but double-clicking seems ok. I will keep working on this.

The hydrus client can now make a system tray icon for those OSes that support it. You can customise it under the new options->system tray panel. You can set the client to minimise to and/or close to system tray, and you can set the icon to always display if you like. There is also a new 'hide_to_system_tray' action under the 'globals' shortcut set.

Clicking on the system tray icon will raise a 'shown' client to the front or show a hidden client. Double-clicking or middle-clicking will do hide/show. The icon also has a menu to show/hide the client, pause/unpause subscriptions or network traffic, and exit the program.

You can also set the client to boot minimised into the system tray, but there may be layout issues once you do show the client–please let me know if you get any!

frames

You can now search for file framerate or 'number of frames' under an expanded 'system:duration' search predicate. It is difficult with the data hydrus currently knows to get very precise framerate, so system:framerate searches +/-5% of what you set. You can't differentiate between 30fps and 29.97fps yet.

You can now sort by number of frames.

Video framerate is now shown in the 'file info summary' text that displays on the status bar, thumbnail menu, and media viewer.

the rest

There are new fixed default danbooru parsers, and an e621 login script, both thanks to submissions by users.

I went over the tag autocomplete fetching code, cleaning up old areas and improving the timing and caching logic. It should be snappier, less flickery, and the bug where system predicates sometimes do not re-load after entering a search tag should be fixed. I expect to revisit this in three weeks for the next 'cleanup' week to further clean this semi-hellcode.

I also cleaned up some old wx->Qt stuff related to thumbnail sort. I believe I fixed the weird issue where certain pages (those that had just loaded a favourite search seemed to be one) would not accept certain sorts (sort by time imported was common here, I think because that is default secondary sort), and some collect/sort interactions are also faster and more reliable.

The tag suggestion boxes in manage tags now only do 'add' operations when you choose their suggestions (i.e. they won't remove an existing current/pending tag). They also remove those tags from their list when you do so. See if you like the feel of this new workflow. Also, a bug where some of these suggestion boxes would not acknowledge new pending tags (e.g. not using them for subsequent 'related' tags searches) after moving to a next/previous media in a manage tags off the media viewer is fixed.

If you ever got screwy URLs in your client (invalid garbage like 'tdunrotduenoh'), the program should be much saner about filtering and displaying and editing these. The Client API no longer accepts invalid URLs when associating URLs.

full list

- system tray icon:

- hydrus now can now make a system tray icon for those OSes that support it. it can be buggy/crashy under non-Windows, where it gets some warning labels

- under the new options->system tray page, you can set whether to show the system tray all the time, minimise the main gui to system tray, close-button the main gui to system tray, and start the program minimised to the system tray

- right-clicking the icon brings up a menu to show/hide the ui, pause/unpause network traffic or subscriptions, and to exit hydrus

- the main file menu now has an option to minimise to system tray

- double-clicking or middle-clicking the icon will show/hide the whole hydrus ui as long as there are no dialogs open

- clicking it will restore the main gui from minimise or raise it to the front

- on an ui hide, the current preview window will be blanked and media viewers will be paused, so any ongoing noise/cpu from them should stop

- a new 'global' shortcut 'hide_to_system_tray' is now available

- starting the client minimised may have some layout issues on first show–I particularly had to fix splitter layouts–please report any more you discover

- .

- framerate and num frames:

- system:framerate search added to system:duration panel. precise framerate is tricky with current hydrus info, so it searches +/- 5% of a given value

- system:number of frames added to system:duration panel

- sort by number of frames added

- duration/framerate/num frames sort moved to their own 'duration' submenu

- framerate added to generic media metadata summary string (which appears in status bar and media viewer, etc…). precise framerate is tricky with current hydrus info, so it is rounded to the nearest integer

- .

- the rest:

- rolling in new danbooru file page parsers that should fix file downloads, thank you to a user for the submission

- rolling in a e621 login script, thank you to a user for the submission

- gave tag autocomplete results fetch code a pass, cleaning up several instances of incorrect or inefficient timing and caching logic and I believe fixing the issue where system preds would sometimes not be loaded after entering a tag

- improved reliability of autocomplete dropdown hiding on background pages (some edge cases where these could still hang around _should_ be fixed)

- improved 'hide' tests in several parts of the program related to the new system tray icon, which should help some other cases–e.g. weird shutdowns now _shouldn't_ ever leave a bunch of floating popup messages

- fixed a bug where pages set to open with all known files/tags domains, which is not supported, was incorrectly substituting tag domain with a file domain, which is even more not supported

- cleaned up some sort code–I believe this has fixed the odd issue where a 'time imported' sort would not work on some pages (such as one loaded from a favourite search)

- fixed the 'related' tag suggestion box not knowing about new pending tags added in a manage tags dialog open on a media viewer after next/previous media transitions while the dialog is open. also it and the file lookup's lists now clear when a new lookup starts

- the tag suggestion boxes are now add-only and remove what you add as you add them! let me know if this feels nice or not!

- the splash window now has a different 'booting/exiting' window title, if you would like to hook it with a window manager

- went over all the 'prep url for display', 'filter urls', and 'normalise url' requests across the program to deal with invalid url (e.g. garbled text) better

- you can now no longer add invalid urls via the client api associate_url call–you'll get 400 instead

- cleaned some thumbnail selection and rendering code, particularly fixing some edge case 'where that media go?' issues where collect-by calls happen during thumbnail waterfalls and so on

- cleaned up some page file domain setting code and misc page management code

- improved accuracy of rendered image cache memory footprint calculations

next week

Next week is a 'medium size' job week. I want to get started on a database-level tag siblings cache, which will improve siblings speed and accuracy across the program. I do not think I will have time to finish it, but I know I can move all this forward.

R: 2 / I: 0 / P: 2 [R] [G] [-]

Is there a way to move a selection of files from one database to another while retaining the tags? I could do it one tag at a time, but it'd be great to have an easier way of doing it.

R: 7 / I: 0 / P: 2 [R] [G] [-]

Version 390

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v390/Hydrus.Network.390.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v390/Hydrus.Network.390.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v390/Hydrus.Network.390.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v390/Hydrus.Network.390.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v390.tar.gz

I had a good week. I spent most of my time cleaning code, but there are a couple of important bug fixes as well.

highlights

Several users have reported that potential duplicate pairs could sometimes come up again for files that had previous been set not related/false positive. This particularly hit users who had reset their potential dupes and re-searched. I had trouble reproducing this, but thanks to the latest user report I was and I traced down the problem. It should now be fixed. If you have been hit by this, please reset your potential duplicates (hit the 'cog' icon on the duplicates page), and then re-search–the bad pairs should not be re-added again.

Several tag autocomplete bugs are fixed today, all for unusual search types. Searching 'hyphenated-namespace:' now gives the correct 'anything' search. 'anything' namespace searches now work for namespaces that have asterisk wildcards. Autocompletes in manage tag dialogs ('write' a/cs) now match results with UPPERCASE inputs. Several more cpu-killer searches (usually some variant of 'get everything' through a clever wildcard) are now impossible to add.

Also, I have fixed the recently borked layout of the 'cancel search' button, which pops up when a page search takes more than three seconds, and fixed a couple of instances where a search page could get stuck in the 'Loading…' state until it was poked.

full list

- fixed a bug that was causing potential duplicates to be sometimes re-added between media groups that were previously set as false positive/not related. I apologise for the inconvenience this bug has caused. if you were hit by this, please reset your potential duplicate pairs (hit the cog button on the dupes page) and re-search, and the bad pairs should not be re-added again

- fixed an issue where tag autocomplete entry in the form 'namespace-blah:' was replacing the hyphen or other 'collapsable' character to space, which then was not searching correctly for the _anything_ namespace search

- 'namespace:anything' searches now work when the namespace itself has a wildcard

- fixed 'write' autocompletes not matching inputs with UPPERCASE letters

- fixed adding tags that start with a colon (e.g. ":D") in 'write' autocompletes

- it should now impossible to enter some 'kill my cpu' queries into tag autocomplete, such as '[asterisk]:anything', even if accidentally entered through the fast-add system

- the 'cancel search' stop button that appears after a search takes three seconds is back to being neatly embedded beside the tag autocomplete input box

- hitting the cancel search button now clears the non-interactable 'Loading' thumbnail media page (with its misleading 'Loading…' statusbar) and returns you to an empty thumbnail page

- loading a favourite search with non-immediate search no longer loads the 'loading…' page. it also saves that new non-immediate status to the page session more reliably

- reworked my linux build environment (pyinstaller=3.5, virtualenv=16.1) so that you can launch the built exe using a symlink

- rolling out a first version of a requirements.txt, any feedback would be appreciated

- rolling out another version of the derpibooru file page parser that no longer duplicates namespaced tags as unnamespaced, thank you to the user who submitted this

- .

- boring stuff:

- .

- moving old pubsub system to Qt signals:

- all the 'refresh query' calls that do changes to the current file search across the program

- the current file and tag domain update calls for search pages

- the clear/set file focus calls when launching and exiting the media viewer browser or archive/delete filter

- the way thumbnails send the current focused media to the preview media window

- the way widgets with shortcuts-based tooltips are notified to update those tooltips when shortcuts change

- the way a thumbgrid sends the current tags to be displayed in the 'selection tags' list

- the way a thumbgrid adds newly imported files' tags to the 'selection tags' list as they fade in

- the way the 'searching/waiting' search button is flipped on and off by shortcut. btw what should be the correct name + label for this button? should it really be an icon?

- .

- cleanup:

- NOTE: the 'include' folder is renamed this week to 'hydrus'. if you have source patches, please update. as I further disentagle code in future, hydrus will ultimately move to typical nested folder/module structure

- decoupled the shortcuts edit ui code from the controller and db, unified how shortcuts are edited, and eliminated db wait when booting shortcuts editing

- decoupled the shortcuts manager from the controller, cleaned all the code, and moved to a nicer reference with proper typing hints

- refactored the frame and media controls of clientguicanvas into separate files

- renamed the hoverframes file to canvashoverframes and updated its classnames to 'canvas' rather than the old 'fullscreen'

- fixed two wx->Qt typo artifacts in the login script edit ui

- reduced some occasional idle memory bloat of clients that have large subscriptions

- cleaned up how media-based taglists are appended with new media

- removed some old booru object update code

- some misc setmedia/clearmedia cleanup

- misc search code cleanup

- misc typing hints to clear up pylint confusion

- misc tag autocomplete code cleanup

- misc 'global' variable cleanup

- misc gui code refactoring, cleanup and typing

next week

Next week is a 'small jobs' week. I will catch up with little jobs, perhaps 'search by framerate' or adding select/remove for filetypes. I definitely want to add close/minimise to system tray, which keeps being put off

R: 8 / I: 4 / P: 2 [R] [G] [-]

Version 389

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v389/Hydrus.Network.389.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v389/Hydrus.Network.389.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v389/Hydrus.Network.389.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v389/Hydrus.Network.389.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v389.tar.gz

I had a great week. I fixed many small bugs, added some quality of life, and am rolling in updated downloaders for e621 and Deviant Art.

downloaders

Unfortunately, last week's e621 downloader was not getting md5 hashes reliably. This coupled with the URL format change often meant an increased bandwidth load for the subscriptions that were trying to re-find their place. Thankfully, another user has provided a more accurate one that rolls into today's update. It also pulls rating tags. If you paused your e621 subs, please resume them again.

Deviant Art have also been through a layout change recently. The existing file page parser was getting a little shaky, so I wrote a new one this week that uses their API. The default hydrus DA downloader is now wholly API-driven, so it should all be more resistant to any further changes on their frontend. If you would like to search for nsfw DA content, the default DA login still seems to work, but Hydrus Companion is also an excellent option.

For advanced users: Thanks to user contribution, there are two newer default file lookup scripts, an updated iqdb-based danbooru lookup and one for direct danbooru lookup using md5 hash.

For advanced users: The parser String Transformation system now has a 'url percent encoding' option for the encode/decode types. This lets you convert complicated text to a URL-parameter-friendly percent-encoded string, or vice versa, if you need to do something like a clever API URL transformation.

the rest

The way the new mpv.conf file was stored was failing for users with portable installs. This file is now cached in the db directory alongside your other user-configurable data. All users will reset to using the hydrus default mpv conf this week (now just in that new location), so if you have been using a different one, please re-set it just this once after you update.

Hitting Escape on a non-empty tag autocomplete text box will now clear that box. This sounds like a very small thing, but give it a go–since shift+home and ctrl+a are diverted to the list below when you have results, it feels nice to have a quick shortcut to reset. Hitting Escape on an empty tag input box continues to cause the box to first lose focus (hiding any floating dropdown list), and then, in the manage tags dialog, closes the dialog.

Short-duration media (e.g. a gif that only lasts 0.5s) now loop properly in slideshows with a longer slideshow period.

The new 'favourite searches' system now has a new 'empty page' default, which updating users will also receive. It is just a nice way to clear a page, but feel free to delete it if you do not like or need it.

The collect-by dropdown now has checkboxes again in Fusion style! It should be fixed for any other styles that had problems, but let me know if you get anything odd now.

A bug where pages created by the new 'duplicate page' command would accidentally share some data updates with their original until a client restart has been fixed.

full list

- downloaders:

- the e621 file page parser is updated again, thanks to a user's contribution. this one gets md5 and file url more reliably, and also gets rating tag

- added a 'e621 file page (old format)' url class to help match and search for files downloaded with the old format. please be aware there is no good solution to auto-convert old urls to a new format yet, so this connection does not (yet) solve the old/new comparison test

- updated deviant art file post parser to use their json api. this should be more resilient to their current layout changes

- the nijie.info login script appears no longer to function. as with exhentai last week, it has been removed to make it easier to log in with hydrus companion. please use hydrus companion if you would like to log into nijie.info

- updated file lookup scripts for 'iqdb danbooru' and 'danbooru md5' thanks to a user's contribution

- .

- the rest:

- the way the mpv.conf works changes this week. it is now correctly fully portable, stored in the db directory beside the .db files. if this file does not exist, the 'default' as stored under the install_dir/static/mpv-conf folder will auto-populate it. if you have been using a non-default mpv conf, please re-set it one time after update, and you should be good

- the code that loads mpv.conf is now more graceful on 'missing file' errors, which now means when both the db conf and the default conf are missing

- hitting escape on a tag autocomplete input that has text will now clear that text! note that hitting escape on an _empty_ a/c input will still do 'lose focus' and then 'close dialog'

- updated the slideshow logic so that if a media with duration has a shorter duration than the slideshow duration (e.g. a gif that lasts 0.5s on a 10s slideshow), the media will keep looping until the duration is up. media that has duration longer than the slideshow time will continue to play through once completely, delaying slideshow progression and then stopping promptly when it has finished

- the string transformation system now allows 'url percent encoding' under the encode/decode type!

- fixed the 'only add existing tags' filter in the tag import options, which was denying all the tested tags. it seems to have been hit by a typo in the last three months

- the 'favourite searches' defaults now include an 'empty page' entry, which is a convenient way to simply clear a page. all users will also get this on update, feel free to delete if you don't like/need it

- opening a new search page from a tag or an active search predicate ('open a new search page for…' or middle-click) now copies the file service (e.g. looking at trash) from the original page

- opening a new search page in the 'all known files' file domain when the tag domain should be 'all known tags' (a currently unsupported combination) now coerces the tag domain to 'all local tags'

- checkboxes should now appear again on the collect-by dropdown in Fusion (and hopefully any other) style

- fixed an issue where entering 'namespace:*' explicitly would show the much less efficient wildcard search rather than the efficient 'anything' namespace search

- fixed an issue where wildcard search could include multiple asterisks in a row

- fixed an issue with page duplication where the main management object was not being duplicated properly until a session reload, meaning the two pages would sometimes share signals and changes

- an old wx delayed hide/show performance hack is removed, making the floating autocomplete dropdown now update more smoothly to resize or move requests, such as when the main gui window is dragged

- the program base installation directory is now calculated more accurately, both when running from source and the frozen build, and when launched using a symlink

- install dir and db dir are now specified in the help->about window

- the petition page content checkbox list now has a taller minimum height

- improved error text reporting in hydrus service login failure, hydrus service delay reason-setting, and all 'cancelled' errors across the program

- the review services panel now has elided… text. when unusually long errors propagate up to its status texts, it now won't suddenly jump to 2,000 pixels wide. full text appears in tooltips

- code refactoring: the tag autocomplete input now now takes responsibility for the active predicate list above it

- refactored some tag lists and added typing hints to improve how current page predicates are determined

- did some prep work for tag filters supporting wildcards, but it isn't ready yet

- cleaned up some wx->Qt data fetching code

- misc code cleanup

next week

Next week is a 'cleanup' week. I want to focus on more wx->Qt code cleanup, removing more old UI hacks, improving layouts, and smoothing out how some behind-the-scenes updates are sent across the program.

I also want to have a deeper look into a bug in the duplicates system where the 'not related' relationship is not sticking when some duplicate search information is reset. I previously could not reproduce this, but I have had more reports of it in the past couple of weeks, so I expect to write some more automated test code to root it out.

R: 6 / I: 1 / P: 2 [R] [G] [-]

Version 388

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v388/Hydrus.Network.388.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v388.tar.gz

I had a great week. The client can now save and load searches.

favourite searches

Every tag autocomplete input text box that searches for files–the most obvious being the one on normal search pages–now has a star icon button beside it. Click this, and you get a menu to save your current search, manage your saved searches, or load up one that is saved!

Currently, the saved information is the list of search terms (the tags and system predicates), the current file and tag domains (e.g. my files/all known tags), whether the system is 'searching immediately' or waiting, and optionally the current sort and collect.

Each saved search has a name and optionally a folder, for easy grouping. I have started all clients with an example inbox processing search, just as an example.

If you, like me, keep ten or fifteen empty search pages open (for me it is mostly different creator+inbox searches), I hope this system lets you collapse it down, making for a lighter and simpler session. It should also help for more unusual workflows like duplicate filtering and even file maintenance jobs.

There may be a couple of bugs in the system, something like a collect-by not being saved or updated correctly. Let me know how you get on!

the rest

A note from the users managing Hydrus Companion: The Chrome Web Store release of Hydrus Companion is no longer available due to publishing issues. If you have been using it in the past, please install the extension manually as outlined here instead: https://gitgud.io/prkc/hydrus-companion

e621 changed their site format, breaking hydrus's default downloaders. This is now updated, thanks to a user's contribution. The URL format has changed, so unfortunately your subscriptions will take longer (but will not redownload files) and note they have hit their periodic limits on their next cycle, which you can ignore. If you use an e621 login with hydrus, you may also want to check your account-based tag blacklist on the site, as I have heard this has also changed.

Hydrus's default mpv configuration file now has some normalisation that seems to work well.

If your subscriptions are paused, or if all network traffic is paused, this is now noted in the status bar.

full list

- favourite searches:

- hydrus can now save, load, and edit favourite searches. this first system stores searches with a name and an optional folder name, and contains search predicates, file and tag domain, whether the search is live or not, and optionally sort-by and collect-by

- this is program-wide and all accessed through the new 'star' icon menu button beside any 'read' tag autocomplete input on search pages, duplicate pages, export folder ui, and file maintenance selection

- wrote a favourite searches manager

- wrote a dialog to manage favourite searches

- wrote a dialog to edit a single favourite search

- wrote load and save search functionality

- autocomplete dropdowns that have buttons beside them now stretch their floating dropdown windows across the button width also

- cleaned a variety of search code, simplifying objects and responsibility

- cleaned up some collect-by ui code

- refactored sort and collect controls to better location

- refactored search constants

- numerous small search code fixes and cleanup

- renamed clientguipredicates to clientguisearch

- .

- the rest:

- a note from the users managing Hydrus Companion: The Chrome Web Store release of Hydrus Companion is no longer available due to publishing issues. If you have been using it in the past, please install the extension manually as outlined here instead: https://gitgud.io/prkc/hydrus-companion

- the default e621 downloader is updated to their new system, thanks to a user's submission. if you log in to e621 with hydrus or the hydrus companion and discover some tags are now blacklisted, please check your blacklist settings on your account on the site

- an old test e-hentai login script from 2018 that is no longer in the client defaults will be deleted from clients that still have it today. if the user has no other login script for e-hentai, the domain entry will be deleted as well. this removes potential technical barriers for users that wish to use hydrus companion to access e-hentai, which is now the recommended method

- hydrus mpv now has an appropriate stream title, which propagates up to the os-level sound mixer. it was previously the ugly hydrus filename

- improved error handling when mpv is passed an invalid conf

- the default mpv conf now has audio normalisation that seems to work ok

- fixed an issue with the 'delete/move out missing/corrupt file' file maintenance job where record deletes were not processing correctly. it now deletes the file record correctly and also clears that deletion record, to make re-import of the correct file, if found, easier

- all hydrus menu labels are now "middle…elided" when they are greater than 64 characters

- all new hdd, url, and simple download pages should now obey the 'remove files when trashed' rule. pages in existing sessions will not

- updated the user-created CutieDuck darkmode qss file to the latest version, which alters the recent hydrus qss styling colours like green/red button labels

- did a full pass of all service fetching–all file and tag services should now present in lists and tabs in service_type, alphabetical order, e.g. for manage tag siblings, the tabs will always be local_tags, tag_repositories, both in alphabetical order

- fixed an issue where a 'get darker or lighter comparison colour' calculation was not working well for black or very dark colours

- if subscriptions or general network traffic is paused, the bandwidth section of the main gui statusbar now says it

- the status bar now tooltips each section

- clarified some labels on the edit url class panel

- moved all delayed focus-shifting code to a more stable system

- cleaned up how the global icon cache is initialised and referenced

- updated the hydrus project gitignore to hide all db, log, server, recovery, and media files that could be under the db directory

- updated the endchan links in the help to have a .org secondary link

- more general code refactoring

next week

Next week is a 'small jobs' week, so I will be back to catching up on small things and general mpv/shortcuts/cleanup work. Also Deviant Art are perhaps going through some more layout changes of their own, so I will be looking at that.

As the virus hits, I expect to keep working on hydrus as normal. I am supplied, low social contact, and healthy. I hope you are as well. If we all end up locked down for a month, I hope hydrus can offer a distraction. I will post if I have to stop for a couple of weeks.

R: 27 / I: 2 / P: 2 [R] [G] [-]

Version 387

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v387/Hydrus.Network.387.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v387/Hydrus.Network.387.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v387/Hydrus.Network.387.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v387/Hydrus.Network.387.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v387.tar.gz

I had a great week mostly fixing things and adding and improving small features.

all misc this week

The 'sort files by' dropdown on all pages is now a button. It launches a menu that groups the different sort types, cutting the long list down into easier to navigate groups. Mouse wheel still works on it!

Also, 'sort by framerate' is added. It just does a simple num_frames / duration calculation for now. Fps isn't surfaced in the UI atm, so I expect in the near future to add it to normal file labels and also to add a system search predicate for it.

The options->sort/collect panel finally got its overhaul. Managing namespace sorts is much more sane, support for namespaces with hyphens (like 'creator-id:') is added, and you can edit the order in which they appear in the sort by menu.

Some logic behind tag autocomplete lookup is improved this week. The way special characters like braces and paretheses are handled is better (for instance if you want to search for '[intensifies]'), and now hyphens and underscores are included in these special rules. Typing in 'blue_eyes', 'blue-eyes', 'blue eyes', or 'eyes' will match all of 'blue_eyes'. 'blue-eyes', and 'blue eyes'! Although the results may still be separate without tag siblings, you should never have to worry about searching an underscore version of a tag again. It will take up to five minutes to update your client to reflect the new rules. This is a simple change to a complicated system, so let me know if it fails anywhere, particularly in namespace or wildcard searches!

Right-clicking on a page tab now shows a 'duplicate page' menu item. It simply makes a complete copy of the page (or page of pages) right next door!

Numerical ratings (the ones with multiple 'stars') can now be set by dragging the mouse. You can click on 2/5 and drag up to 4/5 if you change your mind.

The derpibooru downloader gets an update thanks to a user's submission. The 'no filter' search should work again. Also the new tvch.moe imageboard is added to the supported watchers (thankfully, it was compatible with an existing parser, so this was a quick job).

full list

- the sort-files-by dropdown is now a button that launches a nested menu. it still supports mouse wheel events. it should now be quicker to find what you want!

- added 'sort by framerate' to regular file sort. it works for file search at the db level as well, when mixed with system:limit

- under options->sort/collect, the namespace sort-by ui has finally had its makeover. it now has add/edit/delete buttons and up/down buttons for reordering how the entries will appear. it also deals with bad input better. furthermore, namespaces that have hyphens (like 'creator-id') are now supported in namespace sort (and hence collect-by dropdowns!)!

- numerical (multi-star) ratings can now be set by dragging the mouse across the line of stars

- added 'duplicate page' to the page tab right-click menu! it just makes a copy of the page or page of pages right beside it

- system:everything will now always show up in non-query-page autocomplete dropdowns (such as in the file maintenance dialog)

- wrote a maintenance routine to repopulate and correct the tag text search cache. it is possible to trigger this (though it is typically pointless) from the database->maintain menu

- updated the characters that are ignored in autocomplete tag text search rules, which help skip over unusual characters and assist word-break discovery for searching for tags like '[intensifies]'. as well as the previous brackets, braces, paretheses, quotes, and double-quotes, now slash, backslash, hyphens, and underscores(!) are ignored. searching for 'bbb' will now match a tag 'aaa-bbb', and searching for 'blue_eyes', 'blue-eyes', 'blue eyes', or 'eyes' will match all of 'blue_eyes'. 'blue-eyes', and 'blue eyes'!

- to effect the above change, the client will take a few seconds to a minute to update

- the above tag text search rules now collapse contiguous unusual characters, or combinations of whitespace and characters, better

- namespace and simple wildcard search inputs no longer have the tag text search rules applied to them, meaning you can now search for these unusual characters more specifically when desired

- updated the derpibooru gallery search objects to use their api, thanks to a user's submission. this re-enables the 'no filter' mode

- added watcher support for tvch.moe, which works with an existing 4chan-style parser

- the 'add the ptr' help item now warns the user about the ptr's modern drive storage requirements (4GB download+files, 25GB db). the help files are also updated

- I believe I fixed the sometimes crazy fast media drag-move that could happen in archive/delete and duplicate filters

- fixed an old uncaught wx->qt issue with the simple downloader where editing the formulae would throw an error

- fixed a bug in the 'move highlighted thumbnail' code in the rare case where the currently focused thumbnail can not be found

- text input dialogs are now mostly wider

- refactored some ui code, cleaning up core objects and import hierarchy

- did some controller/gui refactoring, pushing on untangling things

- cleaned up a bunch of no-longer-used import statements

- misc ui code cleanup

- slight rewording of database menu

- prepped shortcuts system to ignore a window-activating click (for the media viewer filters), but can't turn it on yet as media viewer clicks are not yet fully plugged in

next week

Next week is a medium-size job week. I would like to get 'favourite searches' working, so you can save a particular page's search and then quickly load it up later wherever you like.

I would like to add some default ratings services to the client as well, since they are easy for new users to miss.

R: 1 / I: 1 / P: 2 [R] [G] [-]

I had no idea this site was good for anything ethical and useful. Sage. Whatever that means

R: 3 / I: 2 / P: 2 [R] [G] [-]

Version 386

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v386/Hydrus.Network.386.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v386/Hydrus.Network.386.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v386/Hydrus.Network.386.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v386/Hydrus.Network.386.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v386.tar.gz

I had a great week. I mostly cleaned code, moving old wx definitions and calculations up to Qt format, fixing bugs and colours along the way. There is not much significantly new this week, but I am happy to have cleared out some behind-the-scenes mess.

gif and mpv.conf

Some gifs have metadata that says 'play this once' or 'play this five times' rather than looping infinitely. Hydrus now parses this information, and if you tell it to under options->media, it will obey it.

MPV is extremely configurable. It uses a file called 'mpv.conf' to load its settings. Hydrus has a very bare conf file at the moment, but I expect to expand it as we experiment more with settings that are useful for hydrus. Now you can set the path of the mpv.conf file you want to use, again under options->media. I have added three test mpv.conf files to the hydrus static directory–one for high quality, and two for different kinds of audio normalisation–if you are interested in this subject, or if you have your own mpv.conf files you would like to experiment with, please give them a go and let me know what you think. Once we know what works well for different situations, I will make some more well-rounded profiles for users to choose from.

misc code cleanup notes

You can now edit how fast the mouse cursor autohides on the media viewer, including turning autohide off completely. Yet again, under options->media. The default is 700ms.

The window positioning and sizing system has had several calculation improvements, and it will also now be better at rescuing off-screen windows (for instance if you have a multi-monitor setup that changes between client boots), all with better error handling. Furthermore, some windows like the splash screen–which finally has fixed colours–will now appear on the monitor your mouse is on. If you have had problems with window positioning or sizing, particularly if you have an unusual desktop or multi-monitor setup, let me know if this improves or worsens your situation.

If you are interested in editing your own hydrus QSS stylesheet, please check the install_dir/static/qss directory again this week. I have moved many of the previously hardcoded colours in the client (like green/red dialog button labels) to a new 'default_hydrus.qss' file. This file is loaded by hydrus, and is prepended to any custom stylesheet. You can add these hydrus-specific qss class definitions to your own qss to overwrite them. I expect to continue migrating hardcoded colours to this file in future.

full list

- gifs and mpv:

- the client now parses gifs for loop count metadata (some gifs say they should only be played x times through, usually 1). options->media now has a checkbox to control whether this value should be obeyed. both the native viewer and the mpv viewer should follow this. default value is still to loop indefinitely

- if gifs are set to play with the mpv player, those without duration will now still be loaded in the native image viewer. the media viewing options ui now notes this

- the mpv.conf file used in the mpv window can now be changed under options->media. it _should_ update the conf for all open mpv players on options dialog ok. added to the hydrus static mpv-conf directory are three new 'test' mpv confs for high quality and two audio normalisation tests. all test feedback and recommended conf info is welcome

- .

- ui cleanup and improvements:

- the media viewer mouse autohide time is now customisable under options->media, including disabling it completely. it defaults to 700ms

- improved the timing and reliability of the media viewer mouse autohide code

- the mouse should now never autohide while a dialog is open

- improved the bad colours of the splash screen. it should now be all one colour now, no ugly stand-out white square or other hardcoded colours. hydev also deployed his unparalleled gimp skills to get a white fade around the transparent-background hydrus icon, so it should look correct in darkmodes as well

- created a default_hydrus.qss file in the qss folder in order to handle formerly hardcoded colours using hydrus-specific classnames and properties. as well as being loaded by default, this qss file is prepended to any custom stylesheet, so any custom stylesheet that includes its own versions of the hydrus-specific entries will override the defaults. this qss will get more work in future

- added on/off buttons to hydrus default qss and converted existing object to use class and properties to obey this

- added a variety of valid/invalid/warning text colours to hydrus default qss and converted existing text objects to use classnames to obey this

- added accept/cancel buttons to hydrus default qss and converted all green/red buttons across the program to use classnames to obey this

- the migrate database dialog now has an outright 'remove location' button to reduce confusion and speed up removal of high weight locations

- if a location does not exist on the migrate database dialog, it will now stop throwing multiple error popups every time the list slightly changes, and will complain if file rebalancing is attempted, and will provide different 'remove' yes/no messages if that missing location currently has files or not

- slight ui touch-ups to the migrate database dialog

- if a window that remembers its position attempts to re-position to a location not on a current display, the windowing system now attempts to rescue it to the primary display, with appropriate popup messages given and errors caught more gracefully

- extended these off-screen rescue calls to windows that pull their position from their parent. e.g. if you open the options dialog while the main gui is half over the left side of your screen, it should rescue to the primary display

- windows that position off the center of their parent now calculate that reliably on the parent window, not just the parent widget, which never really worked as intended

- windows that have no position memory and no parent to pull center/topleft position from will now appear center/topleft of the monitor your mouse is on

- the splash screen now appears centered on the monitor your mouse is on

- cleaned up and improved a bunch of window/screen coodinate code, moving 'space on screen' calculations to 'space on screen minus taskbar' and similar

- unified a 'dialog is open' check across the program

- cleaned up the old wx->Qt size, coordinate, and colour conversion code

- cleaned up some old wx->Qt calculation code

- improved 'light' and 'grey' colour detection code to now work in HSV

- improved colour changing code to now work in HSV

- improved some internal single-shot scheduled job code

next week

Next week is a 'small jobs' week. I want to fix bugs, improve some ui, maybe add some new shortcut actions and other small features.

R: 15 / I: 11 / P: 2 [R] [G] [-]

Version 385

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v385/Hydrus.Network.385.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v385/Hydrus.Network.385.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v385/Hydrus.Network.385.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v385/Hydrus.Network.385.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v385.tar.gz

I had an ok week. There are more improvements to mpv and the shortcuts system, and a few misc small jobs done as well.

mpv

Some more general improvements this week:

MPV windows now work in slideshows, and all audios/videos are smoother to move to the next file when slideshows are running.

When you scrub the seekbar below a native or mpv video player, the video should now more reliably pause itself while you drag. Also, more kinds of one-frame music videos (where it is just an image webm with a song in the background) should now appear with their seekbars now, and the seekbars for all these unusual videos and audio files should work better.

A zoom bug where video could navigate down a few pixels on zoom in/out cycles is fixed, and the blank audio mpv window now has some hardcoded zoom capability and should scale itself to fit in the small preview window, always letting you interact with its seek bar.

I attempted to get mpv working for macOS but ran into some technical walls, I think because I am limited to 10.12 Sierra. I do not think there are excellent solutions for this, at least for my macOS expertise, so mpv support on macOS is unfortunately on hold until I learn more.

shortcuts

The media viewer gets more shortcut work this week. Until now, hitting enter, return, or escape on any viewer, and middle or double left-click on the normal browser, was hardcoded to attempt to close that viewer. This is now all plugged into the shortcuts system and is editable. It comes under the 'close_media_viewer' command for all the 'media_viewer' sets. Mouse integration is not complete, but we are almost there, and most actions should work, if you want to map, say, alt+left click to something.

Last week's double click support added an interesting bug to existing workflows: whenever you intended to click twice fast–like when using the archive/delete filter quickly–the second click was detected as a double-click instead, and often nothing would happen. This is now fixed. Now, if there is no double-click command mapped, including on any parent window, that second click will be attempted instead as a second single click. So, you can now click through the archive/delete filter fast again.

This new single/double click logic (when any parent window does have a double-click command mapped, the single click substitution does not occur and that double-click does count and is passed up) adds a neat new capability to mpv windows, which, because they scale up so well and usually almost fill the screen, leave little whitespace to click 'outside' the window to do anything but pause/unpause on a left-click. Now, a double-click can be neatly passed up to a parent if there is a mapping. For instance: a double-click on an mpv player in the regular browser media viewer will now pause and then close the window (rather than doing a quick pause/unpause), because 'media_viewer_browser' on the larger window has double-click mapped to close the window. I have also made it default that double left-click in the archive/delete filter is now mapped to 'keep', so in a similar way, double clicking an mpv window in the archive/delete filter will now do 'keep and move on'! If this is a bit complicated to read, don't worry–have a play with it and you'll get the feel. I hope this makes filtering videos a bit nicer now we have mpv.

the rest

Danbooru apparently changed how their tags work behind the scenes, so this week adds new parsers for danbooru that should have everyone working again. Thank you to a user for submitting these.

I fixed some searches, usually ones without inclusive tags, where files in the trash could appear even though the domain was 'my files'.

A popup message now appears while a session is loading, just as a little indicator for slow-loading sessions until I can get better loading progress ui integrated here.

The splash screen looks less ugly, but it is still bad. I will keep working on this to get the greys/whites/blacks and layout margins looking better for all stylesheets.

full list

- mpv:

- the mpv window is now plugged into the slideshow system, so when an mpv window has played its media once, a media viewer browser currently slideshow-ing will now correctly know to move to the next media

- slideshows of videos should progress to the next file faster and more smoothly, with no more half-second blit of the start of the movie before the next file loads

- pausing video/audio is no longer cause for the slideshow to move on–now only the 'has played once through' check will naturally trigger it. you can now more reliably seek/scrub a video during slideshow

- the mpv window and native animation player is now better about pausing while a seek drag (scrub) is ongoing

- various music videos that have 1 frame should now show a seekbar correctly for the mpv window, with correct timecode based seeking

- fixed a bug in the video/audio resizing code that meant zoom in/out cycles would move a video player down a few pixels off center

- the blank audio mpv window now has basic hardcoded zoom support and will scale down to a too-small viewer window, so you can still access the seek bar

- fixed some 'has duration' calculations for audio/video that has duration but no frame count

- .

- shortcuts:

- if a shortcut handler on an individual window does not have a double-click mapping for a command–and furthermore if none of its parent windows that are fully plugged in have one either–it will attempt to map the single-click version of the event as a backup. so now if you have an archive/delete filter, you can click fast again and the double-clicks will be interpreted as single-clicks (unless you map double-click to mean something else on the media_window or one of the media_viewer parents)

- the media viewers across the program are now fully plugged into the shortcuts system for key presses, and half-plugged in for mouse clicks

- 'close_media_viewer' is added to all 'media_viewer' shortcut set types. enter/return/escape are now defaults for 'media_viewer' (applying to all), and middle-click/double left-click are now defaults for 'media_viewer_browser'. this is no longer hardcoded. if you are a madlad, you can now unmap all 'close_media_viewer' commands

- double left-click is now assigned to 'keep' in the 'archive_delete_filter' shortcut set. due to the new click/double click rules above, this means that by default double clicking a video/audio in the archive/delete filter will now mean 'keep and move on' on that fast second click!

- edit shortcut set ui now sorts its command list on load

- significant shortcuts refactoring

- general shortcuts code and debug code cleanup and improvement

- .

- local file import pages and most downloaders will now report more file import steps to their status labels. most will blit by too fast to see, but if it hangs for a bit, you will now see the step it is caught up on. I imagine in most cases, this will be metadata generation for large videos

- fixed a variety of searches that could return files not filtered to the current file domain (e.g. files in trash while in my files, or not in ipfs while in ipfs) when the search did not include an inclusive tag

- updated the default danbooru file page parsers to get their new tag format, thank you to a user for submitting these

- a popup message now appears while sessions are loading. it auto-dismisses once the load is complete

- the edit media view options dialog (under options->media, launched from the filetype list) is now better at disabling non-applicable widgets based on filetype

- fixed an issue where clicking from the autocomplete dropdown floating window to the same control's text input could result in a single flicker-frame where the dropdown is hidden

- tightened the size of the splash screen white background. figuring out appropriate colour from the current stylesheet remains elusive lmao

- cleaned up and wx->Qt converted a variety of event handling code

- updated some 'mime' labels across the program to the new 'filetype' wording

next week

Next week is a cleanup week. I will catch up on some general background code improvement. Some users are getting crashes with the new mpv window swapping system, so I will also look at that.

R: 27 / I: 5 / P: 2 [R] [G] [-]

Version 384

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v384/Hydrus.Network.384.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v384/Hydrus.Network.384.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v384/Hydrus.Network.384.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v384/Hydrus.Network.384.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v384.tar.gz

I had a great week updating the shortcuts system.

shortcuts

The 'new' shortcuts system has been in limbo for some time. I like it, but I never really 'finished' it, and there were still many places across the program that had hardcoded shortcuts. This week moves it forward, mostly for mouse clicks and the new mpv window. As a reminder, you can customise the system under file->shortcuts. There are multiple shortcut 'sets' that apply in different parts of the UI.

First of all, the shortcut edit UI has been rearranged so it is less of a mess. Instead of having all possible commands on the same window, you now select which one you want from a dropdown, and unnecessary widgets will be hidden. Also, the main dialog now lets you restore a shortcut set to default values.

Although mouse support remains limited, the windows that take mouse clicks can now catch more. Double-clicks are now recognised, and if you prefer the feel of a click up instead of down on certain actions, you can now differentiate between click 'press' and 'release'.

Next, the actual 'media windows' across the program, the image, animation, or mpv windows, are now plugged in to the shortcuts system. For now, this mostly means clicks. Left-clicking to pause/play is no longer hardcoded, and I have removed the default 'double-click -> open in an external program' (although you can add it back in as ctrl+double left-click or whatever you like) as this command is less important with mpv's audio and fast high-res scaling. There are two new shortcut sets–'media_viewer_media_window' and 'preview_media_window'–that have actions for pause, pause/play, open_externally, and close/launch_media_viewer. By default, middle-click or double left-click on the preview media window now launches the media viewer, just like on a thumbnail.

And the 'media_viewer_browser' set now has 'show_menu' as an action. It defaults to 'release right-click', but if you would prefer your menus on a click-press, or ctrl+release right-click or something, you can now change it. I expect to do this for the other important menus across the program as well. Also, most menus can now be opened with the context menu key on your keyboard, if you have it, so you might want to even remap right-click to something else.

The new 'global' shortcut set now has actions to exit and restart the application. They should be caught by the main gui window or any media viewer and work just like the actions on the file menu.

Most of the work this week was cleaning up behind the scenes code. I feel great about now extending it to more windows (such as thumbnails, to customise selection click shortcuts) and commands, and eventually planning advanced features like multi-actions so you can build pseudo-filters where a left-click is 'apply this tag, archive, and move to next file'. For now, I hope to slip in little bits of work here and there.

EDIT: A user let me know that the new double-click detection makes the archive/delete filter not work if you click too fast (it now detects the second click as different than a single-click and hence doesn’t map the ‘keep and move on’ etc.. action). I will fix this for next week, but in the meantime, if you are a fast clicker, please add additional shortcut mappings to ‘archive_delete_filter’ for double-click, and you should be working again.

the rest

I cleaned up some of the borked media-switching code I introduced to get mpv working. Transitioning from one media to another should have fewer instances of single-frame stretch. I am going to keep working here, because in some situations there is a single-frame of blank, which can be a bit of a flash.

The tag autocomplete dropdown window should be better at choosing when to hide and show.

I fixed a bug related to the advanced 'clear deleted tag record' command, which is usually fired by the tag migration window. The action was not properly clearing the cache that thumbnails use to fetch their tags, so while the deleted tag was deleted from the master database table, it did not appear so in UI. If you were hit by this, please run database->regenerate->autocomplete caches one time to resync yourself.

If you are a user who uses the parsing system, the string transformation panel has had a significant rework in the past two weeks. It hides/shows unneeded controls just like this week's shortcuts panel, and there is now a live preview of the current step's example and example-transformed text in the individual transformation rule edit dialog, which updates as you type, including showing regex errors! I expect in the near future to add the string transformation system to the filename tag panel soon, swapping out the jank old regex systems there for the nice full-featured converter.

full list

- shortcuts:

- the shortcut system now supports mouse double-clicks–left, right, or middle

- the shortcuts system now differentiates between press or release single mouse clicks–although complete support for release mouse events may be a bit patchy, as full mouse integration is ongoing

- the shortcut edit ui is now simpler–the command type is selected by a list, and the individual command sub-panels hide and show as appropriate–no more stupid 'set command' buttons

- the shortcut edit ui now has a 'restore defaults' button that will restore an individual set back to default settings

- two new shortcut sets are added–'media_viewer_media_window' and 'preview_media_window'. they control pause, pause/play, open_externally, and close/launch_media_viewer respectively. they work on the static image viewer, the native animation widget, and the new mpv player, and they support mouse clicks. the old pause/play (formerly left-click) and open_externally (double left-click) commands are no longer hardcoded

- by default, the preview window's media window now launches the media viewer on a middle- or double-left-click

- 'media_viewer_browser' shortcut set now has 'release right-click' bound to 'show_menu', a new command, which is no longer hardcoded

- most menus across the program can now be opened with the keyboard context menu key

- the 'global' shortcut set now has 'exit_application', 'exit_application_force_maintenance', and 'restart_application' commands

- fixed the rating increment/decrement command option not hiding in non-'media' shortcut sets

- fixed some issues loading edit ui for shortcuts with rating actions

- significant refactoring and some cleaning of shortcut code

- .

- the rest:

- mpv windows should not longer get a single frame of previous-window-stretch when flicking between one mpv media to another with a different aspect ratio on the same media canvas. when a video is caught in a frame of loading, it should now flicker a frame of black

- switching from one static image or native animation to another of the same type _should_ be less likely to do a single frame of stretch when transitioning. when an image or animation transition is caught on a new frame, it _should_ now flicker a frame the same colour as the media canvas background

- the string transformation edit panel's individual transformation rule edit panel has had some more work: much like with shortcuts, the controls now hide and show based on transformation type, the controls' text labels now change based on transformation type, and the example text now updates on any widget change. the manual 'update example' button is removed

- fixed a typo that caused an error when establishing the correct mouse cursor to use over the volume control when hydrus was using PyQt5 (rather than PySide2)

- in order to reduce accidental micro-drags that cause mpv load-pause issues, starting a thumbnail drag now takes more pixels and requires a smoother drag to start, let's see how it goes

- improved the show/hide logic of the floating autocomplete dropdown window. it should now also reliably detect when window focus goes from the dropdown itself to another window

- fixed a bug where clearing the deletion record of a deleted tag would not remove the record from the fast cache that populates thumbnail tags (making it seem on most file loads that the tag still existed). if you were hit by this previously, please hit _database->regen->a/c cache_ one time to resync the cache

- relatedly, thumbnails should now correctly live-update their deleted tags on deletion record clearance updates

- if mpv is not available, opening the about window will now make a popup with the actual import error trace

- significant refactoring of various ui code

next week

Next week is a 'small jobs' week. I have plenty to be getting on with, so I will just try to mix it up with a variety of fixing and improving work. I'd like to push on some more mpv issues like slideshow and custom .conf support in particular.

R: 25 / I: 5 / P: 2 [R] [G] [-]

Version 383

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v383/Hydrus.Network.383.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v383/Hydrus.Network.383.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v383/Hydrus.Network.383.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v383/Hydrus.Network.383.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v383.tar.gz

I had a good week. Unfortunately it was a little changelog-light again, but hydrus now has a nice volume control, and mpv should be working for the Linux build.

mpv volume

You can now set volume and mute properly across the program. Any mpv player that has audio will now have a small speaker icon button next to the seekbar. This controls 'global mute', which silences the whole client without exceptions. This control is also on the top hover window of the media viewer.

When you put your mouse over the speaker icon, a popup appears with a second speaker button and a volume slider. The second button controls mute just for the preview window or the media viewer. By default, the volume sliders are also respective to preview vs media viewer, so you can have a quiet preview vs a loud media viewer if you like. (if you wish, you can change this under options->audio to have them both use the same single global volume setting)

You can also change the mutes and volume from the media canvas right-click menus.

Furthermore, there is a new shortcut set, 'global', which now has actions to set, unset, or flip the global mute. At the moment, it works on the main gui window and any media viewer. By default, I have set ctrl+g to flip global mute on and off. Please feel free to set whatever works for you under file->shortcuts->global.

I expect to add more options and shortcuts for altering volume.

linux

I have updated my Linux dev machine from Ubuntu 16.04 to 18.04, and with it, I have added mpv support for the Linux build.

This has caused some changes to my environment and the build. My build packager has included more library files than it needed, so the release has bloated. I have taken out many .so files that I am fairly certain are not needed, but it is still about 200MB. I will test this more and hope to cut down more in future, as well as improving my PyInstaller configuration to slim this down more naturally.

It is possible that some (perhaps older) flavours of Linux that were happy with the 16.04 build will not be happy with the newer 18.04 libraries. If you use the Linux build, I would be interested in knowing:

1) Does 383 boot for you?

2) Does MPV load correctly and play audio?

Early tests seem to be good, but if you do use an older version, please be prepared to roll back.

full list

- mpv:

- updated the prototype volume/mute controls on the top media viewer hover window to be a proper 'speaker' icon button for mute with a volume slider that pops up or down on mouse-over

- the new volume control is on the hover window and any media that has audio

- the right-click menu of the preview viewer and media viewer now have volume submenus to set mute/volume

- the client now has multiple volumes and mutes:

- for mute, there is a global mute which overrides everything, and the preview and media viewers have their own mutes that just apply there.

- under options->audio, you can choose whether preview windows have their own separate volume value, default is yes they do

- there is a new shortcut set called 'global', which applies on the main gui and the media viewer both, and which currently has actions to alter global mute. by default, ctrl+g flips global mute

- after reports of unusual rendering bugs for some users, the default mpv.conf is now more barebones. more work will happen here

- .

- linux:

- the linux release is now built on Ubuntu 18.04 (was 16.04). unfortunately, my build packager bundled in a variety of surplus libraries, so the archive has bloated somewhat–I have removed some that I am confident are not needed, but I may have made a mistake, and there are likely more that can be taken away

- the linux release now comes with mpv support

- please let me know if you have any errors running this build or loading mpv. early tests seem good though!

- .

- the rest:

- the launch/exit splash screen now uses a cleaner Qt-compatible layout system. It resizes and obeys stylesheets better, colouring text and background according to current style

- removed the 'has duration' text label option from 'audio and duration' options panel as it is no longer used, and renamed the panel back to just 'audio'

- the string transformation edit panel's individual transformation edit panel now shows that transformation step's example string and the transformed string, which is updated by button. this edit panel will get some more love soon, including dynamic hide/show of applicable controls and live updates of the example transformation as you type

- misc ui layout improvements

- misc ui improvements

next week

I would like to put some time into the shortcut system. The shortcut edit UI could do with some slimming down, and I want to integrate more mouse actions, particularly now we have MPV. I would love to have the currently hardcoded clicks like video pause/play and 'open externally' completely configurable so users can set up the archive/delete etc… workflows that work best for them.

R: 7 / I: 0 / P: 2 [R] [G] [-]

Version 382

EDIT: Fixed Links

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v382/Hydrus.Network.382.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v382/Hydrus.Network.382.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v382/Hydrus.Network.382.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v382/Hydrus.Network.382.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v382.tar.gz

I had a good but light week. MPV should be working better, and some of the recent UI animation flicker/jitter should be gone.

mpv

Introducing MPV added a weird bug–after the first time an mpv window was opened, the client would then in some cases fail to animate thumbnail fade or process certain intra-program events unless another event, such as a mouse move over the program, restarted the event loop. Some media viewer transitions have also had some single-frame flicker. I have banged my head at this problem for a couple of weeks now, thankfully improving some behind-the-scenes code along the way, but I could not find the full solution. As it happens, rolling Qt back to a slightly older version (5.14.0 to 5.13.0, for users running from source) seems to have completely fixed it. It could simply be a 'native window' bug in the new version of Qt, but at the moment I still suspect my mostly borked wx-code-trying-to-talk-to-Qt-code. Until I can clean that code up, I am happy to have this easier solution.

I fixed a recent issue with the preview viewer not clearing out media correctly on page changes (which was particularly significant now we have audio) and also fixed an mpv crash for Linux/macOS users who happened to already have MPV available through their OS. Also, MPV is now configured to have a 'lighter touch' on filesystems, so if you have a client with a laggy file storage system (e.g. a very large client stored over a network share) and have experienced extreme latency when trying to open MPV, this situation should be much better. MPV should be fairly stable now, so my big remaining tasks are to get nicer volume controls and proper build support for Linux and macOS.

MPV as a program relies on '.conf' files for most of its configuration. Hydrus now follows this format. If you have your own conf or otherwise would like to edit this, please check out the mpv.conf under install_dir/static/mpv-conf. If you have settings you think should be in the default for all users, let me know. I'll likely let you optionally override where hydrus looks for its conf (where it can't be accidentally overwritten on update) in future.

the rest

There are now a couple of user-created darkmode stylesheets in a default install, so the stylesheet section of options->style now has some entries for everyone. Feel free to check them out–there are still some colours in the client that are hardcoded like green/red button text and anything under options->colours, but I plan to move everything over to Qt in time.

When you zoom out from a zoom that makes the media larger than the media viewer canvas to a zoom that fits inside it, the media will now re-center. See if you like this–I may make it optional.

An issue with the 'the client is already running' dialog not appearing is fixed.

full list

- mpv:

- rolled Qt back from 5.14.0 to 5.13.0 on the releases, which seems to have fixed our 'event queue sometimes halts until mouse move' issue that occurs after initial mpv load. some other ui and media viewer resize jank seems to be cleared up. I dunno what happened with 5.14, and I don't suspect it as the problem nearly as much as my currently borked Qt event processing code, but rolling back seems the easiest solution for now

- fixed an issue that was crashing non-windows that were able to load mpv

- mpv now loads an mpv.conf from install_dir/static/mpv-conf. please feel free to swap in another conf or edit that one as you like. I would be interested in feedback

- default mpv conf is now set to specifically enable some hardware acceleration to improve playback for some users, and to never load sidecar files like subtitles as this was introducing incredibly large load lag for users with large/high latency file storage

- fixed a new issue where preview windows were not unloading media (particularly significant for mpv with audio) on page change and client shutdown

- fixed an issue with global volume propagation to multiple open mpv widgets

- .

- the rest:

- added two dark qss stylesheets from the user-creation github repo to the default install

- when zooming out from a zoom that makes the media bigger than the media viewer canvas to a zoom where it fits, the media will now recenter. see if you like this, maybe it should be an option

- to help forestall unnamespaced filename tag spam in various new-user scenarios, the 'filename' checkbox-and-namespace widget on the filename tagging options panel now initialises with 'filename' as the namespace

- fixed a recent window sizing issue with the 'the client is already running' dialog not appearing

- file export paths that include subdirectories that could possibly have empty text, like "[creator]/[page]", will no longer error when this is so (e.g. if a file in this case has no creator tags)–they will eliminate the subdirectory entirely, becoming "[page]". this should work for all platforms and for any nested subdirectory

- fixed an issue with some fractional dataspeeds below 1KB/s displaying with many significant figures

- improved some custom event handling definition code

- reworked hydrus's internal object publisher/subscriber messaging system to be more Qt-happy

- if the file import tagger is given a neighbouring .txt file to pull tags from that does not decode to utf-8 nicely, it now catches and reports the error more gracefully

- reworded a bit of the installing help and first-start popup to emphasise that hydrus does not auto-update

- added links to https://github.com/Zweibach/text/blob/master/Hydrus/PTR.md , a new guide for the PTR, to the help

- removed the old 'hardcoded shortcuts' help entry, since it is increasingly irrelevent

next week

I took a bit of time off this week and rejiggered my future schedule to reduce some work stress. I feel great. Next week I would like to make some nice mpv volume controls and clear out some miscellaneous small jobs.

R: 7 / I: 1 / P: 2 [R] [G] [-]

Version 381

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v381/Hydrus.Network.381.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v381/Hydrus.Network.381.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v381/Hydrus.Network.381.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v381/Hydrus.Network.381.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v381.tar.gz

I had a good week with a couple of challenges. MPV is now ready for all windows users and is turned on by default.

MPV

Thank you to the advanced users who tested and gave feedback on MPV. I have eliminated the crashes, tightened up the jank, and am now rolling it out to all Windows users by default for video, audio, and gif/apng. All media view settings under options->media will be reset this week.

MPV is a good free media viewer. One of the core benefits of moving hydrus to Qt was being able to plug it into our media player, enabling hardware-accelerated video playback and audio. It looks just like the native player, with the seek bar beneath, but it works much faster, able to play 1080p or 4k videos at 60fps at full or unusual zooms. And of course, it makes noise!

This is early days. I have only just started plugging into MPV, so many features are basic or not yet available. The global volume and mute controls are currently some very ugly controls in the top media hover panel. Slideshows will not move forward on an MPV window (hydrus doesn't know when an MPV player has 'played once through' yet), and some processes like the archive/delete filter will need some extra workflow options now that more users will be playing videos at high res (left-click on the player pauses the video, so to set 'keep' on archive/delete, you'll have to click on some whitespace, of which there is so much less when the video is so big). Please let me know what your top priorities for improvement are, and of course, if you encounter errors or crashes, let me know. I'll keep working.

Even though it was a lot of difficult work, I am overall really pleased with how this has gone. The only big remaining bug that I need to nail down is an unusual thing where after multiple mpv viewings in a preview window, that page will stutter some query/thumbnail loading unless the mouse is moving. This is a slightly frustrating bug, but the benefits of MPV are enough that I am happy to live with it. I will also get it going for Linux and macOS, which I did not have time for this week.

the rest

Assuming that users will want to set/unset MPV and other view/zoom options for filetypes in the coming weeks, I have reworked how all of that works under options->media. By default, you now set view and zoom options for 'all video files' and 'all images', and then if you have specific options for just webms or pngs, you can set those specific options to override the group default. Every user will be reset to the new defaults on update. Please have a play with this this week.

Similarly, I have reworked the UI for system:filetype. The growing list of individual filetypes are now hidden from view when not needed, and the 'group' types have tristate checkboxes for better review. It takes up less space and just feels better.

Also, I have pulled gifs and apngs out of the 'image' group and created a specific 'animation' group for them. This doesn't change much, but it makes it easier to search for or manage settings for static images vs little animations.

I fixed a bunch of the weird layouts that were accidentally introduced last week. Please report any more you find–I am still fighting to convert old wx layout code to Qt's system, so this could happen again in a place I do not notice.

full list

- mpv:

- mpv is now available and the default for all windows users

- I believed I have eliminated the final reported mpv crash

- mpv load and unload delays are greatly reduced. initial load still takes about half a second, but subsequent loads are now as quick as native renderers

- mpv seems to work well for gif and apng

- added a very simple global volume slider and audio mute checkbox to the media viewer top hover window. this was a quick patch–much better controls and shortcuts will come in future

- mpv windows now properly re-show the cursor on mouse movement

- unified mpv mouse press/release handling with native animation–click down now does pause/play and starts a drag event

- unfortunately, in some cases embedding mpv requires overriding local OS number rendering (e.g. 1,234 vs 1.234). hydrus number rendering is now coerced to the english style with commas until we can figure out a better solution–sorry!

- cleared up an issue where simple clicks on page tabs would trigger micro-page drags that were immediately cancelled. this situation was exacerbated when the page being left had an active mpv window. the flicker of page drag cursor is now gone, and some weird situations where static clicks during busy time could move a tab should be fixed

- eliminated the recent issue in the media viewer where transitioning from one media type to another through navigation, particularly mpv->other, would flicker a single frame of the last 'other' media shown(!)

- fixed a bug where repeated mpv views in the preview viewer could disable client file drag and drop

- the bug where thumbnails may not waterfall in unless the mouse is moving after some mpv videos are loaded for a page is relieved but not completely fixed

- if the preview window is collapsed and hidden, media will no longer ever load into it

- fixed an edge-case bug where the mpv window would not like being told to show nothing when it was already showing nothing

- wrapped mpv load errors in a basic graceful catch

- fixed an issue some users had with loading mpv's dll

- .

- file types:

- a new file metatype, 'animation', is added, for gif and apng. these are no longer considered 'image' for a variety of purposes

- the filetype selection panel, which is used in system:filetype and import folder UI, has had an overhaul–it now has tristate 'mime group' checkboxes to represent a half-filled group and expand/collapse buttons to hide the tall filetype lists. individual filetype lists will start hidden unless their default value is a partially filled group

- the media view options have a similar overhaul: they are now collapsed to general filetypes by default. you set view and zoom options for the generalised 'video' type under options->media, and if you want to set specific options for webm or anything else, you can add/delete those types to override the general default

- the new default options for a fresh client are just for these general types. if mpv is available, video, animations, and audio now start with mpv as the default viewer. video and animation zoom is now flexible (not fixed to 50%, 100%, 200%) and will fill the media canvas

- all media view options will be reset to this simple default on update! if you have specific zoom or display preferences, please reset them after the update–but you might like to play with mpv a bit first, as it renders at large and smooth zooms very well

- .

- the rest:

- the new thumbnail right-click file selection routine will now only focus and scroll to the first member of the selection if no other members of the new selection are already in view

- fixed some caching code and sped up the new select/remove menu count generation (which can lag for very large pages) by two to six times

- sped up file filter counting code by about ten percent

- fixed weird layout on: migrate database panel, duplicates page (left and right), edit shortcuts, edit import folder, and the filename tagging panel

- fixed an issue where the media viewer's hover windows might flicker into view for one frame when the mouse moved over the center of the media viewer for the first time

- fixed a media viewer shutdown issue that would sometimes lead to the first file in the list being opened in the shutting-down viewer for an instant or highlighted as the new thumb focus

- the file maintenance system that queues up missing/broken files' urls for redownload will no longer re-select the download page on every new url

- fixed an issue where a downloader's tag blacklist was not being applied on the child files of certain kinds of multiple-file post (such as with pixiv)

- deleting a very long tag should no longer create a very wide confirmation dialog in the manage tags dialog

- fixed some 'the panel grew a bit, but the parent window didn't grow quite enough and now it has scrollbars for two pixels of extra content' sizing issues

- fixed some dialog sizing calculations when the parent window was borderless fullscreen

- maybe fixed a rare event processing bug

- improved quality of some misc data comparison code across the program

- did some significant backend event/pubsub code cleanup, mostly related to getting mpv working a bit cleaner

- improved thumbnail rendering time

- improved smoothness of thumbnail fade animations (at least for when they are working right, ha ha!)

- misc fixes

next week

Unfortunately, I believe that I burned out over the past four to eight weeks. I have been pushing too hard, trying and failing to keep up with my promises, and along with some IRL stuff it nuked my schedule and energy and mood. It hit a breaking point this week, and I realised I was working non-sustainably. I will fix this situation in the coming weeks by altering my schedule. I expect to scale back on overall work hours and hydrus changelog work specifically, focusing instead on keeping myself healthy first so I can face other work (like keeping up with messages and maintaining a productive workspace) and not go nuts. I will also try to promise less when it comes to timeframes so I do not feel bound to stay up late working. I apologise if you have been waiting on me for something–I lost where I was.

I would like to do some more mpv work next week, and do some code cleaning. I will also be taking a bit of time off, so it will be a light week. Thanks everyone!

R: 10 / I: 1 / P: 3 [R] [G] [-]

Version 380

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v380/Hydrus.Network.380.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v380/Hydrus.Network.380.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v380/Hydrus.Network.380.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v380/Hydrus.Network.380.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v380.tar.gz

I had a couple of difficult weeks, with illness and other IRL problems getting in the way, but I got some hopefully pretty neat work done. A new fast video and audio player is available for advanced users to test, and there are a bunch of fixes and ui improvements as well.

A user just notified me that the Duplicates page has crazy layout! I apologise. Nothing is broken, it is just sizing wrong, and part of a longer fight I am having to convert my old wx layout code to Qt. I know exactly what happened here, and I will have it fixed for 381. If you discover more UI like this (the system predicate panels have a bit of it), please let me know.

mpv

This is just for advanced users this week. It is a basic prototype that is not ready for real use. I will improve a bit before turning it on for everyone, hopefully next week.

MPV is a good free video player. One of the core benefits of moving hydrus to Qt was the potential of being able to neatly embed it into the media viewer. I am happy to report that these past two weeks have been successful, and it looks like hydrus is finally getting hardware-accelerated video playback and full native audio support!

If you are using the windows build in advanced mode, please check out the big ugly filetypes list under options->media. Under all the video and audio filetypes, you can now set 'show using mpv' as a show action. Please forgive how large this list has grown–before I turn mpv on for everyone, I am going to rework this list so you have the option to just say 'for all video: xxxxxx', rather than always having every single filetype.

Once you have it set, it should load video almost exactly the same as my native renderer, with the scanbar below for navigation. Audio files will show as a (for-now fixed-size) black box or album art. Be warned: there are no volume/mute controls yet! I have set it at 70% volume for now. MPV has an optional and configurable on-screen display, but for technical reasons I had to stick with my custom controls.

I know what features I need to add, so the feedback I am most interested in this week is in where mpv breaks. If it doesn't work for you, or if it fails to load a file, or it has bad performance, or it sizes wrong, or you get a crash after closing the media viewer, anything like that, please let me know. Performance in my tests has been excellent–rendering 4k 60fps no problem as long as you have any sort of branded GPU. I have had some audio jitter on some unusually encoded videos and some crashes (which I think I have fixed).

One annoying thing is mpv requires a hefty 57MB dll, making the already bloated hydrus download even bigger. One user suggested perhaps putting out two releases–a 'core', and an 'update', which wouldn't have the big stuff. I am still thinking about it–while I like the overall idea, the hydrus build is held together with duct tape, and having a small non-functional release will increase complexity and inevitably cause ugly syncing difficulties whenever the big stuff does need to be updated.

Users running from source will need libmpv and python-mpv to enable this. I haven't pinned down the most convenient and functional way to do this for Linux and macOS yet, so we'll have to figure this out over the coming weeks and get it into the help. Any knowledgeable feedback here would be great.

For the near future of mpv, I would like to: figure out the technical side of including mpv in the Linux and macOS builds; write volume/mute UI and shortcuts; add global volume/mute management; test mpv with gifs and apngs; explore custom shaders and configuration files; and generally polish the whole thing.

quality of life

The tag right-click menu has had a significant overhaul. More commonly used actions are moved up, less common down. Tags with siblings now provide all siblings under the 'copy' menu! System search predicates have better copy and search-action support (like 'exclude' system:archive, which will actually add 'system:inbox').

Right-clicking on the thumbnail grid's 'whitespace' no longer does a 'deselect all' action, so you can now easily right-click in an empty area and go remove->selected!

Across the program, all filetypes are now referred to with human language. Instead of 'image/jpg', it is now just 'jpeg'. Instead of 'application/x-7z-compressed', it is now just '7z'. Please let me know where it looks bad, and if you would like to edit these labels.

The 'new page chooser' dialog now dismisses itself if you click off it! So, if you accidentally middle-click in some page tab whitespace and it annoyingly pops up, just left-click, and it will go away. Also, this dialog now catches enter/return key, and will hit the 'first' button (if you imagine the nine possible button locations as a number pad, starting at top-left). Hitting enter twice will typically open a new 'my files' search page.

A user surprised me recently by asking where to find the 'pause video' shortcut–I hadn't realised there wasn't one! Now this is fixed: 'pause_media' and 'pause_play_media' are added to the 'media_viewer' shortcut set, and 'pause_play_slideshow' is added to 'media_viewer_browser' set. Previously, the slideshow pause/play was hardcoded to space bar–this is now gone, so if you used this, you'll want to set it up yourself.

the rest

I have added simple support for RealVideo, RealAudio, and TrueAudio files. These are some formats, so if you come across some variable bitrate rmvb file or something that doesn't work, please send it in and I will have a look. MPV couldn't handle a couple of my test .rm files' audio tracks, but the .ttas seemed great.

Ever since the CloudFlare-8chan break, I have been telling myself to figure out some new downloader objects for the bunkers everyone scattered to. This job kept being put off, long enough that 8kun came back(!), but I managed to fit in some time in this morning and got 8kun and vch.moe support added for today's release. 8kun was obviously easy to convert from the old 8chan parser, and vch's engine provides an API very similar to 4chan that I was able to duplicate and modify for. I had a proper look at julay, smuglo.li, and endchan, but these use their own API formats (or no API at all), so I will have to write new parsers, which will take longer. I really would like to get them done for 381–please remind me if it looks like this job has slipped away again.

full list

- basic mpv support is added. it comes with the windows build this week, and is a prototype meant for initial testing. the library is optional. users who run from source will want 'python-mpv' added via pip and libmpv available on their PATH, more details in running_from_source help

- took an qt-mpv example kindly provided by a user, updated it to work with the hydrus environment, and integrated it into the client as a new choosable view type under audio/video filetypes under options->media for advanced users

- reworked how the 'start paused' and 'start with embed button' media viewer options work under options->media. these are now separate checkboxes, not combined with the underlying 'show action'. existing embed/paused show actions should be converted automatically to the correct new values

- unfortunately, due to some python/qt/libmpv wrapper mouse interaction issues, mpv's 'on screen controller' overlay is not available

- for now, left click pause/plays the mpv window, just like the native mpv window.

- preview/next frame shortcuts should work for the mpv window when playing video

- no volume/mute controls yet, these will come in the coming weeks, including global mute settings

- updated media show and sizing code to account for mpv widgets

- reworked my animation scanbar to talk to mpv, and for my mpv window to talk back to it

- improved the animation scanbar to be more flexible when frame position and num_frames are not available, both in displaying info and calculating scanbar seek clicks

- mpv api version added to help->about

- .

- new downloader objects:

- thanks to a user, updated the 'pixiv artist page' url class to a new object that covers more situations. the defunct 'pixiv artist gallery page' url class is removed

- added 8kun and vch.moe download support. I got started on julay, smug, and endchan, but they were a little more tricky and I couldn't finish them in time–fingers crossed, next week

- .

- menu quality of life:

- a right-click on thumbnail whitespace will now not send a 'deselect all' event! feel free to right-click in empty space to do an easy remove->selected

- remorked the tag menu layout to move less frequently used actions down:

- - moved the discard/require/permit/exclude search predicate actions down

- - moved 'open in a new page' below select and copy

- - moved copy above select

- and some misc menu layout improvement on this menu

- fixed some labelling with the discard/require/permit/exclude verbs on negated tags

- right-clicking on system search predicates now shows the 'copy' menu correctly

- system predicates that offer easy inverse versions (like inbox/archive) should now offer the 'exclude' verb

- when right-clicking on a single tag that has siblings, its siblings and those siblings' subtags will now be listed in the copy menu!

- copying 'all' tags from a list menu, with or without counts, will now always copy them in the list order

- across the program, all menu 'labels' (menu text items that do not have a submenu and have no associated action, like 'imported 3 years 7 months ago') will now copy their text to the clipboard. let's see how it goes

- .

- other ui quality of life:

- across the program's UI, filetypes are now referred to with simpler terms rather than technical mimetypes. instead of 'image/jpg', it is now typically just 'jpeg'

- the 'remove selected' buttons on the gallery and watcher pages are now smaller trash icon buttons

- the new page chooser will now auto-dismiss if it loses focus–so if you accidentally launch it with a middle-/double-click somewhere, just click again and it'll go away

- hitting enter or return on the new page chooser now picks the 'first' button, scanning from the top-left. hitting enter twice now typically opens a new 'my files' search page

- added pause_media and pause_play_media shortcuts to the media_viewer shortcut set. new clients will start with space keypress performing pause_play_media

- added pause_play_slideshow shortcut to the media_viewer_browser shortcut set. this shortcut is no longer hardcoded by space keypress

- the six default shortcut sets now have a small description text on their edit panels

- the options->media edit panels now enable/disable widgets better based on current media/preview action

- added a checkbox to _options->gui pages_ to set whether middle-clicking a tag in the media viewer or a child tag manager to open a tag search page will switch to the main gui. default is false

- mr bones now reports total files, total filesize, and average filesize

- mr bones now loads your fate asynchronously

- .

- the rest:

- added tentative and simple realvideo (.rm) and realaudio (.ra) support–seems to work ok, but some weirder variable bit rate formats may not, and I have collapsed the various different extensions just down to .rm or .ra

- added trueaudio (.tta) audio support

- fixed a bug from the recent search optimisations where a bare inbox search would not cross-reference with the file domain (so some trash could show up in a simple inbox/'my files' query)

- fixed an issue with searching for known urls by url class where the class was for a third-or-higher-level domain and was not set to match subdomains (this hit 4chan file urls for a few users)

- fixed the issue with 'open externally' button panel not clearing their backgrounds properly

- fixed some of the new unusual stretchy layouts in the options dialog

- removed overhead from subscriptions' 'separate' operation, which should stop super CPU hang when trying to split a subscription with hundreds of thousands of urls

- fixed an issue where the advanced file delete dialog would not show the simple 'permanent delete' option when launched from the media viewer's right-click menu

- fixed the select/remove actions for local/remote

- fixed 'set_media_focus' from manage tags to correctly activate the underlying media viewer as well as set focus

- stopped the 'file lookup script' status control from resizing so wide when it fetches a url

- fixed a rare mouse wheel event handling bug in the media viewer

- reduced db overhead of the 'loading x/y' results generation routine. this _may_ help some users who had very slow media result loading

- cleaned up how the server reports a bootup-action error such as 'cannot shut down server since it is not running'–this is now a simple statement to console, not a full error with trace

- improved client shutdown when a system session shutdown call arrives at the same time as a user shutdown request–the core shutdown routine should now only occur once

- fixed an issue with thumbnail presentation on collections that have their contents deleted during the thumbnail generation call

- misc wx->Qt layout conversion improvements

- updated the github readme to reflect some new links and so on

- misc code cleanup

next week

I pushed it a bit hard this week, so I am going to try to take it easy. Hopefully I will be able to get some volume/mute controls going for mpv and then launch it for all users, including on the Linux and macOS builds. Assuming there are no catastrophic problems this week, I will be overwriting user settings to move all 'show as normal' video and audio options to 'show with mpv'. Then these new imageboard parsers and some more Qt cleanup work.

R: 0 / I: 0 / P: 3 [R] [G] [-]

Remote backups with Hydrus

Hello everybody, I've been wondering how some of other Hydrus Network uses handle backups for their files.

My current setup is like thus: I make weekly local backups of my database, which are then sent to a NAS through UrBackup, From there, I've been using Duplicati with cheap S3-compatible remote object storage.

Now, supposedly Duplicati performs deduplication before uploading, however it still seems like it takes its sweet time on the SQLite database files themselves, so I'm wondering if their format just doesn't lend itself to be deduplicated adequately or if there is some other issue. I don't use password protection (although from what I've seen, it doesn't actually encrypt anything by itself so it shouldn't make much of a difference either way)

So, what other methods do you use to backup your Hydrus files remotely? Ideally it should be with deduplication because although space is not at a premium for me, my upload speeds are very limited.

R: 17 / I: 2 / P: 3 [R] [G] [-]

Hentai Project

Due to the recent take down of ExHentai, is pretty alarming that the same might happen to similar sites. So I propose to anons using Hydrus to save the doujins they like within the platform and tag them along so whenever Hydrus reaches an stage where sharing is easy, maybe you'll have not only a decent alternative to Sad Panda, but a lasting one.

Numerous ideas have been thrown around about it, but here is what the BO have confirmed:

1) Hydrus is not currently great at handling paged media. It can do it, but the workflow is a little awkward. I plan to improve this in future.

2) IPFS support is prototype, and mostly for advanced users who are already comfortable with the program. I'm working on it now.

3) Hydrus is not for everyone. If you have 10,000+ files and want to manage them better than files and folders, you may like it. If you want a clean and professional experience with beautiful UI, you may not.

So it's nothing but a project now. The main reason I do the thread is to keep anons up to support it and use Hydrus for managing whatever manga they downloaded to be used later when the project get to the point it can be used and to post resources and the such.

As so, here are a couple.

An index with tags and names:

https://mega.nz/#!dBxmUADB!4piJItp7ja7_9WbTmoLGo3nMLOP2Fr0AH_ju9W4-PLY

Relevant boards:

>>>/ipfs/

>>>/hentaiclub/

Maybe one day, a true alternative to ExHentai can be done by us, for us.

R: 23 / I: 5 / P: 3 [R] [G] [-]

Version 379

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v379/Hydrus.Network.379.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v379/Hydrus.Network.379.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v379/Hydrus.Network.379.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v379/Hydrus.Network.379.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v379.tar.gz

Happy New Year! Although I have been ill, I had a great week, mostly working on a variety of small jobs. Search is faster, there's some new UI, and m4a files are now supported.

search

As hoped, I have completed and extended the search optimisations from v378. Searches for tags, namespaces, wildcards, or known urls, particularly if they are mixed with other search predicates, should now be faster and less prone to spikes in complicated situations. These speed improvements are most significant on large clients with hundreds of thousands or millions of files.

Also, like how system:inbox and system:archive 'cancel' each other out, a few more kinds of search predicate will remove mutually exclusive or redundant predicates already in the search list. system:limit predicates will remove other system:limits, system:audio/no audio will nullify each other, and–I may change this–any search predicate will replace system:everything. I have a better system for how this replacement works, and in the coming weeks I expect to extend it to do proper range-checking, so a system:filesize<256KB will remove a system:filesize<1MB or system:filesize<16KB or system:filesize>512KB, but not a system:filesize>128KB.

downloaders

I have started on some quality of life for the downloader UI. Several of the clunky buttons beneath the page lists are now smaller icons, you can now 'retry ignored' files from a button or a list right-click, any file import status button lets you right-click->show all/new in a new page, and the file import status list now lets you double-click/enter a selection to show that selection in a new page.

I have rolled in a fixed derpibooru downloader into the update. It seems to all work again.

With the pixiv login script confirmed completely broken with no easy hydrus fix in sight, if you have an 'active' record with the old, now-defunct default pixiv login script, this week's update will deactivate it and provide you with a note and a recommendation to use the Hydrus Companion web browser addon in order to login.

the rest

m4a files are now supported and recognised as audio-only files. These were often recognised as mp4s before–essentially, they are just mp4s with no video stream. I have made the choice for now to recognise them as audio-only even if they have a single frame 'jpeg' video stream. I hope to add support to hydrus for 'audio+picture' files soon so I can display album art better than inside a janked single-frame video.

The 'remove' and 'select' menus on the thumbnail right-click have been improved and harmonised. Both now lay out nicely, with file service options (like 'my files' vs 'trash' when there is a mix), and both provide file counts for all options. Support for selecting and removing from collected media is also improved.

full list

- downloaders:

- the right-click menus from gallery and watcher page lists now provide a 'remove' option

- gallery and watchers now provide buttons and menu actions for 'retry ignored'

- activating a file import status list (double-clicking or hitting enter on a selection of rows) now opens the selection in a new page

- file import status buttons now have show new/all files on their right-click menus

- on gallery and watcher pages, the highlight, clear highlight, pause files, and pause search/check buttons are now smaller bitmap buttons

- as the old default pixiv login script is completely broken, any client with this active will have it deactivated and receive an update popup explaining the situation and suggesting to use Hydrus Companion for login instead

- updated the derpibooru downloader

- .

- search:

- when search predicates are added to the active search list, they are now better able to remove existing mutually exclusive/redundant predicates:

- - system:limit, hash, and similar to predicates now remove other instances of their type

- - system:has audio now removes system:no audio and vice versa

- - any search predicate will remove system:everything (see how you feel about this)

- improved 378's db optimisation to do tag searches in large file domains faster

- namespace search predicates ('character:anything' etc…) now take advantage of the same set of temporary file domain optimisations that tag predicates do, so mixing them with other search predicates will radically improve their speed

- wildcard search predicates, which have been notoriously slow in some cases, now take full advantage of the new tag search optimisations and are radically faster when mixed with other search predicates

- simple tag, namespace, or wildcard searches that are mixed with a very large system:inbox predicate are now much faster

- a variety of searches that include simple system predicates are now faster

- integer tag searches also now use the new tag search optimisation tech, and are radically faster when mixed with other search predicates

- system:known url queries now use the same temporary file domain search optimisation, and a web-domain search optimisation. this particularly improves domain and url class searches

- fixed an issue with the new system:limit sorting where sort types with non-comprehensive data (like media views/viewtime, where files may not yet have records) were not delivering the 'missing' file results

- improved the limit/sort_by logic to only do sort when absolutely needed

- fixed the system:limit panel label to talk about the new sorted clipping

- refactored tag searching code

- refactored namespace searching code

- refactored wildcard searching code and its related subfunctions

- cleaned all mappings searching code further

- .

- the rest:

- m4a files (and m4b) are now supported and recognised as separate audio-only mp4 files. files with a single jpeg frame for their video stream (such as an album cover) should also be recognised as audio only m4a for hydrus purposes for now. better single-frame audio support, including functional thumbnails and display, is planned for the future. please send in any m4a or m4b files that detect incorrectly

- the remove thumbnail menu has been moved to a new, cleaner file filtering system. it now presents remove options for different file services and local/remote when available (most of the time, this will be 'my files'/'trash' appearing when there is a mix), including with counts for all options

- the select thumbnail menu is also moved to this same file filtering system. it has a neater menu, with counts for each entry. also, when there is no current focus, or it is to be deselected, the first file to be selected is now focused and scrolled to

- for thumbnail icon display and internal calculations, collections now _merge_ the locations of their members, rather than intersecting. if a collection includes any trash, or any ipfs members, it will have the appropriate icon. this also fixes some selection-by-file-service logic for collections

- import folders, export folders, and subscriptions now explicitly only start after the first session has been loaded (so as well as freeing up some boot CPU competition, a quick import folder will now not miss publishing a file or two to a long-loading session)

- the subscription manager now only waits 15s before starting first work (previously, the buffer was 60 seconds)

- rearranged migrate tags panel so action comes before destination and added another help text line to clarify how it works. the 'go' confirmation dialog now summarises tag filtering as well

- tag filter buttons now have a prefix on their labels and tooltips to better explain what they are doing

- the duplicate filter right-center hover window should now shorten its height appropriately when the pairs change

- fixed a couple of bugs that could appear when shutting down the duplicate filter

- hackily 'fixed' an issue with duplicates processing that could cause too many 'commit and continue?' dialogs to open. a better fix here will come with a pending rewrite

- dejanked a little of how migrate tags frame is launched from the manage tags dialog

- updated the backup help a little and added a note about backing up to the first-start popup

- improved shutdown time for a variety of situations and added a couple more text notifications to shutdown splash

- cleaned up some exit code

- removed the old 'service info fatten' maintenance job, which is not really needed any more

- misc code cleanup

- updated to Qt 5.14 on Windows and Linux builds, OpenCV 4.1.2 on all builds

next week

Next week is a 'medium size job' week. Now I am more comfortable with Qt, I would love to see if I can get an MPV window embedded into hydrus so we finally have legit video+audio support. I can't promise I can get anything but a rough prototype ready for 380 for all platforms, and there is a small chance it just won't work at all, but I'll give it a go.

Hydrus had a busy 2019. Starting with the jump to python 3, and then the duplicate storage and filter overhaul, the Client API, OR search, proper audio detection, the file maintenance system, multiple local tag services, tag migration, asynchronous repository processing, fast tag autocomplete, and all the smaller improvements to downloaders and UI workflow and latency and backend scheduling and optimisations for our growing databases, and then most recently with the huge Qt conversion. The wider community also had some bumps, but we survived. Now we are in 2020, I am feeling good and looking forward to another productive year. There are a couple of thousand things I still want to do, so I will keep on pushing and try to have fun along the way. I hope you have a great year too!

R: 0 / I: 0 / P: 3 [R] [G] [-]

Let's get this back going!

R: 1 / I: 0 / P: 3 [R] [G] [-]

Version 378

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v378/Hydrus.Network.378.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v378/Hydrus.Network.378.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v378/Hydrus.Network.378.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v378/Hydrus.Network.378.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v378.tar.gz

I had a great, simple week. Searches are less likely to be very slow, and system:limit searches now sort.

all misc this week

I identified a database access routine that was sometimes not taking an optimal route. Normally it was fine, but with certain sizes or types of query, it could take a very long time to complete. This mostly affected multi-predicate searches that included certain tags or system:duration and system:known urls, but the routine was used in about 60 different places across the program, including tag and duplicate files processing. I have rewritten this access routine to work in a more 'flat' way that will ensure it is not so 'spiky'.

Also in searching, I managed to push all the 'simple' file sorts down to file searches that have 'system:limit'. If you search with system:limit=256 and are sorting by 'largest files first', you will now see the 256 largest files in the search! Previously, it would give a random sample. All the simple sorts are supported: import time, filesize, duration, width, height, resolution ratio, media views, media viewtime, num pixels, approx bitrate, and modified time. If you want something fun, do a search for just 'system:limit=64' (and maybe system:filetype) and try some different sorts with F5–you can now see the oldest, smallest, longest, widest, whateverest files in your collection much easier.

There are also some fixes: if you had sessions not appearing in the 'pages' menu, they should be back; if you have had trouble with ipfs directory downloads, I think I have the file-selection UI working again; 'remove files when trashed' should work more reliably in downloader pages; and several tag and selection lists should size themselves a bit better.

full list

- if a search has system:limit, the current sort is now sent down to the database. if the sort is simple, results are now sorted before system:limit is applied, meaning you will now get the largest/longest/whateverest sample of the search! supported sorts are: import time, filesize, duration, width, height, resolution ratio, media views, media viewtime, num pixels, approx bitrate, and modified time. this does not apply to searches in the 'all known files' file domain.

- after identifying a sometimes-unoptimal db access routine, wrote a new more reliable one and replaced the 60-odd places it is used in both client and server. a variety of functions will now have less 'spiky' job time, including certain combinations of regular tag and system search predicates. some jobs will have slightly higher average job time, some will be much faster in all common situations

- added additional database analysis to some complicated duplicate file system jobs that adds some overhead but should reduce extreme spikes in job time for very large databases

- converted some legacy db code to new access methods

- fixed a bug in the new menu generation code that was not showing sessions in the 'pages' menu if there were no backups for these sessions (i.e. they have only been saved once, or are old enough to have been last saved before the backup system was added)

- fixed the 'click window close button should back out, not choose the red no button' bug in the yes/no confirmation dialogs for analyze, vacuum, clear orphan, and gallery log button url import

- fixed some checkbox select and data retrieval logic in the checkbox tree control and completely cleared out the buggy ipfs directory download workflow. I apologise for the delay

- fixed some inelegant multihash->urls resolution in the ipfs service code that would often mean a large folder would lock the client while parsing was proceeding

- when the multihash->urls resolution is going on, the popup now exposes the underlying network control. cancelling the whole job mid-parse/download is now also quicker and prettier

- when a 'downloader multiple urls' popup is working, it will publish its ongoing presented files to a files button as it works, rather than just once the job is finished

- improved some unusual taglist height calculations that were turning up

- improved how taglists set their minimum height–the 'selection tags' list should now always have at least 15 rows, even when bunched up in a tall gallery panel

- if the system clock is rewound, new objects that are saved in the backup system (atm, gui sessions) will now detect that existing backups are from the future and increase their save time to ensure they count as the newest object

- short version: 'remove files from view when trashed' now works on downloader thumbs that are loaded in from a session. long version: downloader thumb pages now force 'my files' file domain for now (previously it was 'all local files')

- the downloader/thread watcher right-click menus for 'show all downloaders xxx files' now has a new 'all files and trash' entry. this will show absolutely everything still in your db, for quick access to accidental deletes

- the 'select a downloader' list dialog _should_ size itself better, with no double scrollbars, when there are many many downloaders and/or very long-named downloaders. if this layout works, I'll replicated it in other areas

- if an unrenderable key enters a shortcut, the shortcut will now display an 'unknown key: blah' statement instead of throwing an error. this affected both the manage shortcuts dialog and the media viewer(!)

- SIGTERM is now caught in non-windows systems and will initiate a fast forced shutdown

- unified and played with some border styles around the program

- added a user-written guide to updating to the 'getting started - installing' help page

- misc small code cleanup

next week

I am going to take a few days off for the holiday and make the next release in two weeks, for New Year's Day. I expect to do some small jobs, push more on the database optimisation, continue improving the UI layout code, and perhaps put some time into some space-clearing database maintenance.

𝕸𝖊𝖗𝖗𝖞 𝕮𝖍𝖗𝖎𝖘𝖙𝖒𝖆𝖘!

R: 0 / I: 0 / P: 3 [R] [G] [-]

did the board go back in time?

R: 12 / I: 4 / P: 3 [R] [G] [-]

Version 377

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v377/Hydrus.Network.377.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v377/Hydrus.Network.377.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v377/Hydrus.Network.377.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v377/Hydrus.Network.377.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v377.tar.gz

I had a good week. I mostly caught up with my smaller jobs queue, just pushing on work that had piled up.

Qt

Thank you for the continuing Qt bug reports. I fixed a variety of typo-broken buttons this week, mostly buried in less-used UI, and if you have set a browser launch path override in the options, hyperlinks across the program should obey that again. I also believe I fixed the annoying issue where media viewer hover windows that needed to shrink (because of switching to a new media that had fewer info lines or known urls) would linger too tall for one frame. EDIT: I'm still having slight hover window resize flicker in my IRL client when I keep my mouse over the top-right hover, I'll give it another go next week.

Furthermore, all non-menubar menus across the program now open on right-click release instead of press. This sounds trivial, but there was a mix of down/up click handling here, now unified, and some things like thumbnail and tag selection are handled more naturally in the initial click down. This also fixes an issue where users who had menu styles with no significant border were sometimes accidentally activating the first menu item on a slow click (the click down would show the menu, the click up would activate it).

Last week's support for styles and stylesheets for Qt seems to have gone well. There is more to do here, but this week brings better error handling for unusual situations, better 'we already have this set, no need to waste CPU re-applying it' detection, and fewer hacky colour overrides, so custom stylesheets should be able to colour regular panels without lots of unintended grey or white boxes.

the rest

The shortcut system should now be able to support any text key. This includes extended characters like ø or æ. If you need to apply a modifier to get these keys (e.g. ctrl+alt+e), these modifiers are likely to show up spuriously in the shortcut's label (e.g. ctrl+alt+æ), but everything should nonetheless line up in the program correctly. Let me know if you discover any new bugs here!

There are also some new shortcut commands this week, for the 'media viewer' set. They are 'pan_top/bottom/left/right_edge' and 'pan_horizonal/vertical_center'. They move the whole image around to edges, or center it again, and should be helpful if you often look at tall or wide comics at 100% zoom. There will be more to come here in coming weeks.

The work on making the UI smoother is also continuing. Session saving and a variety of 'media refresh' events that occur on file import or maintenance are now significantly faster and do the bulk of that work in the background. Users with very large sessions should experience less juddery behaviour during both idle and heavy work periods.

full list

- qt:

- all non-menubar menus across the program now launch on click release. some previously launched on click press. a variety of related click event behaviour is cleaned up, particularly with thumbnail/tag selection on the click down. this also fixes some users' menus immediately activating the first entry on slow clicks in some ui styles

- I think I fixed the annoying single-frame delayed size-down resize on media viewer hover frames when changing media!

- the vast majority of old wx panel background colour hacks are removed, so custom stylesheets should now cover much more of the UI

- improved the new custom style and stylesheet setting, resetting, and error handling code, particularly for not re-applying the same style or stylesheet twice, and for handling un-re-settable styles (seems to be defaults initialised by third-party OS-wide Qt style) gracefully

- fixed hyperlinks not using the custom web browser launch path as set in the options

- fixed the 'migrate entire db' and 'set thumb location' buttons in the migrate database dialog

- fixed a typo bug when launching the url selection tree after adding an ipfs directory to download

- fixed two typo bugs when editing regex favourites and simple downloader formulae

- fixed an issue where custom shortcut sets could not be deleted

- fixed a typo in the edit account type panel

- fixed sorting the login listctrl when there are session logins mixed with non-session logins

- removed some old media viewer hover window display/raise hacks

- retired the 'always show hover windows' debug mode

- the media viewer will no longer perform any drag calculations on anything but left-click drag

- misc Qt code refactoring/cleanup

- .

- url searching:

- the database now stores 'known url' domain information more efficiently. it will take a few moments/minutes to reshape the db when updating

- system:known url's exact url search now runs extremely fast. this will only affect new predicates of this type, not those in existing sessions

- system:known url's domain search now runs much faster and matches subdomains of the given domain. this will only affect new predicates of this type, not those in existing sessions

- system:known url's url class search now runs much faster. this will only affect new predicates of this type, not those in existing sessions

- when entering a regex system:known url predicate, the dialog will now not OK (throwing up an error dialog) if the regex is invalid

- .

- the rest:

- the shortcut system now allows all text characters. if it has text, it should work, but it is the wild west in terms of modifier labelling. anything unusual on your keyboard like ctrl+alt+e to make æ will _display_ as ctrl+alt+æ, but the same key combination will match up in the program all correct

- added shortcut actions 'pan_top_edge', 'pan_bottom_edge', 'pan_left_edge', 'pan_right_edge' to the media viewer shortcut set that will move the current image so the respective edge is aligned with the larger canvas's

- added shortcut actions 'pan_horizontal_center' and 'pan_vertical_center' to do as above but center on that axis

- session save now hangs the UI significantly less, whether triggered by user command or auto-saving 'last session'

- saving of last/exit sessions on client close is a little faster

- the call to refresh thumbnail file info (and redraw if needed) when a file is imported or has metadata-regenererating file maintenance done will now only call for files that are actually loaded, run faster per file, run faster when the client has large collections in its session, and not hang the ui thread when waiting for the new media info to arrive

- like regular popups, modal popups (like those created when big vacuum/analyze jobs jump in) will now only appear if the main gui or an on-parent child has OS focus

- the main gui/on-parent child OS focus test now includes misc child windows like the autocomplete results hover window

- network jobs that fail for one reason or another will now be more reliably cleaned up, and their connections returned to the connection pool. this may fix the 'too many open file handles' errors some users were seeing after long term unreliable network traffic

- fixed an issue where some thumbnails that were trashed or physically deleted were being removed from 'all known files' and file repository views when it was not appropriate

- connection and downloader retry time options now have a wider min/max range when in advanced mode, with an accompanying warning label for the connection panel

- checker options times now have a wider min/max range when in advanced mode, with an accompanying warning label

- cleaned up some shutdown reporting text

- misc debug improvements

next week

The holidays are coming up. As 379 would normally be due on Christmas Day, I will postpone that to New Year, which means next week's 378, on the 18th, will be the last release of the year. I am due for a 'cleanup' week, which will also make for a nice 'safe' release before the break. So I will clean up code, do some more Qt neatening, and otherwise catch up on smaller jobs like I did this week.

I never strictly marked down the day when I started working on hydrus, but December 14th, 2011, the date of the first non-experimental beta, has emerged as a decent birthday. This week marks eight years. It has been a lot of work and a lot of fun. There are a couple thousand things I would still like to do, so if I can, I would like to keep going as I have into the 2020s. I deeply appreciate all the feedback, help, and support over the years. Thank you!

If you would like to further support my work and are in a position to do so, my simple no-reward Patreon is here: https://www.patreon.com/hydrus_dev

R: 2 / I: 0 / P: 3 [R] [G] [-]

thank you mr. hydrus for the software

now i can download my furry porn from 8kun without having to fix my old scraper

R: 0 / I: 0 / P: 3 [R] [G] [-]

Release Tomorrow!

I had a good week. I focused on catching up on smaller work. More bugs are fixed, there's some neat new shortcuts to pan media around, the UI should be smoother when importing files and saving sessions, and the system:known urls predicates now run much faster.

The release should be as tomorrow.

R: 0 / I: 0 / P: 3 [R] [G] [-]

Version 376

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v376/Hydrus.Network.376.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v376/Hydrus.Network.376.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v376/Hydrus.Network.376.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v376/Hydrus.Network.376.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v376.tar.gz

I had an ok to good week. Subscriptions run better and there are more UI fixes and basic theming for Qt.

subscriptions

Subscriptions have always been resource heavy. As last year's downloader overhaul extended their capabilities, we saw increased CPU lag with many subscription operations. I had hoped to completely fix this this week, even permitting subs with >250,000 items cached, but the surgery needed to figure this out was more complicated than I expected. While I otherwise prepped for this next step, I also tightened up subscription timing code and wrote a new scheduler that reduces overall subscription resource needs.

Subscriptions will now try to load when they are due, and only then. So, if a query in your sub is due to check in 45 minutes, or if it has file to download but has run out bandwidth for 45 minutes, it should now run, typically, in 45-60 minutes. They will also respond better to cancelling and pause events, and if only 30 of 100 queries are due for a check in this run, or have outstanding files to download, you will now see x/30 on the popup progress text and gauge, just as better indication of how much work remains.

Previously, subs checked every four hours. It meant a bunch of wasted CPU for subs that were not due, late checks for subs that were due just moments after the check time, and also a delayed 'clumping up' of work. Today's change is subtle but significant, so I am interested in feedback from all users of subs–if you notice anything odd, bad, or great, please let me know. Ideally, subs will be quieter for you, running more often when you are not looking.

Although the data overhaul was too much work to do in one week, I now have a good plan and am in better position to get it done. I really want subs to load and save their metadata in milliseconds so the manage subs dialog can launch equally quickly and the client api can start talking to them. The year is fading, so I am not sure when this will get done, but hopefully this week's scheduling will at least relieve some of the work involved.

Qt

Hydrus now has very simple support for Qt styles. Please check the new options->style page. All users are encouraged to try out the 'style' option, which should have one or more flavours for your OS and also 'Fusion'. These will change how buttons and other widgets are drawn across the program.

Advanced users may wish to play with 'stylesheet' support, which uses css/qss files to change colours and layouts using portable files. This is still prototype, just rough importing of stylesheets from a barebones folder in the install dir, and is likely to be replaced with something more user-friendly and technically sensible in the coming weeks. If you play with this, I am interested in areas where hydrus's old colour system conflicts foolishly with a qss stylesheet (such as some panel backgrounds and the green/red yes/no buttons on many dialogs), and also recommendations for stylesheets you would like bundled as future hydrus defaults.

The macOS build can now do page tab drag and drop again! Media drag and drops between tabs should also work. Everything should line up right and work like the Windows and Linux builds, but please let me know if it does not. A tip here, if you still have trouble with a centered page tab bar, is just to set the Fusion style as above, which will make hydrus look more like a standard PC application. You may prefer the macOS look, but if not, you now have the option.

Beyond a bunch of general bugfixes, I also improved the stability of some review bandwidth panels and some bad 'should I close/exit?' dialogs that were either not respecting 'no' answers or causing unusual behaviour as 'yes' closes were going on.

the rest

Pixiv tag search and twitter username lookup get updated default downloaders this week. Pixiv should be fixed, and twitter should work more efficiently, only getting 'media' tweets.

Client shutdown should now be faster and snappier for most users.

File export filename phrases now support a new '{file_id}' term that substitutes a unique permanent numerical identifier. If you need to create guaranteed-unique filenames, this is easier than using the '{hash}' term.

full list

- subscriptions:

- wrote a new subscription manager to better look after subscription scheduling

- rather than checking every four hours or after manage subs dialog close, subscriptions now record an indication of when they are next due for work, whether that is the estimated next check time or when bandwidth is free on remaining file downloads, and launch in a fifteen-minute window around that time. delays due to previous errors or user cancels are also taken into account. this reduces background cpu and i/o greatly for clients with large subs

- if a sub is paused, or all its queries are paused, it will now never be reloaded after first load until a change via the manage subs dialog

- furthermore, if a single sub takes a very long time to work, the whole sublist can re-cycle if they come up due for more work before it is finished

- if a sub query is DEAD but still has outstanding files to download, it will no longer automatically pause

- subs now clean up more tidily if they are running on a program exit

- the subscription popup now shows check/file progress based on the number of queries that appear to have pending work. instead of 'query 300/450' with 420 that aren't due, you'll get 'query 12/30'. if a query becomes due during a round of checking, another round of checking will run

- if a subscription fails to load from the db, the error is handled better and no more subs will run in that boot

- improved subscription startup checking logic, tightening up various paused/dead/cansync tests

- improved subscription interrupt checking logic, tightening checks on global network pause and various shutdown scenarios

- cleaned up some more subscription code in prep for data storage breakup

- .

- qt:

- added experimental Qt style settings to the new options->style page! all users should now be able to set Fusion style, and perhaps some alternate OS styles. advanced users are invited to play around with QSS stylesheets (although be warned that some of hydrus's custom colour system overrides QSS, so more work is needed here), which will be extended and made user-friendly in coming weeks

- fixed tab position calculations for all tab/media drag and drops for tab bars that are centered or otherwise positioned far off top-left alignment

- fixed tab drag and drop event object handling for macOS. tab and media DnD is now enabled for macOS

- the popup toaster can now unhide if an on-top-of-parent non-modal frame (like review services) is focused (so hitting 'process now' should show you the work)

- fixed a variety of old hacky wx close-window veto tests. the 'close client?' confirmation dialog will now reliably veto a close requent on 'no'/cancel, dialog close events that are vetoed (such as closing the manage tags dialog with pending tags) will now veto more than just the first time, and several bad media viewer archive/dupe filtering cancel and end-of-window events should now work more cleanly and correctly. users who had crashes at the end of filtering may find they are stable again

- as a quick patch against some multiline notes and statuses, list controls now force single-line text in all cells

- list controls now tooltip all cells

- fixed the shutdown splash not updating after the daemons shut down (lmao)

- 'modal' message dialogs, which are created by blocking maintenance tasks such as vacuum, will no longer raise the program to the foreground on creation

- should have fixed the taglist vertical positioning jank that could occur in the row after a tag with a tall emoji unicode character (and also sometimes kanji/hangul)

- fixed a typo bug that was throwing an error for the upnp port widget in the local client server management panel when 'allow non local connections' was checked

- improved stability of bandwidth review panel bandwidth rules refresh

- improved stability of review services bandwidth rules refresh

- improved some dialog cleanup code

- reverted a bad environment-setting change put in last week that was causing some running-from-source users trouble

- misc qt code cleanup

- .

- the rest:

- updated the default pixiv tag search downloader to one submitted by a user. it now uses their api

- updated the default twitter username lookup to a downloader submitted by a user. it fetches just the media tweet feed, making it more efficient. also added (but not linked by default) is a new tweet parser that can fetch most videos using a third-party site, advanced users may wish to play with this

- added a &#123;file_id&#125; term for file export phrases that substitutes a unique and permanent numerical file identifier

- fixed an issue where idle maintenance jobs could sometimes sneak in a few milliseconds of work during certain long shut down pauses, such as while waiting for a 'should I do shutdown work?' dialog to return. program shutdown should be snappier for many users as forced startup delays in these calls will no longer trigger

- added a date 'encode' string transformation rule, which takes an integer timestamp and converts it to a pretty date string. the date rules are now renamed to the clearer 'datestring to timestamp' and vice versa

- fixed page parser edit panel's 'test parse' when string transformations perform pre-parsing conversion. the handling and passing of test data for all the panels here is cleaned up throughout

- system:limit predicate edit panel now has a small label describing its sampling behaviour

- updated the various 8chan links in the client and help to 8kun, let me know if I missed any, and added Endchan bunker link to help menu

- improved some misc status text handling across the program

- refactored cache and manager code into different, simpler files

- updated sqlite on windows build to 3.30.1

next week

It is already December! For me, and I know for many of you, 2019 has just disappeared. Christmas is exactly three weeks away, so I am afraid some of my hopes for significant tag overhaul like tag siblings or processing filters may have to be delayed a little. I have a bit more Qt work to do, but also plenty of regular small jobs and bugfixes, so for next week I will take a simple bit of misc work, nothing huge, just to catch up on some things. I also have messages to catch up on.

If you were an 8chan user and missed the news, the /hydrus/ board is back on 8kun.top. The site is still unstable, and reception from the community is mixed, but against all odds they are back up and improving. I will keep posting release posts on Endchan and 8kun simultaneously for the time being, so please use whichever you feel better about, or just hit me on discord/tumblr/email if you prefer.

R: 6 / I: 1 / P: 3 [R] [G] [-]

Version 375

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v375/Hydrus.Network.375.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v375/Hydrus.Network.375.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v375/Hydrus.Network.375.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v375/Hydrus.Network.375.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v375.tar.gz

I had a great week. There are a bunch more Qt fixes, and a few other things as well.

Qt

I have fixed a bunch more bugs in the Qt code. We are getting to the end now–this is mostly smaller stuff like an unusual dialog button not working, but I have fixed another important memory leak that was causing some backend not to be deleted correctly when a media viewer closed on a video. This should radically reduce memory use for some heavily used clients.

Some windows that were large, or could expand to be, like the options dialog on some pages, were sizing off the edge of the screen. This should be fixed now, and a variety of child-window initial size calculations should be a bit more accurate–Qt manages window size a little differently than wx, and the additional buffer represented by the window title frame and border was not being taken into account.

The menubar menus should work a little snappier this week. Things like the rapid 'pending (xxx)' menu updates when a downloader is importing files with tags. The whole UI should get a little latency benefit from this during these high-traffic times.

I also gave the layout and scaling another go on clients that use UI scaling (when you tell your OS to display UI at >100% on a monitor) on high dpi displays. The experiment from last week did not go how I wanted, with pixelly scaled-up thumbnails, so I have reworked that and fixed the taglist, which was crunching tags together on these displays.

R: 0 / I: 0 / P: 3 [R] [G] [-]

Release Tomorrow!

I had a great week. I have fixed some more Qt problems, including some bad window positioning, thin popup message widths, a video memory leak, and another round of improvements for high-dpi displays. Also are some improvements to the file maintenance system for recovering (including auto-redownloading) missing files, some better network retry timing code, and new downloaders for some broken sites.

The release should be as normal tomorrow. I will be making a release post here as well as Endchan.

R: 17 / I: 6 / P: 3 [R] [G] [-]

Version 362

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v362/Hydrus.Network.362.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v362/Hydrus.Network.362.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v362/Hydrus.Network.362.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v362/Hydrus.Network.362.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v362.tar.gz

I had a mixed week. The duplicates overhaul work is finished.

duplicates work finished

The duplicates storage overhaul is done! Everything is now on the new storage system, the duplicate filter has had some quality of life attention, and now there is some updated help:

https://hydrusnetwork.github.io/hydrus/help/duplicates.html

If you have used hydrus for a bit but haven't checked out the duplicate system yet, this is a good time.

I added some final new things this week: the duplicates filter now highlights common resolutions, like 720p and 1080p, and there is a new 'comparison statement' for common ratios like 4:3 or 16:9. Also, the thumbnail right-click file relationships menu now provides a choice to clear out potential relationships and, if in advanced mode, perform some en masse remove/reset actions on multiple thumbnails at once.

I am now free to move on to another large job. Audio support was very popular at the last vote, so I will spend a couple of weeks trying to get some simple 'has audio' metadata going, but then I am going to address some growing issues related to tag repositories, easier 'I want these tags' management, namespace siblings, and multiple local tag services.

deepdanbooru plugin

If you are an experienced user and interested in testing out some neural net tagging, check this out:

https://gitgud.io/koto/hydrus-dd/

This project by a hydrus user lets you generate tags using the DeepDanbooru model and get them into hydrus in a variety of ways. If you give it a go, let me know how it goes and what I can do to make it work better on the hydrus end.

full list

- duplicates work finished:

- updated the duplicates help text and screenshots to reflect the new system

- duplicate files search tree rebalancing is now done automatically on the normal idle maintenance routine, and its over-technical UI is removed from the duplicates page

- the duplicate filter's resolution comparison statement now specifies 480p, 720p, 1080p, and 4k resolutions and highlights resolutions with odd (i.e. non-even) numbers

- if the files are of different resolution, a new 'ratio' comparison statement will now show if either have a nice ratio, with current list 1:1, 4:3, 5:4, 16:9, 21:9, 2.35:1

- added a 'stop filtering' button to the duplicate hover frame

- made the ill-fitting 'X' button on top hover frame a stop button and cleaned up some misc related ui layout

- added a 'remove this file's potential pairs' command to the thumbnail file relationships menu

- if in advanced mode, multiple thumbnail selection right-click menus' file relationships submenus will now offer mass remove/reset commands for the whole selection. available commands are: 'reset search', 'remove potentials', 'dissolve dupe groups', 'dissolve alt groups', 'remove false positives'

- .

- the rest:

- added link to https://gitgud.io/koto/hydrus-dd/ , a neat neural net tagging library that uses the DeepDanbooru model and has several ways of talking to hydrus, to the client api help

- cleaned up a little of the ipfs file download code, mostly improving error/cancel states

- rewrote some ancient file repository file download code, which ipfs was also using when commanded to download via a remote thumbnail middle-click. this code and its related popup is now cleaner, cancellable, and session-based rather than saving download records to the db (which caused a couple of edge-case annoyances for certain clients). I think it will need a bit more work, but it is much saner than it was previously

- if you do not have the manage tags dialog set to add parents when you add tags, the autocomplete input will no longer expand parents in its results list

- fixed an issue displaying the 'select a downloader' list when two GUGs have the same name

- hitting apply on the manage parsers or url classes dialogs will now automatically do a 'try to link' action as under manage url class links

- fixed (I think!) how the server services start, which was broken for some users in 361. furthermore, errors during initial service creation will now cancel the boot with a nice message, and the 'running … ctrl+c' message will appear strictly after the services have started ok the first time, and services will shut down completely before the db is asked to stop

- improved how the program recognises shutdowns right after boot errors, which should speed up clean shutdowns after certain bad server starts

- the server will use an existing server.crt and server.key pair if they exist on db creation, and complain nicely if only one is present

- the 'ensure file out of the similar files system' file maintenance job result will now automatically remove from/dissolve the file's duplicate group, if any, and clear out outstanding potential pairs

- a system language path translation error that was occuring in some unusual filesystems when checking for free disk space before big jobs is now handled better

- like repository processing, there is now a 1 hour hard limit on any individual import folder run

- fixed an issue where if a gallery url fetch produced faulty urls, it could sometimes invalidate the whole page with an error rather than just the bad file url items

- subscriptions will now stop a gallery-page-results-urls-add action early if that one page produces 100 previously seen before urls in a row. this _should_ fix the issue users were seeing with pixiv artist subs resyncing with much older urls that had previously been compacted out of the sub's cache

- until we can get better asynch ui feedback for admin-level repository commands (lke fetching/setting account types), they now override bandwidth rules and only try the connection once for quicker responses

- misc code cleanup

next week

I was unable to get to the jobs I wanted to this week, so I think I'll go for a repeat: updating the system:hash and system:similar_to predicates to take multiple files and extending the Client API to do cookie import for easier login. And I'll play around with some audio stuff.

R: 16 / I: 2 / P: 3 [R] [G] [-]

Release Tomorrow!

I had an ok week. I did some of the last duplicates work, including matching zooms for files with different resolution ratios and adding a neat pixel-for-pixel duplicate detector, polished some of the logic for the new tag autocomplete search, and cleaned and fixed a bunch of code.

The release should be as normal tomorrow.

R: 22 / I: 2 / P: 4 [R] [G] [-]

Version 361

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v361/Hydrus.Network.361.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v361/Hydrus.Network.361.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v361/Hydrus.Network.361.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v361/Hydrus.Network.361.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v361.tar.gz

I had an ok week. Some final duplicates work is done, and there is some polishing to the new tag autocomplete.

duplicates

The duplicates filter can now detect if static images with the same resolution are pixel-for-pixel duplicates! If they are, it gives one of the standard 'comparison' statements on the right hover window. Furthermore, if one file is a png and the other not, this statement will colour green/red and bias heavily to the non-png, since the png is likely a bloated 'clipboard' duplicate that you don't want. Pixel summary data is not cached long-term, so this routine takes a bit of extra CPU. It only kicks in if both files are images with the same res, but nonetheless please let me know if this makes the duplicate filter too laggy for you. I expect the new 'pixel hash' data will be cached at the db level in future to auto-resolve png/not-png dupes like this.

Also, the duplicate filter will now match the two files' zooms even if their resolution ratio differs! Zoom is now locked so the two files' widths are matched, along with the files' top-left corners. Two files with resolution 1920x1080 and 1912x1080 will now line up pretty good even if you zoom and pan.

And I have written a new system predicate for whether a file is the best file of its duplicate group (also called the 'king'). It provides an easy way to find only the best files, or only those that are not the best. It is bundled with the old 'num duplicate relationships' system predicate under the new 'system:file relationships' predicate.

I also fixed an issue with the 'custom action' button that wasn't letting custom actions go through unless some file was deleted–the final deletion question dialog now has a 'delete neither' choice, which is the default.

And if you want to feel some despair, Mr Bones now reports potential, duplicate, and alternate counts.

The duplicates storage overhaul is pretty much done. There is plenty more I could do, but I have now finished the main db focus of the work. Beyond some final UI stuff, there is only some decent new help to write. I would like to have that done next week, so I can draw a line under this job. It was more work than I expected, but I am overall really happy with it.

tag autocomplete

Last week's autocomplete changes seem to have overall gone pretty well. However, being able to search so fast has revealed some old 'limiters' I had in place to stop certain super laggy searches going ahead. And some old wildcard logic was flawed. Now we have more power with this control, I have been able to clean it up a bit.

First off, entering something like 'character:*' should now work everywhere (although it will likely lag a whole bunch once the final results come in). Also, searches with an explicit namespace, like 'character:ara', will now match 'character:samus aran', just as the simpler 'ara' does. Wildcard searches, like 'char*:sam' or 'c*r:*mus' should be a bit more sensible overall, finding more possible results and matching complicated queries more reliably.

And 'media' autocomplete fetches, which happens on a search page when you start typing with thumbnails already loaded, should also be much faster now (they were lagging last week with high numbers of thumbs). I improved the media search efficiency and added similar 'cancel' tech as I did to the db search last week, so it should now be fairly smooth and fast, even up to 10,000 files in view.

the rest

Just a note: system:size is now system:filesize.

Also, if you are a Linux user, or I have otherwise previously suggested you turn on the options->media->BUGFIX: Load images with PIL option, please check that option again this week and try turning it off. There was a time when PIL was more reliable than OpenCV–the other image library I use–but now things seem to be the other way around, and OpenCV is significantly faster too.

A user reported to me recently that he had an external hard drive he had hydrus installed to die due to overheating. It looks like it was related to heavy hydrus import folder work. This is the first time I have heard of something like this, but it still concerns me a lot. If you are running from a drive that can get similarly very hot, I strongly recommend you ensure you are not running any very heavy, hour-long-plus import or repository processing jobs on it. In the meantime, I will write some pause/throttle options for all the big routines to help users reduce load according to their situations.

full list

- duplicates:

- the duplicate filter now compares the pixel content of static image pairs of the same resolution–if they have the exact same pixels, a comparison statement is added, and if one file is a png and the other not (i.e. the png is likely a useless clipboard copy), the statement notes this and a strong duplicate score is applied

- added 'system:is/is not best file of its group' to search for file kings

- renamed 'system:num duplicate relationships' to 'system:num file relationships'

- wrapped the two file relationship system predicates into one 'system:file relationships' stub predicate that opens to a dialog with two pred panels

- added a 'add potential pairs' command to the thumbnail right-click file relationships menu, which will force-queue files for the duplicates filter

- the duplicate filter now ensures the two medias' zoom is locked so they have the same width through a transition. furthermore, their current dragged top-left position is pinned in the same location. this ensures files that have slightly different resolution ratios (especially when they are just a couple of pixels off) still remain reasonably comparable when switching back and forth

- reworked and simplified how position/drag delta is handled in the media canvas to support the above

- fixed the 'custom action' button on the duplicate filter, which had no 'delete neither' choice and whose 'forget it' button cancelled the whole custom operation, making it impossible to custom action without deleting something. I have added a 'delete neither' green-text button to the front, as the default action

- mr bones now reports on your potential, duplicate, and alternates numbers

- .

- tag autocomplete:

- greatly sped up tag autocomplete search when fetching from a current media view (i.e. from thumbnails in the search page)–it had some CPU-inefficient testing/counting that mattered at high media/tag counts

- greatly improved cancelability of tag autocomplete search when pulling from a current media view–this was resulting in high lag when typing fast with multi-thousand results

- fixed the gui-level tag matching test to match namespaced search inputs with offset subtags (e.g. 'character:aran' now matches 'character:samus aran'), both for wildcard and specific namespaces

- when typing an explicit wildcard tag search that does not end in a *, you will now be presented with two wildcard options–one with the implicit * suffix, one without

- fixed 'write' tag autocomplete inputs (like in manage tags) being able to search for chunky 'namespace:*' explicit wildcard searches

- .

- the rest:

- fixed the ipfs nocopy path translation control saving rows for client file paths outside of the main install path for non-Windows, where it was forgetting on save

- renamed 'system:size' to 'system:filesize'

- sped up some system:inbox searches

- disabled a PIL 'load truncated images' backup mode, which on the current version can seemingly lead to infinite load hangs

- file report mode now prints info when it deletes/recycles a path, including stack traces

- fixed a long-running and silent 'port already running' bug related to setting services on the server that was stopping successful service-set-restart from the client in many situations. 'port is already running' checks that conflict with other processes will now give an immediate error to the client without saving any changes

- the server now prints to the log as it stops/starts/has started its services

- improved how the server can report certain 500 errors

- the 'critical service tag/file reference' repository processing error has been improved: rather than reset the whole repository, it now pauses the repo and resets processing status for just the repo's 'definition' update files (without deleting any existing entries, so they should ultimately reprocess super fast) and also schedules a complete integrity and metadata check for all updated files

- keyboard interrupts from the console should now trigger a clean exit request for the client

- polite and forced shutdown requests when logging off should now trigger a fast exit (i.e. no yes/no dialog, no shutdown maintenance, but otherwise session saved and so on) for the client. this fast exit is noted in the log

- moved the tag and rating service listctrls in duplicate merge options panel to the new listctrl object

- moved the manage regex favourites listctrl to the new object

- updated a bunch of yes/no dialogs to the new panel system

- deleted some old unused dialog code and related unit tests

- fixed up deletion-and-reimport file location handling for lingering media objects, which were not correctly forgetting combined local file deletion record on the reimport

- improved shutdown error handling during repo processing

- deleted the mishimmie default downloader

next week

The duplicates help, is the top thing. I need to draw some diagrams, take some new screenshots, and brush up the existing text to better explain the new system. After that, I will catch up on small jobs. I'd love to have multiple system:hash search added (for searching a bunch of md5s, say), maybe some subscription thumbnail publishing cleanup, and perhaps some Client API work, where I'd like to have web browser cookie import for easy login.

Once the duplicate work is done, I expect to do a little work on audio support, likely basic 'has audio' metadata for files, and then crash on some client/tag-repository & PTR overhaul.

R: 1 / I: 1 / P: 4 [R] [G] [-]

Release Tomorrow!

I had a mixed week. The duplicates overhaul is done, with some final duplicate filter improvements, advanced thumbnail commands, and some updated duplicates help, and I fixed a heap of bugs.

The release should be as normal tomorrow.

R: 38 / I: 10 / P: 4 [R] [G] [-]

Version 360

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v360/Hydrus.Network.360.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v360/Hydrus.Network.360.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v360/Hydrus.Network.360.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v360/Hydrus.Network.360.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v360.tar.gz

I had a great week. There are a bunch of little things and an important speed overhaul to tag autocomplete.

tag autocomplete

I have never really liked the tag autocomplete workflow. It once blocked the UI completely, and some unusual timing options were needed to make it even useable, but it still often lagged out or just responded with gigantic lists in judders. After chipping away at the problem, this week I am finally updating it to what I really wanted.

So, the main change is that autocomplete results are now fetched as soon as you type. It responds very quickly and overall, I think, feels great, particularly once you have put four or five characters in. You get what you want as soon as you type, and can hit enter right away. The db 'job' to fetch results is now completely divorced from the UI and is able to cancel much faster when you type a new character, so the artificial fetch-start latency requirements of the old system are no longer needed.

There are now just two options for tag autocomplete, under options->speed and memory: whether tag autocomplete should fetch as you type (default on), and a character threshold to switch from 'exact match' autocomplete searches to 'full' ones. This 'exact match' defaults to 2 characters, and means if you type 'sa', you will only get results for 'sa' rather than 'sa*', which is generally a pretty giant laggy list, but if you type 'sam', you'll really be searching 'sam*' and get all matching autocomplete tags. You can reduce or increase this 'exact match' threshold as you prefer, or turn it off completely and get full autocomplete for any input.

Also, I have added some quality of life features. If results do not appear within 200ms, you'll now get a 'loading results…' label in the dropdown for feedback, and any 'static' results, such as exactly what you typed for a manage tags dialog input, or the special 'namespace:*anything*' for a search input, will appear immediately so you can select them without having to wait. Also, entering 'character:sa' will now trigger the same smaller 'exact matches' test as for the unnamespaced 'sa', rather than searching for the whole giant list of tags beginning 'character:sa*' as happened previously.

Furthermore, a query like 'char' no longer matcher 'character:samus aran'. This was a neat idea, but it proved too unwieldy IRL and is now better served by wildcard queries such as 'character:*' or 'char*:*'.

Overall, I am very pleased with the change. Please give it a go, and you'll see the difference immediately. However, tag autocomplete is a complicated system with different workflows, and I have made some big changes, so if you are, say, the sort of user who types fast and just hits enter, and you find the new results are a bit 'flickery' or something for you, let me know and I'll see if I can smooth it out.

more file maintenance

I brushed up the new file maintenance UI under database->maintenance->review scheduled file maintenance to have a nicer list and have added three new jobs:

One fixes file permissions on Linux and OS X. For a while, due to an oversight, file imports have been getting 600 permissions on Linux and OS X. I have fixed this now to be 644 (so, for instance, a nocopy ipfs instance or a network file share running on another user can access the files) for all new files, and this new file maintenance job will retroactively try to fix the permissions of existing files.

The other two are for the duplicates system. One explicitly regenerates similar files metadata, and the other checks if a file is improperly in or out of the similar files searching system, and fixes it if it is in the incorrect place (scheduling metadata regen as appropriate). Both are mostly for debug purposes but will get real use when I eventually add videos to the duplicates system, which will be a giant CPU job we'll want to spread out with the nice new non-blocking file maintenance pipeline. The old maintenance button on the duplicates page that kept eligible files 'up to date' is now gone as a result, and any outstanding jobs there (although most users shouldn't have any by now) should be migrated to the new file maintenance system on update.

full list

- tag autocomplete:

- after various tag autocomplete async work, fetch timings get a complete overhaul this week. the intention is for a/c jobs to appear as fast as possible, with good ui feedback, without interrupting ui while they work. feedback on how this works IRL would be appreciated

- there are now just two autocomplete options under options->speed and memory:

- - whether autocomplete results are ever fetched automatically, defaults to true

- - the max number of characters in the input that will cause just exact results vs. full autocomplete results, defaults to 2, can be None

- namespaces are no longer searched from an unnamespaced query ('char' no longer matches 'character:samus aran'). this proved too slow for real use, and remains better available with explicit namespace searches such as 'character:' or 'char*:*'

- the 'exact results' character limit now also applies to subtags of namespace searches! so, entering 'character:a' will deliver the same short exact match results as just 'a'–no more gigantic lists when you put in a simple namespace

- improved tag results caching to deal with the new non-namespace matching on subtag input

- tag autocomplete dropdowns will now display a non-selectable 'loading results…' label when results take more than 200ms to load.

- tag autocomplete dropdowns will now also display 'static' tags, such as 'namespace:*anything*' for 'read' inputs and the exact entered text and possible siblings/parents for 'write' inputs, during loading. so, typing 'character:' just to get the special 'character:*anything*' predicate is now simple and does not need a whole load wait to enter!

- cleaned up some tag listbox code to handle parent selection and navigation better along with the new label type

- greatly improved autocomplete search logic in the critical text search portion, collapsing it into one cleverer and more easily cancellable query rather than two or three simpler ones with potentially gigantic lists thrown back and forth

- improved speed of autocomplete cancel for certain large lists with many siblings

- .

- file maintenance:

- the new file maintenance ui now shows scheduled jobs in a listctrl, and only shows jobs that have outstanding work. you can clear/do work on multiple selected jobs

- the file manager should now try to guarantee at least 644 permission on file imports (previously, it was only trying to add 600, which lead to problems with nocopy ipfs running on another user etc…)

- added a file maintenance job to check and fix file permissions

- added a file maintenance job to regenerate similar files metadata

- added a file maintenance job to check if a file should be in the similar files system–if it should and isn't, it is queued to get its metadata data regenerated, and if it is and shouldn't be, it is removed

- the previous bulky similar files metadata regen job from the duplicates page is now removed, and any outstanding scheduled regen will be transferred to the new file maintenance manager on update

- .

- client api:

- added POST /manage_pages/focus_page, which makes the given page the current page in the main gui

- added help and unit tests for this new call

- client api is now version 9

- .

- the rest:

- fixed an issue recording media viewtimes when no max viewtime is set

- fixed the new missingdirectory errors not printing the missing path

- fixed an issue with some human-started repository actions waiting silently on bandwidth when it was not intended (e.g. account refresh)

- export folders now raise proper errors and pause themselves if their path is not set, does not exist on the file system, or is not a directory (previously, they silently stopped work without error)

- cleaned up some misc import folder code, and put in additional protections to the delete/move code to ensure folders cannot be so actioned if they somehow end up in the path import queue

- when unpinning a file or directory from ipfs, the clientside service now first checks that the current daemon considers it pinned (previously, this 500 errored when the object was not pinned due to a reinitialised daemon etc…)

- fixed an issue with the new ipfs path translation control, which was forgetting values when the clientside path was outside of the default db structure

- media objects that transition from trashed to physically deleted but remain in view will now correctly be aware of their complete previously-deleted status (rather than being simply remote, as they were before until a client restart)

- improved some of the recent duplicates db update code to pre-optimise the new tables on update (some users were getting slow behaviour due to mis-scheduled analysis maintenance)

- extended the new panel system to deal with custom button panels and moved the duplicate filter 'commit and continue?' dialog to the new panel system

- moved the archive/delete and duplicate filter 'commit and finish' dialog to the new panel system

- wrote a new question panel for the typical yes/no dialog used across the program and started a cleanup job to migrate all 140-odd instances of this over

- fixed an issue where a program instance that quit due to a user deciding to leave an already running instance in place would clear the original instance's 'running' file in its shutdown, meaning subsequent runs would charge ahead and hit 'database is locked' problems on db init!

- wrote a new 'similar files metadata generation report mode' to provide debug info on this cpu/gpu intensive routine

- added 'why use sqlite?' entry to the help faq, with a link to prkc's excellent document about the subject, https://gitgud.io/prkc/hydrus-why-sqlite/blob/master/README.md

- also added prkc's excellent Linux package requirements information to the 'running from source' help page

- fixed some old py 2.7 references in running from source help and an old link in ipfs help

- moved the 'file viewing statistics' menu down on the database menu

- fixed some dialog Escape key event handling

- fixed some ui ancestory testing code

- improved some misc similar files system code

next week

Next week is a cleanup week. I'd like to just do some boring rewrites and ui code updates, and I'll see if I can hack away at the last duplicates overhaul jobs.

R: 16 / I: 5 / P: 4 [R] [G] [-]

How do you tag? Metholodogy discussion thread

To quit spamming the Q&A thread (sorry) I agree this should be made into its own thread.

I'd like this to be a place to ask questions on "how should I tag this" for those that don't use the PTR and think you may be able to be more intelligent with how you tag. Also a place to discuss how you tag things and, most important, WHY that way. For example, what sort of issue were you looking to solve or what did it make easier by doing it that way?

T

o start off the discussion - I'm going to write a small essay on my thoughts on tags/tagging. It is my belief that tagging should be, above all else, two things

1) Simple - There are few scenarios where you must think if something should be tagged or not. Edge cases and oddities should be minimized at every opportunity. The number of tags should be kept small enough to maintain the entire list in memory. This is important for consistency. Speaking of which, the second item is…

2) Consistent - Tags should be well defined and used consistently. If a tag is often forgotten to be applied then it is a bad tag and should be either removed from the system or renamed/redefined as to be applied more consistently. In order to keep consistent tags it is also a requirement that all tags are able to be kept in memory to be applied properly. If you have so many tags that you are incapable of remembering them all then you will likely be tagging inconsistently. It is okay to add new tags to cover areas.

A tag that is used infrequently because the number of items in the gallery are too few is acceptable but is discouraged if the term is rarely searched for (either as an inclusive or exclusive term). Remember: tags are meant to help you search for images. This is important when it comes to naming tags.

I use a collection of 288 unnamespaced tags and 11 namespaces. The tags are mostly descriptors of clothes, physical traits, and objects. Namespaces are used for more meta information like character: artist: series: or whether a piece of art is SFW or pornographic.

Some of the more heavily opinionated choices I've made in regards to my tags include:

The tag "dog" is for actual animals and anthropomorphic characters are tagged using a namespace of "species:". There are no "doggirl", "catgirl", "mousegirl" tags, but "species:dog", "species:cat" instead. This rule was made because of Kemono Friends, Strike Witches. Because the rull is for all anthropomorphic characters, I end up with humorous scenarios like Upotte! and Girls Frontline using species:gun and Azur Lane using species:submarine, species:cruiser, etc. Along these lines, "cat ears" is not a tag that is used. "animal_ears" is the tag for all sort of animal ears and to specify which kind of animal ears, include a species: tag in the search.

The series:original does not exist. Any artwork that lacks a series: tag is assumed to be an original work, otherwise it would have a series: tag saying which series it is from.

Character names never include the series in parenthesis. This is what the series tag is for. To search for a specific character who shares the name of a character from another series - you include the series: tag in the search.

The series is always the name of the first season - even if a character comes from season two which has a different name. This is primarily because of Fate/Grand Order but also series like THE iDOLM@STER. This was done for simplification purposes - as remembering which season/spinoff of a show a character is from leads to mistakes and inconsistent tagging.

Underscores are always used for spaces. This is done so that wildcard searches function properly.

My namespace for SFW/NSFW is "Lewdness" and has 3 levels: SFW, NSFW, Pornographic. NSFW is defined as any visible nudity so very lewd art is still "SFW" as long as it does not contain nudity.

The cutoffs between "white", "silver", and "grey" are defined with a hexadecimal color code and the darkest color of the hair is sampled to determine the cutoff.

R: 0 / I: 0 / P: 4 [R] [G] [-]

Release Tomorrow!

I had a great week. As well as fixing bugs and catching up on small jobs, I put some time into the new file maintenance manager and significantly reworked and improved the search timings of the tag autocomplete system. Tag results now come in as fast as they can–it feels completely different.

The release should be as normal tomorrow.

R: 12 / I: 0 / P: 4 [R] [G] [-]

Version 359

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v359/Hydrus.Network.359.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v359/Hydrus.Network.359.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v359/Hydrus.Network.359.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v359/Hydrus.Network.359.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v359.tar.gz

I had an excellent and very full week.

file maintenance

The new file maintenance system now has some nice UI under database->maintenance->review scheduled file maintenance. It has two pages. It is a little advanced, but if you have some hydrus experience, please feel free to have a play around with it.

The first page reviews what is currently scheduled to run. It shows job counts for each type and lets you manually start work on that job type or cancel them.

The second lets you schedule new jobs. It uses the standard file search interface, so you can queue up a thumbnail regen for all webms with a certain tag, say, or recheck file metadata on all pngs (just in case they were truly apngs). There is also an advanced button to run talks on repository update files.

Furthermore, the new file maintenance manager now handles file integrity checks. The old 'file integrity check' entries under the database menu, which would check all files in your system in one go, are gone, now handled on the nicer system that doesn't interrupt work if you force it and will otherwise chip away at any large job in smaller pieces in idle time. Missing files or files will incorrect data will now automatically have any known URLs exported to .txt files, and incorrect files will also be exported, all to an appropriate directory beneath your main db dir.

I would like to add saved file searches within the next month or so, at which point I will attach some recommended jobs here, such as detecting apngs and correcting some webms that were once detected as mkvs.

duplicates

The thumbnail right-click menu now lets you reset or undo file relationships! If a file has duplicates, you can remove it from that group, and if it has alternates, you can pull it from that. You can also completely 'dissolve' the groups and clear out false positive relationships. It gives you little yes/no explanatory dialogs for each action, if you aren't quite sure what you mean to do.

I expect to add commands to apply these sorts of operations to entire selections in the next few weeks.

The duplicates storage overhaul is coming to a close. I have a few more cleanup jobs like this, some filter interaction improvements, and some proper help to write. We are getting there!

ipfs nocopy

I have added an advanced and experimental 'nocopy' IPFS pin mode to the client. This permits IPFS to share files straight from your hydrus file store, without having to make a copy. If you are not familiar with this, I recommend you not try. I would like feedback on this from the users who are enthusiastic about it so I can iterate on it and make it easier for regular users in future.

Essentially, the IPFS review/manage services panels now have a bit of nicer UI to talk to the daemon. You can check that 'nocopy' is turned on and enable it under manage services. Unfortunately, nocopy will only work on any file that is beneath the parent of the main ipfs conf directory, which is typically your USERDIR, so if your hydrus client_files folder is above that, or on another drive entirely, you will need to remap the locations with some symlinks(!) before it will work. The manage services panel has some help on it, but please feel free to ask me if you need some more.

I tested this, and it appears to work, but I am not a big IPFS user, so I would appreciate anyone who is giving this a proper shake-down and letting me know what you think. I'd love to add some easier plug-n-play IPFS sharing ability to the client, but we'll need to get over some hurdles first.

I expect to write a new IPFS downloader or similar that will pull multihash data from public http gateways so users can get into this more without having to set up a daemon.

deviant art

Deviant Art seem to be rolling out a complete site redesign that breaks the existing login script and gallery & file page parsers. Much like a recent Pixiv rollout, not all users are being affected at the same time–it looks like users who remain logged in from a previous session are not seeing the update as quickly.

I have written completely new objects for DA for this update. There is a new login script and parsers, and now there is a new downloader just for tags. Unfortunately, I think you need to be logged in–or otherwise have some particular cookies established from previous activity–to access the new download searches, or else you will get 404. If you are logged in and get this 404, try resetting your login and then logging back in. Dragging file page URLs straight on to hydrus seems to work fine, it is just this gallery search step that needs a 'new' session set up before it works.

I am afraid I have also had some CloudFlare 500 server error results with the new search. I suspect DA hasn't completely rolled out their new tech yet or something weirder is happening. These gallery 500s can pause the downloader hydrus-side annoyingly, so I will write some ui next week to let you try again quicker. In any case, the new downloaders I am putting out today are an improvement on 'it doesn't work at all', but they may not be everything we need. If you are a keen DA user, please let me know how you get on.

the rest

I fixed the 'file lookup script' GET problem. I apologise for the inconvenience.

I am also rolling out an improvement to the shimmie parser this week that pulls source time and md5 hash.

full list

- ipfs nocopy:

- wrote a new panel to better show ipfs daemon status and added it to the review and manage ipfs service panels

- added nocopy config review and enable status and buttons to this new panel

- added an EXPERIMENTAL 'use nocopy' checkbox to the ipfs manage services panel

- added accompanying WEWLAD path translation ui to enable nocopy when your hydrus media storage paths are inaccessible to the ipfs daemon for nocopy purposes. a help button explains this more–it currently needs some symlinking, so non-advanced users should stay away

- if everything is set up, ipfs nocopy seems to work! I am not totally happy about the setup required here, so feedback from advanced ipfs-fluent users would be appreciated and we can iterate on this

- improved stability of ipfs daemon/version checking code

- .

- file maintenance:

- wrote some proper file maintenance ui under database->maintain->review scheduled file maintenance!

- for existing work, the new file maintenance ui shows how much work is scheduled for each job type and lets you cancel that work or run it manually

- for new work, the new file maintenance ui lets you queue up work of any type for files you select with the standard tag autocomplete search interface! you can schedule all pngs to be rescanned in case they are truly apngs, or regen thumbs for all files imported before a certain date, or whatever you wish. you can also queue up repository update files

- the file maintenance manager can now deal with repository update files when it does a complete file metadata regen

- the file maintenance manager now takes responsibility for checking file presence and file integrity. the old 'check file integrity' options under database->maintenance, which did all files in one go, are now gone

- file integrity checks will now always export broken files and missing/broken files' known urls to .txt files to your db_dir/missing_and_invalid_files. appropriate popups and log data will be sent as well. also, the known urls will be both exported on a per-file .txt basis and appended to one unified .txt

- if a file now fails to parse on a metadata reparse, it is now automatically checked for file data integrity

- if a repository encounters a missing, invalid, or incorrect filetype update during update processing, it now schedules all updates in the repo to be appropriately rescanned by the file maintenance manager

- if the storage subdirectory directory does not exist on a client file path request or thumbnail-add attempt, a special error will now be raised with instructions to reconnect the location or shut the client down immediately

- cleaned up some ffmpeg mime-detection logspam

- .

- duplicates:

- added several single-file thumbnail right-click dissolve/reset duplicate actions:

- - reset search status

- - remove from duplicate group (if in one and not the king)

- - dissolve duplicate group (if in a group)

- - remove from alternate group (if in one)

- - dissolve alternate group (if in one)

- - clear false-positive relations (if it has some)

- added some new code to deal with dissolution and member extraction at the db level

- when a member is extracted from alternate group, its constituent files are now requeued for potential search

- multi-selection duplicate right-click actions are now available to non-advanced-mode users

- wrote some unit tests for the new dissolve/reset actions

- cleaned up some misc duplicates code

- .

- the rest:

- fixed a recent bug in the file lookup script GET call–I apologise for the mistake

- the main gui page tab menu now lets you sort page tabs by the number of files they have

- deviant art seem to be rolling out a new page format. this week hydrus introduces completely new deviant art downloader objects that, fingers crossed, will update any existing users smoothly and also provide new tag search functionality. users who are still logged in may still be getting the old page format. if this is you, and this update does not work (although I _think_ it should, even so), please try clearing your existing login and logging in again

- new deviant art login script, artist + tag GUGs, gallery url classes, file and gallery parsers

- updated the shimmie file page parser to pull source time and md5

- improved the 'process now' advanced button to only focus on actual specific outstanding processing. previously, it was also checking for new metadata when due, which, when the server was not available, could seemingly idle for a time before actually processing updates due to the new delaying connection retry code

- wrote a new 'file import report mode' mode to help->debug->report modes

- fixed a progress display issue with the janitorial petitions processing page

- improved accuracy of sibling and parent petition counts, and properly capped them at 1000

- mapping petitions are now grouped by namespace, and will come in more manageable chunks

- fixed the server launch-and-init test debug code

- misc string-to-string control improvements to support the new ipfs edit ui

- removed the old 'continual tag archive sync' legacy code from tag services, which has been semi/non-functional for a long time

- cleaned up the annoying separator hanging on the end of certain tag right-click menus

- cleared out the 'Exception ignored in' spam that is often printed after the log closes

next week

I pushed it a bit hard this week, so I will take it easier as I catch up on a variety of smaller jobs. I also need to catch up on my messages.

R: 15 / I: 3 / P: 4 [R] [G] [-]

Version 358

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v358/Hydrus.Network.358.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v358/Hydrus.Network.358.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v358/Hydrus.Network.358.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v358/Hydrus.Network.358.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v358.tar.gz

I had a great week doing duplicates work and fixing bugs.

duplicates

I split this big duplicates storage overhaul into three jobs, and this week marks the third and final job done. Like alternates and duplicates information, potential pairs are now stored in a unified and more efficient way.

On the front end, you may notice your potential pairs queue shorten again on week. It will also shrink faster as you process in the duplicate filter, which will present more 'useful' duplicate pairs first and apply your decisions more intelligently at the db level.

All the code is simpler except for one key area. If you notice certain the 'show some random potentials', duplicate filter or potential counts load time take way too long, please let me know about your situation. It is possible I will have to revisit this complicated 'join', although in my tests it is performing well.

Also, I have written the new record to stop alternate pairs coming up in the duplicate filter repeatedly, as some users have experienced. These relationships are now more concrete, and this concreteness is plugged into duplicate merge operations and so on. You may see one more round of alternates appearing, and then they will be saved properly.

Now everything is stored on the new system, there are two main jobs remaining: re-adding various administrative commands like remove/dissolve to properly undo relationships, and adding some options and improvements to the duplicate filter workflow.

the rest

Pixiv changed their format recently, so hydrus's default parser broke. This should be automatically fixed this week. Thanks to a user who sent in this fix.

The issue where mouse scroll events were not being caught when a media viewer did not have focus is also fixed.

The 'watcher' page now reports file and check status in the 'status' column! I missed this somehow when I added it for the gallery downloader. This makes it just a little easier to see what a list of threads is currently doing.

I may have fixed the problem where exiting manage tags from a media viewer sometimes falls focus back to the main gui. Please let me know if you still get this (and if so, if you know a way you can reliably repeat this behaviour).

I improved some of the network engine's 'this connection ended early' checks. This may have fixed some issues users had downloading images and page data from some unreliable servers, but if it does not, please send me any incomplete jpegs and the URLs they came from so I can check further on my end. Also, the whole system is more strict about response lengths now, so if you discover false-positive network failures here, please report them.

Also some server issues related to last week's client api authentication improvements (such as file repository file upload sometimes breaking) should be fixed.

new client api library

If you would like to work on the Client API using Node.js, check out the new module a user wrote here:

https://github.com/cravxx/hydrus.js

This is now in the help along with the rest of the API here:

https://hydrusnetwork.github.io/hydrus/help/client_api.html

full list

- duplicates:

- the final large data storage overhaul work of the duplicates work big job is done–potential duplicate information is now stored more sensibly and efficiently. potential pair information is now stored between duplicate file groups, rather than files themselves. when duplicate file groups are merged, or alternate or false positive relationships set, potentials are merged and culled appropriately

- your existing potential data will be updated. the current potential pairs queue size will shrink as duplicate potential relationships are merged

- the duplicate filter now presents file kings as comparison files when possible, increasing pair difference and decision value

- potential pair information is now stored with the 'distance' between the two files as found by the similar-files search system. the duplicate filter will serve files with closer distance first, which increases decision value by front-loading likely duplicates instead of alts. distance values for existing potential pair info is estimated on update, so if you have done search distance 2 or greater and would like to fill in this data accurately to get closer potentials first, you might like to reset your potential duplicates under the cog icon (bear in mind this reset will schedule a decent whack of CPU for your idle maintenance time)

- setting alternate relationship on a pair is now fixed more concretely, ensuring that in various search expansions or resets that the same pair will not come up again. this solves some related problems users have had trying to 'fix' larger alternate groups in place–you may see your alternates compared one last time, but that should be the final go. these fixed relationships are merged as intra-alternate group members merge due to duplicate-setting events

- a variety of potential duplicates code has been streamlined based on the new duplicate group relationship

- improved how a second-best king representative of a group is selected in various file relationship fetching jobs when the true king is not permitted by search domain

- one critical part of the new potential duplicates system is more complicated. if you experience much slower searches or count retrievals IRL, please let me know your details

- expanded duplicates unit tests to test potential counts for all tested situations

- fixed a bug where alternate group merging would not cull now-invalid false-positive potential pairs

- the rest:

- updated the default pixiv parser to work with their new format–thank you to a user for providing this fix

- fixed the issue where mouse scroll events were not being processed by the main viewer canvas when it did not have focus

- file page parsers that produce multiple urls through subsidiary page parsers now correctly pass down associated urls and tags to their child file import items

- updated to wx 4.0.6 on all built platforms–looks like a bunch of bug fixes, so fingers-crossed this improves some stability and jank

- updated the recent server access-key-arg-parsing routine to check access from the header before parsing args, which fixes an issue with testing decompression bomb permission on file POST requests on the file repository. generally improved code here to deal more gracefully with failures

- the repositories now max out at 1000 count when fetching pending petition counts (speeding up access when there are large queues)

- the repositories now fetch petitions much faster when there are large queues

- frames and dialogs will be slightly more aggressive about ensuring their parents now get focus back when they are closed (rather than the top level main gui, which sometimes happens due to window manager weirdness)

- rewrote a bad old legacy method of refocusing the manage tags panel that kicks in when the 'open manage tags' action is processed by the media viewer canvas but the panel is already open

- hitting 'refresh account' on a paused service now gives a better immediate message rather than failing after delay on a confusing 'bad login' error

- improved login errors' text to specify the exact problem raised by the login manager

- fixed a problem in the duplicates page when a status update is called before the initial db status fetch is complete

- the manage tag siblings panel now detects if the pair you wish to add connects to a loop already in the database (which is a rare but possible case). previously it would hang indefinitely! it now cancels the add, communicates the tags in the loop, and recommends you break it manually

- added a link to https://github.com/cravxx/hydrus.js , a node.js module that plugs into the client api, to the help

- a variety of user-started network jobs such as refreshing account and testing a server connection under manage services now only attempt connection once (to fail faster as the user waits)

- the 'test address' job under manage services is now asynchronous and will not hang the ui while it waits for a response

- fixed some unstable thread-to-wx code under the 'test access key' job under manage services

- improved some file handling to ensure open files are closed more promptly in certain circumstances

- fixed some unstable thread-to-wx communication in the ipfs review services panel

- improved the accuracy of the network engine's 'incomplete download' test and bandwidth reporting to work with exact byte counts when available, regardless of content encoding. downloads that provide too few bytes in ways that were previously not caught will be reattempted according to the normal connection reattempt rules. these network fixes may solve some broken jpegs and json some users have seen from unreliable servers

- fixed watcher entries in the watcher page list not reporting their file and check download status as they work (as the gallery downloader does)

- the client api will now deliver cleaner 400 errors when a given url argument is empty or otherwise fails to normalise (previously it was giving 500s)

- misc cleanup

next week

I had hoped to do some IPFS work this week, but I ran out of time to do it properly. This is now the main job for next week. Otherwise, I will do some of this final duplicates work and some misc small jobs.

R: 3 / I: 0 / P: 4 [R] [G] [-]

Release Tomorrow!

I had an excellent and very full week. I have extended the new file maintenance system and written some UI to manage it, added some undo/reset functions for the new duplicate system, fixed a variety of bugs, and figured out an advanced-users-only extention to the IPFS service to permit the experimental 'nocopy' pin mode,

The release may be late tomorrow, as there is more than usual to test.

R: 19 / I: 4 / P: 4 [R] [G] [-]

Hydrus Scripts

ITT post/discuss/jerk off to/improve scripts to use with hydrus

One of my scripts was broken as shit and I just fixed, but I figured it was better to graduate from the Q&A thread and make it easier to find.

R: 14 / I: 0 / P: 4 [R] [G] [-]

Version 357

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v357/Hydrus.Network.357.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v357/Hydrus.Network.357.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v357/Hydrus.Network.357.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v357/Hydrus.Network.357.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v357.tar.gz

I had a great week fixing bugs and cleaning up other small jobs.

shortcut and hover window fixes

The recent menubar open/close detection did not work well on OS X and some versions of Linux. It misfiring broke some shortcut processing and hover window display. The good news is that as I went into that code more closely this week, I discovered several other menu, shortcut, and hover window problems that were causing long-term issues. All clients should have less janky hover window placement and behaviour, but particularly on non-Windows.

Specifically, the OS X shortcuts-in-media-viewer issue is fixed. Also, on all platforms, hover windows should size and place better, with less flicker, and handle mouse and keyboard interaction more sensibly. A small memory-leak that was affecting OS X and some Linux versions is also cleared up, which is likely to improve some increasing menu-lag and other jank. I know there is more work to do here for certain Linux window managers, so continued feedback would be appreciated here.

Semi-relatedly, the advanced 'custom' shortcut sets are now hidden from non-advanced-mode users under file->shortcuts.

the rest

The popup toaster (on the bottom-right, where messages appear) now has a collapse/expand button! So, if your subscriptions suddenly appear and get in the way of your browsing, you can now finally hide them. As a result, the little 'dismiss all' summary bar now always appears when you have some messages up.

The duplicate filter's new jpeg quality comparison now has customisable 'weight' scores under options->duplicates. Also, it seems I messed up the default values here, swapping high vs very high values. I have fixed the defaults and set them to reset this week so all users get decent values. If you prefer these values different, please check them again after update.

Several places across the program that have 'write' tag autocomplete inputs, such as manage tag siblings and parents, now have paste buttons beside the tag input box. They let you enter tags straight from the clipboard, including the standard newline-separated format for entering multiple tags at once. It is now much easier to set up twenty tags to have a certain parent.

Downloaders now report their 'added' time without seconds, so '12 minutes' instead of '12 minutes and 23 seconds'. The continue text updates here were making large fresh download lists pretty flickery, so this should calm things down a bit.

When bandwidth is blocked, rather than just saying 'yes', the network->review bandwidth usage window now reports how long until bandwidth is free!

full list

- client api:

- the client api can now receive the access key through a GET or POST parameter rather than the header

- the client api now supports GET /session_key, which provides a temporary key that gives the same access as its permanent access key with the Hydrus-Client-API-Session-Key name through header or GET/POST param. it expires after 24 hours of inactivity or if the client is restarted

- the GET /manage_pages/get_pages call now returns the unique 'page_key' identifier that will be useful in future page management when multiple pages share a name

- the POST /add_urls/add_url command now takes 'destination_page_key' to exactly specify which page you would like a URL to end up on. if the page is not found, or it is the incorrect type, the standard page selection/creation rules will apply

- cleaned up some serverside request processing code

- cleaned up some misc client api permission checking code

- updated client unit tests to check the new changes

- updated client api help to reflect the new changes

- cleaned up some GET and POST parameter parsing

- client api version is now 8

- .

- shortcut and hover window fixes:

- moved the canvas shortcut processing code more towards the new shortcut system

- the OS X shortcut-in-media-viewer issue, which was being boshed in a similar way to the main gui last week, should now be fixed

- when the hover windows have focus, they now pass shortcuts up to the canvas parent more reliably

- removed a legacy menu highlight-tracking system that was malfunctioning and generally throwing a slow-memory-leaking wrench in several places, particularly some non-Windows situations

- the 'menubar is open' test code is now only active for Windows. the other platforms have mixed reliability with menubar open/close events

- some related OS X hover-window flickering and hiding-under the main page problems (having problems due to thinking menus were open) are also fixed

- some hover window flicker on certain focus changes due to clicking focus windows should be fixed

- hover windows now try to size themselves a little better on init, which reduces some initial flicker or false-positive single-frame display on some systems

- extended the hover report mode to report some 'ideal' pos/size info as well

- under file->shortcuts, custom shortcuts are now hidden for non-advanced-mode users

- .

- the rest:

- the popup message toaster now always shows its 'dismiss all' summary bar whenever any messages are being displayed. the summary bar now also has a ▼/▲ button to collapse/expand its messages!

- added duplicate comparison score options (under options->duplicates) for the new jpeg quality estimator

- fixed the default duplicate comparison score values, which appeared to be reversed for higher vs much higher–they will be reset to these new defaults on update, so recheck them if you prefer different

- in manage tag siblings and parents, the filename tagging dialog, and some misc options panels, tag autocomplete input controls now have 'paste' buttons to make entering many results much easier

- to reduce update flicker, the downloader and watcher pages do not list seconds in their 'added' column ('12 minutes 24 seconds ago' is now '12 minutes ago')

- improved clipboard access cleanup on in-clipboard errors, which was sometimes leading to error popups or clipboard lockup

- rather than the simple 'yes', the review bandwidth usage dialog now puts the waiting estimate (like '12 minutes 50 seconds') in the 'blocked?' column

- improved external program launch code for non-Windows to remove hydrus-specific LD_LIBRARY_PATH completely when no OS default exists to restore it to. this should fix ffmpeg connection for certain installs

- fixed a rare bug when initial media results of a page failed to load due to a subset of unexpectedly unfetchable file records

- gave the rare 'ui freezup on dialog close' event yet another pass. closing via escape key should now be immune to this

- 'remote' files that were once in the client but since deleted now have the 'trashed' icon and will state so on their right-click info summary lines

- fixed various instances where selection-from-list dialogs were failing to sort their list based on underlying data object uncomparibility. an example of this was when selecting which queries to pull from a separating subscription

- on the edit parser panels, fetching test data successfully via the quick button or the manual URL entry will now set that URL in the example parsing context

- on the edit parser panels, the subsidiary page parser's separation formula now launches with the correct example data (the original data from the parent dialog, rather than the post-separated data) on which to test separation. this should nest correctly for multiple subsidiary page parsers

- to reduce server load spikes, clientside petition processing now approves very large mapping and file petitions (such as a petition to delete one tag from 50k files) as a sequence of smaller chunks

next week

I want to do two things: moving 'potential' duplicate information to a more efficient data structure and improving some IPFS connection tech, particularly seeing if we can figure out 'nocopy' so IPFS can share without having to copy files to its own data store. These will both be fairly big jobs, so we'll see how it goes.

R: 0 / I: 0 / P: 4 [R] [G] [-]

Release Tomorrow!

I had a great week working on the duplicates overhaul and fixing bugs. Potential duplicate pairs are not stored in a sensible way, with the duplicate filter now serving up more useful comparisons and clearing out the pending potential queue faster. Also, the issue with the media viewer not capturing mouse scrolls while not focused is fixed, as is the default pixiv parser, file repository file upload, and several more specific issues.

The release should be as normal tomorrow.

R: 24 / I: 4 / P: 4 [R] [G] [-]

Version 356

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v356.tar.gz

I had an excellent and busy week catching up after E3. Duplicate better/worse/same relationships are now stored more logically and efficiently, and a variety of bugs are fixed. Maintenance is also easier to control.

duplicates

The second phase of the big duplicates rewrite is now done! Rather than the old system of trying to store better/worse relationships for every possible member of a duplicate group, which is very complicated, requires many human decisions, and is not very useful in practise, this new system instead lumps all duplicate files into a single group with a single 'king' representing the best file of that group.

The groups are transitive, so putting a file into a group with a better/worse/same command–or even merging a group into another–is now just a single action. Rather than comparing a file with every member of a group, it is now just merged as a simple member or the new king. The overall workflow of dealing with large duplicate groups is now much much faster. Furthermore, alternate relationships are stored between groups of duplicates, not individual files, so again the number of choices you need to make is automatically reduced as you process your normal duplicate filter queue.

Searching and managing duplicate file groups is also easier. When searching for system:num duplicate relationships, you simply search for how many 'duplicates' a file has, rather than how many better/worse/same. Right-clicking on a thumbnail to open up the 'file relationships' menu–which is now available to all users–now presents the same simple 'duplicates' count-and-show-in-a-new-page entry and will say whether the file is the best file of its group. If the file is not the best of its group, you can view that king or set the current one as king with one click.

Future updates will show king/not king status in the thumbnail UI. I will also write a new system search predicate to filter for king/not king.

Your existing better/worse/same data will be converted to the new system on update, with appropriate kings chosen. If in recent weeks you noticed some files that seemed to be both alternates and duplicates, this will now auto-resolve as the duplicates are merged together.

The new jpeg quality estimator, which will sometimes present 'high quality vs low quality'-style statements in the duplicate filter, now appears for all users. It isn't perfect, but it works ok.

The last phase of the rewrite will be to update potential pairs. It will reduce the potential pair queue count even further and improve the pair-elimination efficiency of the decisions presented in the filter (essentially by forcing king vs king comparisons). I want to take a week to do some smaller duplicates work (like commands to undo/dissolve existing relationships intelligently), so with luck I expect to have that done for 358. I am overall very pleased with the work here so far, although there is still a good amount to do, and some proper new help to write up.

the rest

I fixed the popup spam that the new clipboard watcher was falsely making on many non-matching URLs. It was some test code that was misfiring–even though the URLs were not being added since they were not matched, a higher part of the system thought they were not being added due to a problem with pages. I am sorry for the confusion–it is now fixed.

Also the unusual OS X shortcuts issue, where shortcuts were not working on some OS X installs until a menu was opened! This was due to a weird OS X specific problem in the recent main gui menubar detection code. If you are on OS X and notice your menubar stops updating, please let me know.

I put a bit of time into the maintenance code this week. Most maintenance jobs, like repository processing and thumbnail regeneration, now use the same 'clock' to figure out if they should be working. They now know if they are running in idle time or shutdown time or forced by the user, and will cancel out faster depending on the situation as needed. The other upshot of this is that the file menu now has a 'exit and force shutdown maintenance' option, and the shutdown splash finally has a 'stop shutdown maintenance' button! I regret that this took so long to come in–but now it is, and it works well!

You can now also force file maintenance from database->maintenance->file maintenance. It tells you if no work needs to be done.

full list

- duplicates:

- moved better/worse/same quality duplicates relationships to the new 'king' group-based model. rather than tracking every relationship, duplicates are now stored in groups with a single 'best' file

- as a result, duplicate relationships are now transitive! saying that one king is duplicate to another will merge groups. the 'better' king is the new king, and 'same quality' kings choose one of the kings pseudorandomly. advanced exceptions: saying that a king is better than a basic member of another group or saying that two members are same quality is still valid but will simply 'poach' the non-king member from the other group in order to ensure the wrong king doesn't end up on top in the eventual merge. saying KingA is same quality as MemberB will merge the groups with KingB as the new king (since it is presumably same/better quality to all members of A)

- the thumbnail right-click 'duplicates' entry is now renamed to 'file relationships' and is no longer advanced mode only. the 'find similar files' entry is folded into this

- the thumbnail 'file relationships' menu now shows a simple 'duplicates' count rather than the old messy better/worse/equal. it will show all the members of a duplicates group when clicked. the menu also notes if the focused file is the best file of its group. if it is not, you will get the option to show the best file or make the focused file the best

- as a result, it is now much simpler to view a group of duplicates and overrule a 'best quality' member as needed

- added the 'media' shortcut 'duplicate_media_set_focused_king' to shortcut setting a 'best quality' file

- the system:num duplicate relationships now has the simpler 'duplicates' entry, to search on size of the entire group. searching for kings/not kings will come soon

- due to the new duplicate transitivity rules, potential pairs are now eliminated at a much faster rate!

- setting duplicate relationships will overrule false positive or alternate relationships already in place

- manually setting alternate relationships to more than two thumbnails at once will now set each file as alternate to every other file in the selection, completely eliminating potential pairs within the group. if you try to do this to large groups of files you will get a longer yes/no confirmation message just to make sure you aren't overwriting some potential dupes by accident

- all existing better/worse/same relationships will be converted to the new group storage in this update, with appropriate kings determined. potential pair queue counts will be reduce accordingly, and the temporary alternate/duplicate confusion from the alternates update will be auto-resolved by merging truly duplicate 'alternates' together

- fleshed out the duplicate test code significantly to handle the new dupe groups and their interactions with the recent false positive and alternates changes

- refactored some db test code into separate client/server/duplicates files and cleaned up dupe tests readability

- potential pairs are now the only component of the new system still on the old pairs system. the duplicate filter will still serve up some inefficient (i.e. non-king) comparisons

- .

- the rest:

- fixed the issue where many clipboard-watcher-caught URLs that did not match were producing false-positive 'could not generate new page for that URL' error popups

- the clipboard text-fetcher now tests against incompatible clipboard types (like a screenshot) better, and all instances of text fetching now report errors more gracefully and with more information

- fixed the unusual OS X issue where many shortcuts were not being processed after client boot until the top menubar was opened and closed. a variety of other blocking-while-menubar-is-open issues that were false-positive misfiring are now fixed as well, please let me know if you still have trouble here

- the file menu now has an 'exit and force shutdown maintenance' option to force-run outstanding maintenance jobs

- when shutdown maintenance work is going on, the shutdown splash screen now has a 'stop shutdown maintenance' button!

- cleaned up some file maintenance manager maintenance locking and shutdown cancel logic

- moved all the idle-mode maintenance checks to a new system that explicitly defines idle/shutdown/forced maintenance work and tests those states in a unified manner, checking idle mode and the new splash cancel button status and so on more reliably. a lot of maintenance should cancel out quicker when appropriate

- misc shutdown logic cleanup

- added a 'file maintenance' option to the database->maintenance menu that forces the new file maintenance manager to run its queue. it'll make a little popup as it works, or a note that no work is due

- the 'regenerate' thumbnail menu is also available to all users

- jpeg quality estimates are now available for all users in the duplicate filter. they only display when the two jpegs' quality have different labels

- the jpeg quality estimator now handles some unusual jpegs that load with empty quantization table arrays

- the duplicate filter now handles bad jpeg quality estimations gracefully

- cleaned up some ffmpeg communication code

- the ffmpeg debug text that spawns on a help->about call that fails to discover ffmpeg version information now prints stderr output as well. if you have been hit by this, please give it another go and let me know what you get

- the same ffmpeg 'no response' error on file parse now popups and prints some debug info and returns a better error

- dialogs and windows on the new panel system now support a new pre-close tidying system

- the manage tags dialog and window will now cancel any pending large tag autocomplete queries on close

- regular gui pages now support a new pre-close tidying system

- search pages will now cancel any pending search results loading or tag autocomplete queries on close

- improved reliability of the popup message manager chasing the main gui when it is sent to another screen by a keyboard shortcut (such as shift+win+arrow on Windows). it should work now if the mouse cursor is in either window. please let me know if this causes trouble for virtual display navigation

- the network engine now waits significantly longer–60s–on connection errors before trying again, and with every failed attempt will wait n times longer again. when in this waiting state, a manual user cancel command cancels it out faster

- I believe I have fixed/improved a situation where media viewer hover windows would sometimes disappear immediately after appearing on some Linux window managers

- improved hover window report mode to state more focus info in case the above is insufficient

- to better link the two requests and consume bandwidth under strict rules more precisely, the override bandwidth rule that kicks in when a file page has a single file is now 3 seconds instead of 30

- updated options->connection page to specify that 'socks4a'/'socks5h' is needed to force remote dns resolution

- sped up tag parents initialisation

- repositories now group tag sibling and parent petitions by the parent/better tag's namespace

- removed some old network 'death time' code that is no longer useful and was interfering with heavy petition processing

- the log now flushes itself to disk every 60s rather than 300s

- misc fixes and cleanup

next week

This week was a crush, so I focused on the duplicates work and bug fixes and cleanup. I have a variety of small 'new stuff' jobs to catch up on next week. Some more shortcuts to add, some client api work, some quality of life like pasting tags into siblings/parents. And some smaller auxiliary duplicates work.

I enjoyed taking a week off, but I am glad to be back to work.

R: 1 / I: 0 / P: 4 [R] [G] [-]

Release Tomorrow!

I had a great week fixing bugs and catching up on other small jobs. Some OS X and Linux UI issues (including shortcuts and bad hover windows in the media viewer) are fixed or improved, some downloader-page flicker is reduced, the client api is moved forward a little, and the popup system now has a neat collapse/expand button to hide subscriptions and so on while they work.

The release should be as normal tomorrow.

R: 36 / I: 17 / P: 4 [R] [G] [-]

Version 355

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v355/Hydrus.Network.355.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v355/Hydrus.Network.355.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v355/Hydrus.Network.355.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v355/Hydrus.Network.355.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v355.tar.gz

I had a good week catching up on small jobs.

duplicates

I added a 'trash file' button to the duplicate filter hover window, tucked up in a new row with the cog button so they fit on thinner windows. If both potential files are worth deleting, hitting this is an easy way to dismiss them immediately. I also fixed some over-aggressive potential pairs removal from last week's alternates update. Users with many alternates may have lost some valid intra-alternate-group potential pairs, which I expect to add back in once this whole overhaul is complete.

The duplicate action buttons (and several others in the media viewer) are now aware of the shortcuts that can do their actions, and they will display these shortcuts in their tooltips! I am quite pleased with how this turned out, and as I slowly work on unifying program shortcuts to the new system, I hope to have this in more places.

Users in advanced mode get a new jpeg quality estimator comparison on the duplicate filter panel–when two jpegs appear to have fairly different quality, you will get a new 'A vs B' statement about it. This estimator is a bit hacked together, not totally precise, so I would be interested in your overall feedback. I expect to revisit it and add some options before I enable it for all users. Advanced mode users also get duplicate merge options back for 'alternates'.

the rest

I wrote a simple clipboard watcher for URL imports! Check it under network->downloaders. You can watch for 'watchable' URLs like imageboard threads and other 'recognised' URLs like booru pages. It checks for new text every second, adds URLs to pages just like a drag-and-drop event (i.e. to currently selected page if it is valid, otherwise the left-most valid page, otherwise making a new page), and can handle newline-separated lists of URLs. It is off by default.

Note that the client isn't happy about making new pages while it is minimised (their layout messes up), so if you clipboard-watch a bunch of URLs and it needs to make a watcher page for them, I have it set now so they will queue up until it is restored. If you leave your client minimised and want to send it URLs, I recommend you have a page set up for clipboard-watched (or Client API-added) URLs to land on until I can improve this.

The file viewing statistics system gets a pass this week–preview and media views now have minimum view times before they are counted, and they also cap to maximum values (so leaving a preview view up won't accidentally add a heap of time like 1 day 3 hours to your total viewtime. It is also now off by default for the duplicates filter. Check the new settings under options->file viewing statistics. If you got hit by some bad numbers before due to some accidental fast or prolonged views, check the new command under database->file viewing statistics->cull file viewing…, which will attempt to compensate for any bad values based on the new min/max caps. A yes/no dialog explains what it does.

There are several new shortcuts to help navigate the manage tags dialog, all under the 'main_gui' shortcut set: Four are called 'show_and_focus_manage_tags_XXX' and let you move keyboard focus to the various 'suggested' tag columns (recent/favourites/file lookup/related). Combined with 'set_search_focus', default Ctrl+S, which jumps you back to the tag autocomplete input, you should now be able to enter suggested tags completely without the mouse. Furthermore the new 'focus_media_window' will move focus back to the media viewer canvas if the manage tags window is launched from one (say for archiving or rating by shortcut without having to close manage tags).

You can now quick-enter 'namespace:' in a search page's autocomplete tag input to get the special (and efficient) 'namespace:*anything*' search predicate.

I fixed the popup-error problems that could come up when fading 'collected' thumbs! I apologise for the inconvenience here and have added some weekly pre-build tests to catch simple problems with collected thumbs in future.

full list

- duplicates:

- retuned the new alternate and false positive setting code to be less aggressive about removing potential pairs. users with alternate groups who updated to 354 may have lost some intra-alternate potential pairs, which I expect to fill back in once the potential pairs system is updated

- the duplicate filter center-right hover window now has a trash button, which is moved to its own row with the cog icon where they will fit on thinner windows better

- users who are in advanced mode now have access to duplicate merge options for 'alternates' again–but be careful with this, it is advanced. if you had merge options for alternates set up before, your old options _should_ return, but make sure to check it

- if you are in advanced mode, the duplicate filter can now show an experimental jpeg quality comparison statement. if this works out, I will add custom scores and improve this otherwise based on feedback , so comments would be appreciated

- .

- client api:

- the client api has a new 'manage pages' permission!

- /manage_pages/get_pages now fetches a structure of the current page session! there's some help for it

- in future, I expect to add a 'select page' command and get/remove/highlight URL(s) for downloader pages, and reveal the unique page identifier for better targeting here and for add_url commands

- client api is now version 7

- .

- the rest:

- fixed a stupid issue related to drawing collection thumbnails that was causing a lot of errors. I apologise for the inconvenience and have added pre-build tests to catch these simple mistakes in future

- added a clipboard url watcher! there are two options now under network->downloaders–one for watcher urls, and one for all other _recognised_ urls (i.e. post and gallery urls, and file urls that have matching url classes). it checks every second, sends urls to pages according to the same rules as a drag and drop event, and will handle newline-separated lists of urls

- when the client is minimised and needs to create a new downloader page because of an 'add url' command through this new clipboard watcher or the client api, it will now queue up the url and its page creation for when the client is next restored (page creation while minimized causes layout bugs). it returns an appropriate result text to the client api in this case

- added four 'show_and_focus_manage_tags_XXX' shortcuts to the 'main_gui' shortcut set that let you select and focus the four possible suggested tags panels on the manage tags dialog. these are slightly special focus events that do some 'first tag selection' work as well and should let you make the whole process keyboard-only

- added a 'focus_media_viewer' shortcut to the 'main_gui' shortcut set that focuses the media viewer from a media viewer's manage tags window

- file viewing statistics gets an update: it now has its own options page where you can set min/max for preview and media viewing times. the global turn on/off control is moved from database menu to this page, as is a new control for enabling it on the duplicate filter (where you will be flicking back and forth and perhaps do not want lots of views recorded). duplicate filter file stats now default to disabled

- added a new command to database->file viewing statistics that will cull your current stats based on the new min/max values to compensate for previous accidental '3day 4hour' preview view that snuck in. a yes/no dialog explains how it works before it goes

- wrote a 'mixin' class to let my buttons automatically note current shortcut mapping information to the bottom of button tooltips

- the duplicate commands and a variety of other media viewer buttons that work in the new shortcut system now report their current shortcuts in their tooltips!

- quick-entering 'namespace:' in a search page's tag autocomplete input will now automatically swap in the special 'namespace:*anything*' predicate

- cleaned up code around setting a custom temp_dir through the "temp_dir" command, and made it test the given path is indeed a directory and writable-to. if not, the program dumps out with an error popup

- the client and server now similarly test that the directory db is a dir and writable-to

- rearranged some critical boot error handling code and made the client's boot error handling throw up some ui

- fixed the client's repairdb function when needing to regen autocomplete caches

- autocomplete regen now publishes status updates to the splash screen

- fixed the client's repairdb to wait correctly for the ui in case the user wants to bail before repair

- the client's repair db now repairs the new local_tags_cache table if missing

- fixed an issue with showing the manage logins panel when the domain had some unusual cookie expiry numbers

- to reduce initial load gui-event crunch (which affects some systems' layout reliability), initial session load now happens after a 0.5s delay

- cleaned up some image load error handling

- when FFMPEG version information cannot be understood in help->about, a popup note appears and debug information is printed to the log

- the advanced content update panel now puts up a 'working…' dialog while it processed a job

- in the client

- all 4XX and 5XX network exceptions' error texts are now prepended with the actual status code

- extended the hover window debug report mode

- improved some misc temp and permission testing code

- improved some misc testing code

next week

It is E3 next week, so I am going to take my shitposting vacation week as usual. 356 will therefore be on the 19th. I would like to convert same/worse/better duplicate relationships to the new duplicates system, as I recently did for alternates and false positives. Otherwise I will do a bit of code cleaning.

R: 1 / I: 1 / P: 4 [R] [G] [-]

Release Tomorrow!

I had an excellent and busy week catching up after E3. As hoped, I completed the second phase of the duplicates overhaul–all better/worse/same quality duplicate information is now stored more logically and efficiently. Duplicates processing is faster and duplicate groups are easier to manage. I also fixed a heap of bugs, including the clipboard watcher popup spam and then recent OS X shortcuts issue, and improved some maintenance code. The exit splash screen now has a 'stop doing maintenance' button!

The release should be as normal tomorrow, maybe a little late as there is a bit more to test.

R: 13 / I: 4 / P: 5 [R] [G] [-]

Hydrus dev, I'm pretty sure I invented the proof to the big bang. If I get popular, I'll shout you out.

That's not why I made this thread, though. I'm so anxious I feel like vomiting. I never said this out loud anywhere, but besides porn I have nothing else in my life besides Hydrus. I use Hydrus extremely casually, anyway. But Hydrus helps sort porn, so… I was gonna shout it out.

I was gonna shout out my two favorite artists, too. I didn't have any plans to shout out anything else… So I imagine it'd be a huge influx of traffic.

I can't handle this at all. I never asked for this at all. No one will believe me until it happens anyway.

I have an appointment to show it to someone next week. It's fucked. I wish I could just be in protective custody and sleep for 20 hours and it'll be better then maybe.

R: 23 / I: 18 / P: 5 [R] [G] [-]

how boned are you, anon?

Make my day showing me your boners, lads.

I hope i'm not a degenerate by this board's standards.

R: 37 / I: 6 / P: 5 [R] [G] [-]

Version 354

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v354/Hydrus.Network.354.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v354/Hydrus.Network.354.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v354/Hydrus.Network.354.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v354/Hydrus.Network.354.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v354.tar.gz

I had a great week. The first duplicates storage update is done, and I got some neat misc fixes in as well.

false positives and alternates

The first version of the duplicates system did not store 'false positive' and 'alternates' relationships very efficiently. Furthermore, it was not until we really used it in real scenarios that we found the way we wanted to logically apply these states was also not being served well. This changes this week!

So, 'false positive' (up until recently called 'not dupes') and 'alternates' (which are 'these files are related but not duplicates', and sitting in a holding pattern for a future big job that will allow us to process them better) are now managed in a more intelligent storage system. On update, your existing relationships will be auto-converted.

This system uses significantly less space, particularly for large groups of alts like many game cg collections, and applies relationships transitively by its very structure. Alternates are now completely transitive, so if you have a A-alt-B set (meaning one group of duplicate files A has an alternate relationship to another group of duplicate files B) and then apply A-alt-C, the relationship B-alt-C will also apply without you having to do anything. False positive relationships are more tricky, but they are stored significantly more efficiently and also apply on an 'alternates' level, so if you have A-alt-B and add A-fp-M, B-fp-M will be automatically inferred. It may sound odd at first to think that something that is false positive to A could be nothing but false positive to B, but consider: if our M were same/better/worse to B, it would be in B and hence transitively alternate to A, which it cannot be as we already determined it was not related to A.

Mistakenly previously allowable states, such as a false positive relationship within an alternates group, will be corrected on the update. Alternates will take precedence, and any subsequently invalid false positives will be considered mistakes and discarded. If you know there are some problems here, there is unfortunately no easy way at the moment to cancel or undo an alternate or false positive relationship, but once the whole duplicates system is moved over I will be able to write a suite of logically correct and more reliable reset/break/dissolve commands for all the various states in which files can be related with each other. These 'reset/set none' tools never worked well in the old system.

The particularly good news about these changes is it cuts down on filtering time. Many related 'potential' duplicates can be auto-resolved from a single alternate or false positive decision, and if you have set many alts or false positives previously, you will see your pending potentials queue shrink considerably after update, and as you continue to process.

With this more logically consistent design, alternate and false positive counts and results are more sensible and consistent when searched for or opened from the advanced mode thumbnail right-click menu. All the internal operations are cleaner, and I feel much better about working on an alternates workflow in future so we can start setting 'WIP' and 'costume change'-style labels to our alternates and browsing them more conveniently in the media viewer.

The actual remaining duplicate relations, 'potential', 'same quality', and 'this is better', are still running on the old inefficient system. They will be the next to work on. I would like to have 'same quality' and 'this is better' done for 356, after E3.

the rest

The tag blacklists in downloaders' tag import options now apply to the page's tags after tag sibling processing. So, if you banned 'high heels', say, but the site suddenly starts delivering 'high-heel shoes', then as long as that 'high-heel shoes' gets mapped to 'high-heels' in one of your tag services, it should now be correctly filtered. The unprocessed tags are still checked as before, so this is really a double-round of checking to cover more valid ground. If you were finding you were chasing many different synonyms of the tags you do not like, please let me know how this now works for you.

The annoying issue where a handful of thumbnails would sometimes stop fading in during heavy scrolling seems to be fixed! Also, the thumbnail 'waterfall' system is now cleverer about how it schedules thumbnails that need regeneration. You may notice the new file maintenance manager kicking in more often with thumbnail work.

Another long-term annoying issue was the 'pending' menu update-flickering while a lot of tag activity was going on. It would become difficult to interact with. I put some time into this this week, cleaning up a bunch of related code, and I think I figured out a reliable way to make the menus on the main gui stop updating as long as one is open. It won't flicker and should let you start a tags commit even when other things are going on. The other menus (like the 'services' menu, which will update on various service update info) will act the same way. Overall menu-related system stability should be improved for certain Linux users as well.

The new 'fix siblings and parents' button on manage tags is now a menu button that lets you apply siblings and parents from all services or just from the service you are looking at. These commands overrule your 'apply sibs/parents across all services' settings. So, if you ever accidentally applied a local sibling to the PTR or vice versa, please try the specific-service option here.

full list

- duplicates important:

- duplicates 'false positive' and 'alternates' pairs are now stored in a new more efficient structure that is better suited for larger groups of files

- alternate relationships are now implicitly transitive–if A is alternate B and A is alternate C, B is now alternate C

- false positive relationships remain correctly non-transitive, but they are now implicitly shared amongst alternates–if A is alternate B and A is false positive with C, B is now false positive with C. and further, if C alt D, then A and B are implicitly fp D as well!

- your existing false positive and alternates relationships will be migrated on update. alternates will apply first, so in the case of conflicts due to previous non-excellent filtering workflow, formerly invalid false positives (i.e. false positives between now-transitive alternates) will be discarded. invalid potentials will also be cleared out

- attempting to set a 'false positives' or 'alternates' relationship to files that already have a conflicting relation (e.g. setting false positive to two files that already have alternates) now does nothing. in future, this will have graceful failure reporting

- the false positive and alternate transitivity clears out potential dupes at a faster rate than previously, speeding up duplicate filter workflow and reducing redundancy on the human end

- unfortunately, as potential and better/worse/same pairs have yet to be updated, the system may report that a file has the same alternate as same quality partner. this will be automatically corrected in the coming weeks

- when selecting 'view this file's duplicates' from thumbnail right-click, the focus file will now be the first file displayed in the next page

- .

- duplicates boring details:

- setting 'false positive' and 'alternates' status now accounts for the new data storage, and a variety of follow-on assumptions and transitive properties (such as implying other false positive relationships or clearing out potential dupes between two groups of merging alternates) are now dealt with more rigorously (and moreso when I move the true 'duplicate' file relationships over)

- fetching file duplicate status counts, file duplicate status hashes, and searching for system:num_dupes now accounts for the new data storage r.e. false positives and alternates

- new potential dupes are culled when they conflict with the new transitive alternate and false positive relationships

- removed the code that fudges explicit transitive 'false positive' and 'alternate' relationships based on existing same/better/worse pairs when setting new dupe pairs. this temporary gap will be filled back in in the coming weeks (clearing out way more potentials too)

- several specific advanced duplicate actions are now cleared out to make way for future streamlining of the filter workflow:

- removed the 'duplicate_media_set_false_positive' shortcut, which is an action only appropriate when viewing confirmed potentials through the duplicate filter (or after the ' show random pairs' button)

- removed the 'duplicate_media_remove_relationships' shortcut and menu action ('remove x pairs … from the dupes system'), which will return as multiple more precise and reliable 'dissolve' actions in the coming weeks

- removed the 'duplicate_media_reset_to_potential' shortcut and menu action ('send the x pairs … to be compared in the duplicates filter') as it was always buggy and lead to bloating of the filter queue. it is likely to return as part of the 'dissolve'-style reset commands as above

- fixed an issue where hitting 'duplicate_media_set_focused_better' shortcut with no focused thumb would throw an error

- started proper unit tests for the duplicates system and filled in the phash search, basic current better/worse, and false positive and alternate components

- various incidences of duplicate 'action options' and similar phrasing are now unified to 'metadata merge options'

- cleaned up 'unknown/potential' phrasing in duplicate pair code and some related duplicate filter code

- cleaned up wording and layout of the thumbnail duplicates menu

- .

- the rest:

- tag blacklists in downloaders' tag import options now apply to the parsed tags both before and after a tag sibling collapse. it uses the combined tag sibling rules, so feedback on how well this works irl would be appreciated

- I believe I fixed the annoying issue where a handful of thumbnails would sometimes inexplicitly not fade in after during thumbgrid scrolling (and typically on first thumb load–this problem was aggravated by scroll/thumb-render speed ratio)

- when to-be-regenerated thumbnails are taken off the thumbnail waterfall queue due to fast scrolling or page switching, they are now queued up in the new file maintenance system for idle-time work!

- the main gui menus will now no longer try to update while they are open! uploading pending tags while lots of new tags are coming in is now much more reliable. let me know if you discover a way to get stuck in this frozen state!

- cleaned up some main gui menu regeneration code, reducing the total number of stub objects created and deleted, particularly when the 'pending' menu refreshes its label frequently while uploading many pending tags. should be a bit more stable for some linux flavours

- the 'fix siblings and parents' button on manage tags is now a menu button with two options–for fixing according to the 'all services combined' siblings and parents or just for the current panel's service. this overrides the 'apply sibs/parents across all services' options. this will be revisited in future when more complicated sibling application rules are added

- the 'hide and anchor mouse' check under 'options->media' is no longer windows-only, if you want to test it, and the previous touchscreen-detecting override (which unhid and unanchored on vigorous movement) is now optional, defaulting to off

- greatly reduced typical and max repository pre-processing disk cache time and reworked stop calculations to ensure some work always gets done

- fixed an issue with 'show some random dupes' thumbnails not hiding on manual trashing, if that option is set. 'show some random dupes' thumbnail panels will now inherit their file service from the current duplicate search domain

- repository processing will now never run for more than an hour at once. this mitigates some edge-case disastrous ui-hanging outcomes and generally gives a chance for hydrus-level jobs like subscriptions and even other programs like defraggers to run even when there is a gigantic backlog of processing to do

- added yet another CORS header to improve Client API CORS compatibility, and fixed an overauthentication problem

- setting a blank string on the new local booru external port override option will now forego the host:port colon in the resultant external url. a tooltip on the control repeats this

- reworded and coloured the pause/play sync button in review services repository panel to be more clear about current paused status

- fixed a problem when closing the gui when the popup message manager is already closed by clever OS-specific means

- misc code cleanup

- updated sqlite on windows to 3.28.0

- updated upnpc exe on windows to 2.1

next week

The duplicates work this week took more time than I expected. I still have many small jobs I want to catch up on, so I am shunting my rotating schedule down a week and doing a repeat. I will add some new shortcuts, some new tab commands, and hopefully a clipboard URL watcher and a new way of adding OR search predicates.

R: 0 / I: 0 / P: 5 [R] [G] [-]

Release Tomorrow!

I had a good week doing small work. The error-popup issue with collected thumbnails is fixed, and there are new shortcuts to better allow keyboard-only navigation of the manage tags dialog, an experimental jpeg quality estimator for the duplicate filter, a fun clipboard watcher for easy URL import, and some new options for file viewing statistics.

The release should be as normal tomorrow.

R: 15 / I: 3 / P: 5 [R] [G] [-]

Version 353

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v353/Hydrus.Network.353.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v353/Hydrus.Network.353.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v353/Hydrus.Network.353.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v353/Hydrus.Network.353.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v353.tar.gz

I had a great week. I finished the basics of the new file maintenance system I wanted, cleaned up the duplicate filter a little more, and fixed a bunch of bugs.

file maintenance system

There are a number of large file re-checking jobs the client wants to do, both now and in the future. Going back to figure out more accurate video durations and image rotations, discovering webms that were formerly incorrectly detected as mkvs, eventually integrating videos into the duplicate checking system, all of these will require a combined whack of maintenance CPU that I don't want to hit all at once. I have previously sketched out some disparate systems for these jobs, but none were really doing the trick, so this week I unified it all into one nice system that can handle all sorts of jobs. This new system is simple for now but will get more work in future.

You do not have to do anything, but if you pay attention to your maintenance work, you may notice some new file metadata and thumbnail jobs running in the background or on shutdown. This new system fits into normal maintenance just like database analyzing or repository processing. You can govern whether it is permitted to run in regular idle time and/or shutdown time under options->maintenance and processing and also change its in-built 'throttle', which limits the number of files it will work on (rather than running full bore on what in future may be quite large jobs). The default throttle is 200 files every day, which for most jobs on most machines will be about 30 seconds to three minutes work. Do not expect it to do much work yet.

Existing file regeneration routines now work through this system, and it does its job much better than before. If you hit right-click->regenerate->x on some thumbnails, the job now runs in a regular popup button (rather than the locking 'modal' one from before), letting you keep browsing while it works. And if you select more than 50 thumbnails (think, say, right-clicking on 2,000 video files and saying to regenerate their thumbnails if they are the wrong size), you will now get the option to schedule that big job for later, at which point those 2,000 jobs will end up in the normal idle maintenance queue, to work at 200 files a day or whatever you wish.

This system is fairly opaque at the moment. You can trigger it with the thumbnail right-click, and certain db operations may schedule new jobs for it, but there is no UI to review it yet. In the coming weeks, I expect to write a new 'review' window off the database menu that will let you review total pending jobs, start work manually, and add and remove pending jobs en masse through the regular search interface. I'll slowly integrate more of the client into it as well, letting it add more jobs into the queue by itself.

Let me know how this all works for you!

duplicate filter

The duplicate filter interface got some more work this week, particularly in cleaning up some of my original version's over-engineering. The actions you can choose on the right panel are now split more clearly into 'yes, these files are duplicates, and here is how' decisions vs the 'alternates' and 'not dupes' decisions. Also, 'this file is better' is now split into two buttons for 'delete the worse file' and 'keep both'. This 'delete or not' is split at the shortcut level into two actions as well, if you wish to map both. Existing shortcuts (left-click by default in the filter) will update to the 'and delete the worse file' version.

The complicated 'duplicate action options' object (which governs how to merge metadata across duplicates) therefore no longer handles file deletion. It is also now only attached to the 'better/worse' and 'files are the same' actions–we never found a good reason to merge metadata across all alternates or 'not duplicates', so I have removed it completely. If you want a complicated file delete action, hitting the 'custom action' button now asks you if you wish to delete the file you are looking at, the other one, or both.

Also, to reduce confusion with alternates–which are also technically not duplicates–'not duplicates' is now renamed across the program to the more precise 'not related/false positive'. The 'false positive' action is a record in the db saying 'despite the similar files search thinking these files were related, it was incorrect, so do not bring it up again'.

My hope is that filtering is a bit faster here. If two duplicate files are of very different quality, it is still easy to delete the bad one, but if they are more close and you want to keep both, it is now just one click.

As for the big db-level rewrite, I prepped the duplicate db code for it this week. I am standing at the cliff-edge and feel great about jumping off, so next week I hope to get started on the new code properly and migrate one or both of the current 'alternates' and 'false positive' data to the new system.

the rest

I fixed an issue with the recent 'collect by' session saving where the accompanying sort was not being renewed on a session load. Also, several problems with collected media and sort by 'approx bitrate' are fixed.

There's a new checkbox under options->sort/collect that makes it so the default sort is updated every time you click a new sort in regular browsing. It sounds a pain but is actually pretty neat!

The 'all local files' domain is now hidden from view in new page selection and the tag autocomplete dropdown if you are not in advanced mode. This domain, which is fairly technical and covers both trash and 'my files' and the sometimes-hidden repository update files, is often confusing to new users and is rarely useful even for people who know what it does.

If you use the client's local booru and need to override its host when you copy an external link, this option has moved from options->connection to the local booru's manage services panel. You can also override scheme and port as well! The old host override option is gone completely, and the only other place it was used, the manage upnp dialog, now fetches this info more efficiently and fails more gracefully.

full list

- duplicate filter:

- duplicate action options no longer handle file deletion

- renamed 'not duplicates' across the program to 'not related' or 'false positive'

- 'alternates' and 'not related/false positive' duplicate actions no longer have duplicate action options. no merge content update now occurs on these actions

- the duplicate filter hover panel now splits 'this is better' decisions into two buttons–whether to delete or keep the worse file

- when selecting 'custom action' in the duplicate filter hover panel, it now asks if you would like to delete the current file, the other file, or both

- the 'duplicate_filter_this_is_better' shortcut action will be auto-updated to 'duplicate_filter_this_is_better_and_delete_other'. an alternate 'duplicate_filter_this_is_better_but_keep_both' is now also available

- the 'duplicate_filter_not_dupes' shortcut action will be auto-updated to 'duplicate_filter_false_positive'

- separated the buttons on the duplicate filter hover panel to more carefully split 'yes, files are duplicates' vs other decisions

- in prep for the duplicate db overhaul, refactored all PHash search code and Duplicate management code apart

- misc other prep work for duplicate db overhaul

- .

- file maintenance:

- wrote a new unified manager to handle various long-term file maintenance tasks like regenerating file metadata and thumbnails

- options to govern how this manager can run are now in options->maintenance and processing. you can enable it for idle and shutdown maintenance time and give it a throttle to limit how fast it will work on files, defaulting to 200 per day

- unified the previous db-level attempts at file maintenance to the new system, which supports async job queueing, and moving regen code up to the new manager, out of the db lock

- unified a variety of file and thumbnail regen code to work through the new simpler and saner path

- the right-click->regen thumbnail commands now run through the new manager and no longer need a modal popup. you can keep browsing while they work. they will also not hang the ui as the old system could on big jobs

- when right-click->regenning on more than 50 thumbnails, you now get a dialog asking if you want to do the job now or put it off later

- file maintenance tasks can now run in shutdown time! you will get previews of the jobs with file counts and status progress reports on the shutdown splash

- cleaned up some file extension renaming and dupe-removing code

- in future, I will move the current file integrity check to this new system and have some ui to prompt and set up other big jobs, like fixing various historical misparsing issues

- thumbnail resizing during thumbnail fade that resizes down is now more efficient

- moved the ClientFilesManager to ClientFiles.py

- .

- the rest:

- the 'manage upnp' dialog now moves the duplicated external ip display from the column up to the status text at the top. it fetches the ip after the initial mappings fetch is done. this ip is no longer affected by the external host override option

- cleaned up options->connection page and removed the now defunct external host override option

- the manage services page for the local booru now has optional override for scheme, host, and port for the 'copy external url' function

- fixed an issue with the recent 'collect by' session saving where a restored session that needed a collect was not sorted

- fixed an issue with collections being sorted by approx bitrate

- added a new checkbox to options->sort/collect to set it so the default sort updates every time you choose a new sort anywhere

- fixed an issue with 'remove trashed files from view', which was incorrectly removing on 'all local files' pages

- the 'all local files' file domain, which is frequently confusing to new users, is now no longer an option for new file pages or the autocomplete file domain if the user is not in advanced mode

- the client now searches for versions of urls both with and without a final '/' character when looking up file url import status at the db level and in import lists. system:known_url is unfortunately still an inefficient mess

- improved how the server code deals with some connectionLost errors

- cleaned up and unified some older dialog button code

- fixed a problem in manage tag siblings when petitioning existing pairs and then cancelling when asked for a reason

- fixed a miscount issue when uploading pending tags while many new tags are coming in. progress would sometimes be -754/1,234, ha ha

- db maintenance, repository sync, and file maintenance processing will all now wake on a force idle mode call

- deleted some old code

- misc fixes and cleanup

- some misc gui layout fixes

next week

I have quite a few smaller jobs waiting for me, so other than the new duplicate db tables, that's top priority. Some UI bugs to deal with, maybe some Client API work, an experimental jpeg quality estimator, possibly support for some new filetypes, and hopefully a fun new way to quickly add very complicated OR search predicates thanks to a clever user's work.

Just a note, E3 is coming up soon and I will take my shitposting vacation week for it as usual. I think it'll be 356 that's delayed a week.

R: 15 / I: 1 / P: 5 [R] [G] [-]

Sorting Motivation Thread

ITT we post about our duplicates, tagging, and so on - our stats, goals, landmarks, and so on.

Never give up anons, that tidy catalog of content is waiting for you!

R: 4 / I: 0 / P: 5 [R] [G] [-]

Release Tomorrow!

I had a great week. The new 'false positive' and 'alternates' duplicates db storage design and migration worked out well, although it did take most of my time. Other than that, I did a variety of little new options and bug fixes. The annoying bug where a handful of thumbnails sometimes stop fading in seems to be finally fixed!

The release should be as normal tomorrow.

R: 0 / I: 0 / P: 5 [R] [G] [-]

Release Tomorrow!

I had a great week. I finished the first version of the new file maintenance system I had planned, cleaned up the duplicate filter workflow a little more, and fixed a bunch of bugs.

The release should be as normal tomorrow.

R: 24 / I: 1 / P: 5 [R] [G] [-]

Version 352

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v352/Hydrus.Network.352.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v352/Hydrus.Network.352.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v352/Hydrus.Network.352.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v352/Hydrus.Network.352.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v352.tar.gz

I had a good week. .ico files are now supported, 'collect by' status is remembered in gui sessions, and I fixed a bunch of bugs.

duplicate overhaul plans

I started the duplicate overhaul work this week with some planning and experimentation with existing data. My original thought here had been to exactly replicate existing functionality just with a more efficient database schema, but having gone through the various edge-case insertion and merge operations, I believe the current system is overcomplicated for what we are actually using it for.

Most of all, the current system tries to form a chain of 'better/worse' comparisons so all dupes within a 'same file' group are ranked with each other. A decent number of human decisions are needed to determine this ranking, but the data is currently not displayable, and we haven't really noticed that absence. For most practical circumstances, what we really want to determine is what files should actually be considered dupes, and which of those groups is then the best. Most users delete the 'worst' of a pair in any case. Supporting a system that simply tracks a group of duplicates with a single King is more intuitive and reliable, and it is quicker to work with.

This 'King of a group' idea also maps nicely to how we use 'tag siblings'–having a complicated tree or chain of worst tag to best is not as useful as simply replacing all the lesser members of a group with the King as needed. When I get to overhauling tag siblings, I expect to make a similar change.

But in the meantime, for duplicates, I now have a plan. I expect to spend a few more weeks filling out the full details in code, and then I will switch us all over. The existing workflows should remain the same, just with fewer and easier comparisons. I will not do much specific work on file alternates, but they will be feasible to start on once the db overhaul is done. I will also continue to put time into the duplicate filter ui itself. Overall, I feel good about it all. I'd like the whole thing to be done within 8-12 weeks.

otherwise all misc this week

.ico files should now be supported! .cur files (which are basically the same) should work as well.

'collect by' settings are now, finally, saved in page sessions! If your default collect by settings include any ratings services, they will be forgotten on update, so you will have to reset them in options->sort/collect this one time.

I fixed the stupid issue where media viewer hover windows were popping up over manage tags and some other dialogs. This was due to a flaw in the changes from the new always-on-top duplicate hover panel–I apologise for the inconvenience. Some related OS X specific weirdness should be cleaned up as well.

The 'unclose_page' shortcut (default Ctrl+u) now uncloses pages in the correct order!

When a media fits the media viewer exactly (so 100% zoom fits the width or height exactly), the 'zoom switch' action (default 'z') now correctly restores back to 100%!

'open externally' should work better for some custom program paths. The flash projector (for .swf files) was opening without an ui, for instance. If you have had other programs seem to open in the background from open externally calls, please give them another go and let me know if they now work for you.

full list

- the client now supports importing .ico files! (.cur should be supported too)

- finally, 'collect by' is saved for sessions! if your default collect by previously included ratings services, it will forget them this one time–please reset it under the options->sort/collect

- fixed the issue where the media viewer's hover windows were hovering over child dialogs (manage tags, ratings, or known urls)

- improved some os x hover window focus handling for the new always-on-top duplicate action window

- the entries on the 'sort by' list on gui pages are now subcategorised better. it should be a bit easier to find what you are looking for

- the 'sort by file: approximate bitrate' sort option now sorts still images as well by filesize / num_pixels

- to reduce confusion, sort by mime and system:mime are now renamed to 'filetype'

- fixed an issue where the 'unclose_page' shortcut was restoring pages in reverse order (unclosing least-recently-closed-first rather than most-recently-closed-first)

- improved rigour of video framerate estimation

- stopped the video metadata parser from opting to manually frame count videos with size >128MB or num_frames estimate >2,400

- fixed the forced manual frame count to deal with frame counts >9999

- the 'ffmpeg not found' error on file import will now put up a popup message once per boot informing you of this problem more broadly and steps to address it

- fixed some underreporting issues with subprocess_report_mode

- fixed an issue with some yes/no dialogs returning 'no' on escape/window_close_button rather than 'cancel', which affected cancelability some db maintenance questions

- fixed an issue where media that fitted the media viewer canvas width or height exactly at 100% zoom would not respond to zoom switch events to restore non-100% zoom to 100%

- when a local server's CORS mode is turned on, Access-Control-Allow-Origin is now correctly added to GET/POST requests with an Origin request header

- improved reliability of some timestamp rendering code, which should help some users who had trouble opening cookies management page after malformed cookie import

- I believe I fixed an issue with 'open externally' on certain custom paths where the external program could spawn without an ui (flash projector did this). please let me know if your 'open externally' calls start making terminal windows everywhere

- fixed a runtime stability issue with the new duplicates page and slow-updating counts that come in after the page has been deleted

next week

Next week is an 'ongoing' week, where I work on a medium-sized improvement to an existing system. I think I would like to put some time into my 'background file maintenance' plans, unifying the current prototype systems into one that runs nicely in idle time and adding some ui controls for it. There are several pending file-reparsing jobs I would like to queue up (e.g. checking mkvs vs webms with modern file parsing code, and apng discovery, scheduling background thumbnail regen, and fixing bad old frame counts and durations), and if I want to integrate videos into the duplicates system, I'll need a better framework here to schedule that retroactive CPU work. Otherwise I have a couple little cleanup jobs to be getting on with, and I'll start some new duplicates db code.

R: 5 / I: 0 / P: 5 [R] [G] [-]

Bibanon and WebUI thread

It looks like the Bibanon wants to use Hydrus as a server, and they really want to slap a webUI on top of it… but Hydrus Dev don't have it on priority list.

They have though about using Szurubooru for the WebUI but rr- don't want to update it. See: https://github.com/rr-/szurubooru and rr-@sakuya.pl

Maybe it is time to draft up a sample API set and WebUI for them.

From antonizoon 2 Sep 2018

Hey there, it's been a very long while, but it's good to see that you've
kept working on Hydrus ever since.

One of the issues with hydrus for me in the end, which is why we never
ended up using it, is because of the lack of a webui. As such I ended up
utilizing Szurubooru, as can be seen here: https://eikonos.bibanon.org

However, what I can say is that the dev team at the Library of Congress
was very interested in adding tag support to their image tag system
which is below. It's currently utilized with the aim of crowdsourcing
transcriptions of images, but it lacks tagging or search like we would
have in a booru. Luckily, we already have the hydrus protocol and
szurubooru right? They aim to launch in October.

I find this to be a perfect chance to integrate Hydrus with a webui as
well as obtain some useful features from their existing system as well.
What do you think?

https://github.com/LibraryOfCongress/concordia

Possible CSS frameworks

https://tutorialzine.com/2018/05/10-lightweight-css-frameworks-you-should-know-about

https://dzone.com/articles/top-10-lightweight-css-frameworks-for-building-fas

https://www.catswhocode.com/blog/top-10-lightweight-css-frameworks-for-building-fast-websites-in-2018

http://www.creativeweblogix.com/blog/12-lightweight-css-frameworks

https://speckyboy.com/responsive-lightweight-css-frameworks/

Also whether to use Angular/React/Vue (JS web) vs Django/CherryPy/TurboGears/Flask/ Pyramid (Python Web) comes into question

Contacts

Twitter - https://twitter.com/BAnonoma

IRC - irc.rizon.net #bibanon

Discord - https://discord.gg/0kPf8xuFPFa3iihT

R: 751 / I: 130 / P: 5 [R] [G] [-]

Q&A Thread: For simple questions that don't need their own thread

Here you can ask questions so that the board is not clogged with small threads.

>>6021 has reached its bump limit, so I made a new thread.

R: 0 / I: 0 / P: 5 [R] [G] [-]

Release Tomorrow!

I had a good week. I added .ico file import support, got 'collect by' settings saving through gui sessions, and fixed a whole bunch of bugs (including the annoying media-window-hover-frame-over-manage-tags issue).

The release should be as normal tomorrow.

R: 14 / I: 3 / P: 5 [R] [G] [-]

Version 351

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v351/Hydrus.Network.351.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v351/Hydrus.Network.351.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v351/Hydrus.Network.351.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v351/Hydrus.Network.351.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v351.tar.gz

I had an ok week. I sped up several systems and added a new processing panel to the duplicate filter.

The 'next big job' poll is finished! I will next be focusing on overhauling the duplicate filter's db structure (including sketching out support for file 'alternates') and further improving the ui-side workflow.

duplicate filter

Seeing that the duplicate filter work was popular in the poll, I was happy to put a bit more time into it this week.

Most importantly, the duplicate filter now has a new always-on-top panel to make reviewing differences and making decisions easier. Essentially, I have pulled the 'this has higher resolution' statements and the action buttons out of the top hover window and put them in their own box to the middle-right. It stays on top, so you can always see it, and I have expanded the different statements to explicitly state each file's relevant values, such as '550KB >> 141KB', and to colour their text green/blue/red based on that difference. You can now make at-a-glance decisions for easy pairs. Let me know how it works for you!

I also have tweaked the 'show some random dupes' and 'duplicate filter' database routines to sample results more cautiously, meaning that duplicate filters with very large search domains (like system:inbox or system:num_tags>0) will work significantly faster. The initial search step still has to run every time, but the second 'sampling' stage takes barely any time at all. If your dupe work still takes a long time to count up or load pairs to filter, I'll recommend again to just add a 'creator:' tag.

The next 'big job' work here is to overhaul the duplicate database tables to work in more intelligent 'groups' rather than my initial simple 'pairs' system. This will compact the duplicates data, speed up many operations, massively simplify transitive duplicate logic, and lead to alternate file structure support. I'll also probably copy the basic structure to tag siblings and parents, which are a similar data structure also currently stored in pairs. This job will take some work, but I know the general thrust of what I want to do.

faster code and some misc work

I cleaned and improved a bunch of old code this week. First off, image loading now uniformly uses a faster library, so image imports and some thumbnail creation should be a little bit faster and deal with some rare image rotations more reliably. Secondly, the way the tag siblings and parents managers construct themselves on client boot is significantly faster. And a new 'local' tag cache–which will take a minute to construct when you update the client–will speed up many tag-related operations, particularly file results building, especially right after the client boots. There are many changes here, so please report any bugs you see.

review services now uses nested notebooks instead of my old 'listbook' control. I don't really like how it looks now, but the code behind the scenes is a lot saner. The last instance of this bad listbook is the options dialog, which is even worse behind the scenes but a bit too big to fit into a single notebook–I am still thinking about what to do with it.

It is just a little thing, but subscription popups now list file download progress x/y in terms of the current job rather than total subscription history! So, if your sub has 400 files already downloaded and finds 10 more in a sync, the popup will now say (and display a progress gauge for) a much more helpful 3/10 rather than 403/10.

full list

- wrote a new (always on top!) hover window for the duplicate filter that sits on the middle-right. the duplicate cog button and action buttons are moved to this new window, as are the file comparison statements

- the duplicate file comparison statements now state the relevant actual metadata along with better '>>'-style operators to highlight differences and green/red/blue colouring based on given score. it is now much easier to see and action clearly better files at-a-glance

- improved some hover window focus display calculations to play with the new always-on-top tech

- both the 'show some random dupes' button and finding dupe pairs for the filter should be a bit faster for very large search domains. the basic file search and indexing still has to run, but the second sampling step in both cases will bail out earlier once it has a decent result

- core image handling functions now uniformly use OpenCV (faster, more accurate) by default, falling to PIL/Pillow on errors. image importing in the client and server should be a bit faster, and some unusual image rotations should now be read correctly

- the server now supports OpenCV for image operations, it _should_ also still work with only PIL/Pillow, if you are running from source and cannot get it

- unified all thumbnail generation code and insulated it from suprises due to unexpectedly-sized source files, fixing a potential client-level thumbnail generation looping bug

- gave all image processing a refactor and general cleanup pass, deleted a bunch of old code

- wrote a new 'local tag cache' for the db that will speed up tag definition lookups for all local files. this should speed up a variety of tag and file result fetching, particularly right after client boot. it will take a minute or two on update to generate

- sped up how fast the tag parent structure builds itself

- the review services panel now uses nested notebooks, rather than the old badly coded listbook control. I don't really like how it looks, but the code is now saner

- similar-files metadata generation now discards blank frames more reliably

- subscription popups now report x/y progress in terms of the current job, discarding historical work previously done. 1001/1003 is gone, 1/3 is in

- made the disk cache more conservative on non-pre-processing calls

- cleaned up some file import code, moving responsibility from the file locations manager to the file import object

- updated the ipfs service listctrl to use the new listctrl object. also cleaned up its action code to be more async and stable

- I believe I fixed a rare vector for the 'tryendmodal' dialog bug

- fixed a bug in presenting the available importable downloader objects in the easy drag-and-drop downloader import when the multiple downloaders dropped included objects of the same type and name–duplicate-named objects in this case will now be discarded

- unified url_match/url_class code differences to url class everywhere

- updated some common db list selection code to use new python string formatting

- plenty of misc code cleanup

next week

I'll now tackle this dupe db work. I'll plan and experiment first. Otherwise, next week is a small jobs week. I'd like to add .ico support and add pages' 'collect by' dropdown value to sessions so it gets saved.

R: 55 / I: 5 / P: 5 [R] [G] [-]

Next Big Job Poll

Here is the poll for the next big job:

https://www.poll-maker.com/poll2331269x9ae447d5-67

You can vote for multiple items. I expect to start work on the top-voted item at roughly the time of the v351 release post, in two weeks.

Please feel free to discuss and ask about items on the poll in this thread.

R: 13 / I: 4 / P: 5 [R] [G] [-]

Version 350

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v350/Hydrus.Network.350.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v350/Hydrus.Network.350.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v350/Hydrus.Network.350.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v350/Hydrus.Network.350.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v350.tar.gz

I had an ok week. Some IRL things cut into my hydrus time, but I got some good work done. Some bugs in the new duplicate search system are fixed, and I improved advanced file delete and export handling.

The poll for the next 'big job' is up here: >>12358

duplicate filter

The search addition to the duplicate filter went fairly well, but there were a couple of significant bugs. The 'ghost pair' issue–where a queue would sometimes have a final pair that would never display and lead to high CPU until the filter was closed–is fixed, and safeguards added to catch similar issues in future. The issue with undercounting on large search domains (typically where the dupe page's file query was non-system:everything and covered >10,000 files) is also fixed, but giving the filter 500,000 custom files to work with can be really quite slow. I will keep working here to see if I can speed up big searches like this without compromising accuracy, but if you find your dupe searches are working too slow, try adding a creator: tag to bring the search size down–it works well.

The duplicate filtering workflow itself is still a v1.0 pain. I expect to put some work in here in the coming weeks (and doubly so if dupe work wins the big poll), likely highlighting the differences between the two images with an always-on-top panel and better at-a-glance decision-making for easy comparisons.

advanced delete

This week brings an optional advanced file delete dialog. You can turn it on under options->files and trash and also set custom file delete 'reasons' to assign. Once it is switched on, any file delete under the thumbnail grid, media viewer, or duplicate filter manual file delete will instead give a richer dialog that lets you physically delete files immediately (i.e. skipping the trash) and deleting files without leaving a delete 'record' (which is useful if you want to easily reimport those files later on).

You can also choose from your set reasons or type a completely custom one. These saved reasons reappear when that deleted file fails to import in future (due to being recognised as previously deleted), so if you are interested in tracking why you have previously deleted files, please check this out.

the rest

Export folders now have more timing options, basically the same as import folders. You can pause them individually, tell them not to run periodically, and force them to run manually under a new submenu under file->import and export folders->run export folder now.

The Artstation downloader seems to be not working due to a Cloudflare issue. I am leaving the downloader in for now, in case it starts working again, but if you have a subscription for it, I recommend you pause it for now. The default downloader for new users (and you, if you have it still set as the default) is now safebooru.

If you use the Dolphin file manager, check out this add-on that uses the Client API: https://gitgud.io/prkc/dolphin-hydrus-actions

Advanced users only: Missing file folder recovery is better–if a file folder (one of the subfolders, like 'f39' or 'ta2') is missing on boot but found in another 'known' location, the client will detect the moved folder and propose an auto-update of its records rather than forcing you to go through the complicated manual recovery dialog. If for some reason it is complicated or slow for you to move your folders from A to B through the migrate database dialog, you can now shut down your client, move the folders manually in explorer or wherever, and then reboot the client and be up and working in one click! Just make sure that hydrus knows about B in its locations under migrate database beforehand, and it will do the rest.

full list

- the duplicate filter no longer applies the implicit system limit (which defaults to 10,000 files) on its search domains, which solves the undercounting issue on large search domains. duplicate operations will be appropriately slower, so narrow your duplicate file queries (adding a creator: tag works great) if they take too long

- fixed the duplicate pairs filter's 'ghost pair' issue. it was failing, when 'both files' was unchecked, to remove pairs that included one file that was non-local. this accidental inclusion resulted in incorrect non-zero count and filter/random pairs that could not display correctly

- insulated against potential future iterations of this problem (likely that one of the pair was deleted by another process while a filter is ongoing), with a notification and graceful exiting of the duplicate filter while saving progress

- the 'show random duplicates' button now puts the 'base' of the group (to which all the others are potentially matched) as the first thumbnail

- added a new 'advanced file deletion' section to 'files and trash' options page to turn on a new advanced dialog and set custom file deletion reasons

- if this new dialog is turned on, a delete event from thumbnail grid, regular media viewer, or the duplicate filter's manual delete will launch it. it permits you to delete physically (skipping trash) in one step or delete physically without leaving a deletion record (for easier later re-import) and choose one of the deletion reasons in the new options panel or set a one-time custom reason

- export folders now have more run-controls: 'run regularly', 'paused', and 'run now'

- the file menu now has a 'run export folder now' submenu just like for import folders– it is simple now to set up an export folder that only runs when you tell it to

- updated the on-boot missing file folder recovery process to automatically 'heal' file location mappings when a missing folder is actually in a location that is known (essentially, you can now manually move a bunch of folders from one known location to another while the client is off and it will recover automatically now). error dialogs will appear in this case summarising the problem and proposed fixes with a chance to bail out if you do not want it to happen

- added a new frame type to 'gui' options page called 'regular_center_dialog' for yes/no style dialogs that are better in the center of the parent window

- the custom web browser launch path and file type 'open externally' paths are moved from 'files and trash' to a new 'external programs' options page

- as the superior '–temp_path' program launch parameter now exists for both client and server, I have removed the limited 'BUGFIX: temp folder override' option from the client's 'files and trash' page and use in the actual code. if this option was important to you, please migrate to the –temp_path launch parameter, which covers temp usage more comprehensively

- as the artstation downloader is now non-functional, apparently by a cloudflare issue, the default gug for new users (and anyone with artstation set atm) is now safebooru

- added dolphin file manager add-on link to the client api help

- some misc file metadata fetching cleanup

next week

Next week is a cleanup week. Beyond boring ongoing code cleanup and rewrites, I would like to move some 'default' image parsing to preference OpenCV over PIL uniformly, including for the server (which currently does not need OpenCV). Having image work preference one library over the other at different times causes problems when they disagree about some image metadata (e.g. whether a file is rotated). OpenCV is fairly easy to get for all platforms now, and it generally runs faster and better, so I feel good about adding it to the server and making it the primary choice in all situations.

We breezed past 500 million mappings on the PTR in the past couple of weeks! This is great, and I am really thankful for everyone's contributions, but this growth is putting pressure on several different areas. You may have noticed we finally hit my 256GB/month bandwidth limit on the shared PTR account at the end of April. Regular PTR-syncing client databases are also getting huge, about 18GB. I have multiple plans for how to deal with the various issues here and expect to chip away at them throughout 2019. I may reserve a month or two of future 'big job' time to handle this work.

R: 4 / I: 1 / P: 6 [R] [G] [-]

Release Tomorrow!

I had an ok week. I cleaned a whole bunch of code, sped up image importing and the new duplicate operations, and wrote a new always-on-top action panel for the duplicate filter that makes for faster processing.

The release should be as normal tomorrow.

R: 10 / I: 0 / P: 6 [R] [G] [-]

API Thread

ITT: We propose new features that can be solved by using the API, and recommend new API commends for it

R: 1 / I: 0 / P: 6 [R] [G] [-]

local variable 'photoshop' referenced before assignment

Hey,

awesome tool, really enjoying it.

I did encounter an error that came up three times when fetching files from Sankaku (out of probably around 10k+ files imported so far, I've only just started using hydrus last week):

local variable 'photoshop' referenced before assignment… (Copy note to see full error)

Traceback (most recent call last):

File "include\ClientImportFileSeeds.py", line 1178, in WorkOnURL

self.DownloadAndImportRawFile( file_url, file_import_options, network_job_factory, network_job_presentation_context_factory, status_hook, override_bandwidth = True )

File "include\ClientImportFileSeeds.py", line 571, in DownloadAndImportRawFile

self.Import( temp_path, file_import_options )

File "include\ClientImportFileSeeds.py", line 790, in Import

( status, hash, note ) = HG.client_controller.client_files_manager.ImportFile( file_import_job )

File "include\ClientCaches.py", line 1144, in ImportFile

file_import_job.GenerateInfo()

File "include\ClientImportFileSeeds.py", line 283, in GenerateInfo

self._file_info = HydrusFileHandling.GetFileInfo( self._temp_path, mime )

File "include\HydrusFileHandling.py", line 233, in GetFileInfo

( ( width, height ), duration, num_frames ) = HydrusImageHandling.GetImageProperties( path, mime )

File "include\HydrusImageHandling.py", line 241, in GetImageProperties

( ( width, height ), num_frames ) = GetResolutionAndNumFrames( path, mime )

File "include\HydrusImageHandling.py", line 274, in GetResolutionAndNumFrames

pil_image = GeneratePILImage( path )

File "include\HydrusImageHandling.py", line 92, in GeneratePILImage

pil_image = PILImage.open( fp )

File "site-packages\PIL\Image.py", line 2690, in open

File "site-packages\PIL\Image.py", line 2676, in _open_core

File "site-packages\PIL\JpegImagePlugin.py", line 783, in jpeg_factory

File "site-packages\PIL\ImageFile.py", line 103, in init

File "site-packages\PIL\JpegImagePlugin.py", line 373, in _open

File "site-packages\PIL\JpegImagePlugin.py", line 139, in APP

UnboundLocalError: local variable 'photoshop' referenced before assignment

The URLs in question are these ones:

https://chan.sankakucomplex.com/post/show/7921258

https://chan.sankakucomplex.com/post/show/7921259

https://chan.sankakucomplex.com/post/show/7921260

I didn't specifically add the URLs, I did a tag search for "1girl lying_on_bed".

I've retried a few times and got the same error. Downloading the actual images and trying to import them manually results in the same error.

I'm using hydrus version 349 on Windows 10 Pro (version 1809).

It's not an issue for me since I don't care about these three files anyway, but I thought you might want to know.

R: 0 / I: 0 / P: 6 [R] [G] [-]

Release Tomorrow!

I had an ok week. I fixed some bugs from the recent duplicate filter changes, wrote a new advanced file delete dialog, and added new timing controls to export folders.

The release may be late tomorrow.

R: 26 / I: 5 / P: 6 [R] [G] [-]

Version 349

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v349/Hydrus.Network.349.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v349/Hydrus.Network.349.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v349/Hydrus.Network.349.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v349/Hydrus.Network.349.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v349.tar.gz

I had a great week. The duplicate filter work went really well, and the manage tags dialog has a neat new button for fixing siblings and parents.

The poll for the next 'big job' is live here >>12358 ! The direct link is https://www.poll-maker.com/poll2331269x9ae447d5-67

duplicate filter

The duplicate filter page has its old 'file domain' button swapped for a full file search context. This allows you to see 'potential pair' counts, show some random pairs, and launch the full duplicate filter on just a subset of your dupes! For instance, you might want only to filter from a certain creator, or only on very small jpgs. You can also check a box to set whether both of the pair must match your search or only one.

The duplicate page has a significant relayout as well. It now has two tabs–one for the maintenance/search side, and one for the actual filtering–and shows its data in a clearer and less laggy way. The new search data is saved with your page, and on a per-page basis, so if you like you can set up several duplicate processing pages for your different queues.

This system works faster and better than I had expected. Please check it out and let me know what you think. It is usually fairly fast unless you give it a very slow search to work with.

One bug I just noticed with my IRL client is if you give it a specific search with more than 10,000 files (like for me system:archive), the duplicate count will be too low and will change slightly on refreshes. This is because of the 'implicit' system:limit on all queries, which caps search results to 10k files (or whatever you have it set to in the options), and causes the duplicate filter to instead be sampling the potential search domain on each query. This is not true in this case for system:everything, which here uses a search optimisation to get the full count. So, I suggest you throw in a <10k tag for this week, and I will see if I can tackle this problem for 350.

I will update the help for this once I know the ui is settled.

manage tags

The manage tags dialog has a new 'siblings and parents' button that auto-hard-replaces siblings and adds missing parents! It even works on multiple file selections, so if you have a bunch you want to fix, you can just ctrl+a and hit the button and it will sort it out for each file. It gives you a little summary yes/no dialog before it fires just to make sure everything looks sensible.

Beyond some additional sibling/parent code cleanup on that dialog, there is also a new 'cog' option to stop triggering 'remove' actions if you 'enter' a tag that already exists. If you turn this on, only a double-click or delete-key press on the list will make for a 'remove' result.

the rest

I spent some time cleaning up file import and thumbnail generation, particularly for videos. Videos have improved metadata parsing, and thumbnail generation should be much faster. A variety of unusual webms should now be importable and have correct frame counts. Thumbnail right-click->regen commands also work faster and update thumbnail metadata more cleanly and reliably.

If you would like pages to always focus their 'input' boxes when you switch to them, there is now an option for this under the 'gui pages' options panel.

full list

- duplicate filter:

- the duplicate filter page now has a full-on real-deal file search object to narrow down the duplicate filter, potential duplicate count, and 'show some random dupes' search domains! it also has a 'both files match' checkbox that determines if one of both files of the potential pairs should match the search!

- the duplicate filter page has multiple layout changes as a result:

- the main management area is now split into two pages–'preparation', for doing maintenance and discovery work, and 'filtering', for actioning the potential dupe pairs found

- the 'filtering' page will select by default, but if 'preparation' needs work, its name will become 'preparation (needs work)'

- the 'filtering' page now has file search ui and the 'both files' checkbox instead of the file domain button. this search data is saved on a per-page basis

- the two pages' status texts are now updated on separate calls which have been rewritten to be asynchronous (with 'updating…' text while they work). both now have explicit refresh buttons to force them to update

- the additional non-unknown pair counts listed on the filter area, which were irrelevant to filtering and sometimes confusing, are now gone. it only lists the 'unknown' pair number

- the duplicate filter page's help button no longer has the awful 'simple help' entry. the full html help will get a pass in the coming weeks to reflect the new search changes

- the duplicate file db code received significant refactoring and improvement to support searching the potential dupe space while cross-referencing the new file search context (and still falling back to the fast code when the search is just blank/system:everything)

- misc duplicate file db code cleanup and refactoring

- while in advanced mode, you can no longer select 'all known files' file domain for an export folder (and now the duplicate filter page) search context

- making a file delete action in the duplicate filter (by hitting delete key or the button on the top hover window, which both trigger a dialog asking to delete one or both) now auto-skips the current pair

- .

- manage tags:

- the manage tags has a new 'siblings and parents' button that will auto-replace incorrect siblings and auto-add missing parents! it works on multi-file selections as well! it gives you a summary yes/no dialog before it fires

- the manage tags dialog has a little logic cleanup r.e. siblings and parents and their cog auto-apply options. the auto-application now only applies on add/pend actions

- the manage tags dialog has a new cog button option to not trigger 'remove' actions from an autocomplete dropdown or suggested tag input action when the tag already exists

- .

- the rest:

- gave video metadata parsing another pass–it now detects 'hidden' incorrect framerates due to advanced 'skip frame' codec settings and is more accurate at determining frame count and duration, including some fixed offset calculations that was sometimes adding or discounting a few frames

- manual video frame count, when needed, is now faster and produces better error text

- fixed a critical bug in thumbnail regen that was sometimes potentially looping regen on files with unusual rotation exif information

- significant improvements to how the client file manager handles thumbnail identifier information, saving a great deal of time for file import and thumbnail regeneration code of videos

- fixed an issue where regenerated file metadata was not propagating up to the ui level in real time

- cleaned up some thumbnail cache initialisation code

- the 'generate video thumbs this % in' option is moved from the 'media' to 'thumbnails' options page

- to simplify code, and in prep for the idle-maintenance-rewrite of this system, the database->regen->thumbnails call is now removed

- all three fields of text on serialised pngs now wrap, and they pad a little better as well

- added a new option to the 'gui pages' options page to force input text box focus on page changes

- fixed a small type issue with the server's session cookie code and some new library versions

next week

I've felt increasingly ill this week, so I am going to take a day or two completely off. Otherwise, next week will be catch-up for smaller jobs and bug reports and other messages, which I have fallen behind on. Please have a think about the big jobs poll, and let me know if you have any questions. Thank you for your support!

R: 2 / I: 0 / P: 6 [R] [G] [-]

hydrus is shit nigga just use folders and subfolders

my 12345 says i'm right

R: 104 / I: 9 / P: 6 [R] [G] [-]

Next Big Job Poll Discussion

Ok lads, as I am now finishing up OR search, I am soon going to be free to work on a new 'big job'. I am pleased that I was able to make simple Client API and OR search in much faster iterations than previously. I hope to continue like this, keeping the next big job 8-12 weeks at the most before running a new poll.

The current list is:

Just catch up on small work for a couple of months

Reduce crashes and ui jitter and hanging by improving ui-db async code

Clean up code and add unit tests

Improve tag siblings/parents and tag 'censorship'

Add ways to display files in ways other than thumbnails (like 'details' view in file explorers)

Add text and html support

Add Ugoira support (including optional mp4/webm conversion)

Add CBZ/CBR support (including framework for multi-page format)

Add import any file support (giving it 'unknown' mime but preserving file extension)

Improve 'known urls' searching and management

Explore a prototype for neural net auto-tagging

Add support for playing audio for audio and video files

Add ui for waifu2x and other file converters/processors

Write some ui to allow selecting thumbnails with a dragged bounding box

Add popular/favourite tag cloud controls for better 'browsing' search

Improve the client's local booru (this likely now means a backend migration to the Client API)

Improve duplicate db storage and filter workflow (need this first before alternate files support)

Improve shortcut customisation, including mouse shortcuts

Add ratings import/export, and add 'rating import options' to auto-rate imports

Add more commands to the undo system

Improve display of very large/zoomed files in the media viewer

Set thumbnail border colours on user-editable rating and namespace conditions

Improve hydrus network encryption with client cert management and associated ui

Add tag metadata (private sort order, presentation options, tag description/wiki support)

Improve file lookup scripts and add mass auto-lookup

Add multiple local file services (which will enable true nsfw/sfw partition)

Add an incremental number tagging dialog for thumbnails (for adding page:n etc… to a sequence of files)

Permit custom ordering of thumbnails, through mouse-dragging or otherwise

Allow user to have multiple open split tab columns or separate windows with one or more pages

Improve rating workflow by providing score representatives to compare with

Add file modified/creation timestamp searching and sorting

Write an URL Repository so clients can share known url mappings

Add animated thumbnails for videos (animating on mouseover)

Allow multiple custom 'open externally'-style file launch commands for files

Add version tracking to downloader system objects and explore remote fetching of updates

Expand file notes system

I will put up this poll with the 349 release post and then select whatever seems to be on top by 351, with the proviso that I will try to discount any non-organic (e.g. botted) votes. You will be allowed to vote for multiple items. I am happy to work on any of it.

Please feel free to suggest new items or ask for longer explanations of any of the above. I will edit the list as new items are agreed on.

R: 0 / I: 0 / P: 6 [R] [G] [-]

Release Tomorrow!

I had a great week. The new search system for the duplicate filter went really well and is ready for everyone to use. I also improved and sped up some video import and thumbnail generation code, and I added a neat button to the manage tags dialog that lets you automatically fix siblings and missing parents.

The release should be as normal tomorrow.

R: 20 / I: 4 / P: 6 [R] [G] [-]

Version 348

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v348/Hydrus.Network.348.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v348/Hydrus.Network.348.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v348/Hydrus.Network.348.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v348/Hydrus.Network.348.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v348.tar.gz

I had a good week. It is mostly small updates and fixes.

all misc this week

After the recent weeks' thumbnail work, I became frustrated with how laggy the client's file system could get under heavy load. This week I have written a new file access locking system that has less latency under heavy simultaneous use and is also safer for certain edge cases. Multiple import queues and thumbnail fading and regular media browsing will all interact with less lag now.

The Client API has several improvements: the /add_tags/add_tags call has its 'hashes' parameter fixed and now does sibling-collapse and parent-expansion on its 'add' tags (which can you turn off if you want with the new 'add_siblings_and_parents' parameter). Also, by default the Client API service (and the client's Local Booru) no longer log their requests (you can turn this back on if you like under manage services) as this was very spammy (100MB+ logs!) and not useful for most users. And if you are an advanced user who needs CORS pre-flight for an API interface you are making, please check the new CORS checkbox also under manage services. It defaults to off. This CORS stuff is new to me, and I have made very basic responses–please let me know if it does not do enough for you.

I also wrote a first version of a manager to deal with bitmaps (which are used to draw various custom things to screen) in a centralised, safer, and more efficient way. Some non-Windows clients should now be a bit more stable, and some heavy-load animations may be just slightly smoother (as discarded bmps are now recycled). Let me know if you run into any trouble with this (or if your memory use spikes unreasonably). I expect I will do a bit more work here next week.

An issue with the yiff.party downloader not fetching certain post attachments (they got 'ignored' status previously) should be fixed. You will have to re-search yiff.party again to get the correct URLs (and not just retry any 'ignored' you have atm).

full list

* wrote some OR search help to 'getting started with tags' help page

* wrote a new multi-reader, single-writer lock object for the client file manager, along with some unit tests for it.

* updated the file and thumbnail access and regen and maintenance code to use the new lock. various access is now faster when available and overall safer. there is still work to do here

* adjusted file import to be less aggressive about locking, which should reduce some file/thumbnail access lag during heavy imports

* the thumbnail space estimate in the migrate database dialog is now more adaptive to the new more flexible thumbnail size system and specificies better that it is an estimate

* the client api's /add_tags/add_tags call now collapses siblings and expands parents on an add/pend call. this can be turned off by setting the new optional parameter 'add_siblings_and_parents' to false. the help is updated regarding this and the client api version is incremented to 6

* fixed the client api's /add_tags/add_tags call for the 'hashes' parameter, which was failing to parse, and added an accidentally missing unit test to check this in future

* the client local services (the booru and client api) now have two new options under their 'manage services' panel: 'support CORS', which turns on cross-orogin support (which is experimental for now, so defaults to False), and 'logs requests', which controls whether your log will be spammed with request reports (this also defaults to False), which should clear up some 100MB+ log hassle for people using the Hydrus Companion browser add-on

* hydrus services now respond correctly (albeit sparsely) to OPTIONS requests, and if CORS is enabled, to CORS OPTIONS requests. there are unit tests for this that seem to work ok, but I think we'll need to verify it irl

* finished a first version bitmap manager to handle all wx bitmap creation and destruction, including recycling mid-steps

* updated all simple wx bitmap creation and destruction calls across the client to use the new bitmap manager, improving stability and saving some CPU

* fixed some incorrect button alignment flags that were causing problems for clients set to assert check these values

* added a new yiff.party file url class to the defaults that matches a new file attachment format

* updated the 'url' content parser so if a parsed url is in the form 'summary_text url', as some booru source fields sometimes specify, the preceding summary text is removed, cleaning up the resultant url

* silenced an old server connection lost error that was needlessly loud

* silenced the client network engine from additionally log-printing SizeException errors when a downloading file (usually gif) exceeds file import options rules

* improved misc window destruction code

* updated supported mime list in 'getting started with files' help and website index

* misc cleanup

next week

Next week is an 'ongoing long-term job' week. I would like to (finally) add a file search object to the duplicate filter, which will allow you to restrict a series of potential dupes to a certain tag, or only archived files, or whatever else you would like. While I am in that code, I will also see if I can do some easy duplicate system de-jank work overall.

The first version of OR search is now complete. If you missed the boat, there is some official help for it now here: https://hydrusnetwork.github.io/hydrus/help/getting_started_tags.html A thread to discuss what 'big job' I might work on next is here >>12152 . I will catch up on this thread next week and make the poll alongside the v349 release thread next Wednesday.

R: 23 / I: 3 / P: 6 [R] [G] [-]

Version 347

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v347/Hydrus.Network.347.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v347/Hydrus.Network.347.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v347/Hydrus.Network.347.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v347/Hydrus.Network.347.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v347.tar.gz

I had a good week. OR search is essentially finished, and I cleaned up and fixed a variety of other things.

A new 'big thing to work on next' poll will be going up soon. If you are interested, please check out the discussion thread here:

>>12152

or search

As previously discussed, I have moved OR predicate construction to the standard dropdown list below the tag input. It now appears as the top result, where you can hit enter on it to submit it as-is. Also, while under construction, a 'cancel' button and 'rewind' button (to remove the most recent OR-term added) will appear on the same panel. You can also hit Esc to cancel a currently under-construction OR predicate.

As a reminder, hold shift when you enter a tag to start an OR chain. Further shift+enter events will append new tags to the chain, and a bare enter will cap it off. I will write out some proper help for this.

OR search is basically finished as a v1.0 now. I still have some last tidy-up jobs to do, but I am overall happy with it.

the rest

I built on the past weeks' thumbnail experiments and have written a two-stage thumbnail rendering system that gets thumbs on screen faster (even if they are the wrong size, so will look fuzzy), and regenerates any needed clearer versions in the background and replaces them in-place on screen over the following seconds. It is much smoother and faster than before, and it is pretty neat to see a fuzzy thumb suddenly fade into a clearer version, but I still have a little work to do here.

Now, when you trash a file, a context-appropriate 'deletion reason', such as 'Deleted from Media Page.' will be saved. These statements are mostly trivial, but duplicate filter actions will specify a bit more about the duplicate processing action type. This text will be recovered in an import status window for 'deleted' status results, just as a help if you want to investigate closer (e.g. perhaps you are not sure why a particular file failed to import, but then you see the reason is you already decided you have a better duplicate version of it). Any files deleted before this system will just give "Unknown deletion reason."

Adding OR search caused a couple of search flaws: bare system:rating searches were delivering since-deleted files, and some searches with combinations of OR predicates with regular tags were delivering subsets of the real results. I believe I have fixed both of these, and now many previously slow OR searches should run quite a bit faster, especially when accompanied by with non-OR predicates.

I gave export folders a pass and fixed several bugs and inefficiencies, particularly for 'synchronised' folders that produce subdirectories from their filenames, which were often deleting those subdirectories. Also, and export folder or manual export event that attempts to produce a file path above the base export directory (e.g. if the generated filename begins with ..\ or ../) will now fail with some error text to explain what happened. If you use export folders a lot, particularly 'synchonised' ones, please let me know if you still get any unusual behaviour.

full list

- or search:

- under construction OR predicates now present at the top of the regular tag results list, prepended with 'OR: ', and skipping default selection

- this new OR line is enter-able, which will present it as-is, rather than adding new preds

- hitting escape on a 'search' tag input box that is empty but has an under construction or predicate will cancel the or pred

- hitting escape on a 'search' tag input box otherwise should more reliably kill its focus when the dropdown is a float window

- improved OR search efficiency significantly with dynamic OR search triggering based on other search predicates. OR searches including negated '-tag' components should be massively faster when paired with non-OR tag or file search predicates

- I believe I fixed a search issue that would sometimes return insufficient results when OR preds are mixed with certain other combinations of tags

- improved reliability of some thumbnail refresh calls

- cleaned up a bunch of OR handling ui code

- .

- the rest:

- after previous weeks' experiments, wrote new double-layer thumbnail loading system–now too-small thumbs will quckly scale up fuzzily straight to screen, and then in the coming seconds, the nice regenerated full-size thumb will be made and drawn in place as ready. it presents much faster and looks better, but there is some cleanup to do here that I will tackle next week

- all local file trashing events now record a context-appropriate deletion statement such as "Deleted from Media Viewer." this value is recovered in 'deleted' import status 'notes'. You will mostly see 'Unknown deletion reason.', for files deleted before this new system, but it will populate with appropriate info over time

- fixed a search optimisation that was not cross-referencing with file domain, meaning for instance that bare system:rating calls were returning since-deleted files

- upnp management window now uses new listctrl

- cleaned up some old custom page-naming code

- added a 'data' debug call to clear out all cached thumbnails and force an instant ui thumb reload

- fixed the trash bmp misalignment, ha ha

- removed e-hentai login script from the defaults, since this testing script is not appropriate for new users

- dejanked some media viewer video transitions by cleaning up animation bar rendering and smoothing out video buffer initialisation

- cleaned out some surplus subprocess wait calls that were hanging some systems on various 'open externally' calls

- fixed multiple syncing problems with 'synchronise' export folders that produce files with subdirectories. subdirectory structures should now be synced correctly and empty folders deleted

- export folders that collapse multiple file results to the same duplicated name should, after the next run, do less overwriting to this same name

- if an export folder or the regular export dialog makes a file destination path that is above the chosen directory (e.g. if the path starts with ../ or ..\), the export job will error out with an explanation

- big manual file exports _should_ be politer to the ui and cause fewer hangs

- doing page tab drag and drops may have less post-drop ui jank on linux, continued feedback would be appreciated

- moved 'reason' handling for all content updates to its own area, which neatens many content update data handling issues

- fixed petitioning a tag via a shortcut, which had bad reason handling

- fixed an issue with committing pending ipfs items that was overchecking service permissions

- fixed some remaining bad wx code in the unit tests

- misc file status reporting cleanup

next week

I'll tidy up some last OR search stuff and clear out some small jobs. I would like to reduce some lag when the client file manager has a lot of competing access (e.g. when lots of new thumbnails need to be generated), and I would also like to improve some Linux stability with some unified bitmap management.

R: 54 / I: 3 / P: 6 [R] [G] [-]

Hydrus software optimization thread

ITT: create proposals for making Hydrus more optimized.

Proposal: Why can't Hydrus switch to MariaDB?

If it is faster, then it should be better. The only trouble is having the need to rewrite the queries, which from an SQL standpoint should be a non-issue, right?

List of Databases with Open Source License and Open Source APIs:

SQLite - Currently used in Hydrus, has minimal features

MySQL - A more well-rounded SQL Database with user management

PostgreSQL - An SQL with complex features with less performance

MariaDB - SQL/NoSQL database with heavy optimizations

ElasticSearch - A literal search engine instead of a normal Database

Teradata - IDK

https://www.digitalocean.com/community/tutorials/sqlite-vs-mysql-vs-postgresql-a-comparison-of-relational-database-management-systems

https://www.infoworld.com/article/2611812/mysql/mysql-face-off--mysql-or-mariadb-.html

R: 0 / I: 0 / P: 6 [R] [G] [-]

Release Tomorrow!

I had a good week, mostly working on small jobs and fixes. File and thumbnail access should be less laggy, some drawing code should be more resource efficient and stable, and the client api has a bit of work as well.

The release should be as normal tomorrow.

R: 26 / I: 0 / P: 6 [R] [G] [-]

Version 346

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v346/Hydrus.Network.346.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v346/Hydrus.Network.346.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v346/Hydrus.Network.346.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v346/Hydrus.Network.346.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v346.tar.gz

I had a great week. There's a bunch of fixes and thumbnail improvements, a more powerful manage urls dialog, and a prototype OR search to try out.

thumbnails

Last week's big thumbnail change seems to have gone fairly well overall. There do not seem to be any huge errors anywhere, and the simpler file system and code is great, but I was annoyed by how slow certain regeneration scenarios turned out to be, and how some thumbnail resizes were a bit blurry.

To address this, this week I have cleaned and improved a bunch of thumbnail loading code. It was much nicer to do now I only have one thumbnail file to juggle. Thumbnails should now be a less blurry when they scale up, and they should generally load a bit faster whenever a lot of regenerations need to be done (you should notice CPU-expensive video thumbs load last of all, while easy ones like pdfs and pictures should come in first).

I was also annoyed that even small changes in desired thumbnail size would force a complete regen, which was particularly laggy when scaling up. So, to smooth out the loading time when you have recently changed the desired thumbnail resolution, I have added an experimental regeneration throttle. When the difference between what is available and what is desired is just a few pixels (and so a scale will not be too fuzzy), rather than regenerating from source, the client will now mostly just scale up or down what it already has. When the difference is great, it is more worth spending the CPU on a full regeneration to get a clear thumbnail, so it will regenerate more often. It still eventually regenerates everything, but when the difference is slight, the real workload will be spread over several loads. For some example numbers, if the thumb on your disk is 120x120 and your options want 125x125, the client will expensively regenerate from source only about 29% of the time and scale up the original 120x120 about 71% of the time. But 120x120 vs a desired 250x250 will regenerate/rescale about 84%/12% of the time. This is a big experiment, but I am generally pleased with it: it does speed up thumbnail load in many situations that were previously needlessly slow, and the relative fuzziness is generally not a noticeable problem. I hacked the percentage ratios out of some statistical polling that I can definitely tune in future, so please let me know if your situation produces too many very fuzzy thumbs.

Also, if you are in 'advanced mode', the thumbnail right-click menu now has more 'regenerate' options: as well as doing a complete file reparse and thumb regen, you can now choose to just regenerate thumbnails or only regenerate thumbs if their size is wrong. If you are an advanced user, please feel free to play with these new commands, particularly if you notice fuzzy thumbs from the throttle above, and let me know how it goes.

I expect to do more work on thumbs in the coming weeks. As well as tuning the new regeneration throttle, I would like to expand the new database-level thumbnail regeneration modes to work during maintenance time. I have long planned a 'file reparse' db maintenance mode so I can finally fix all the ancient mistaken animation frame counts and mime mismatches like apngs/pngs and mkvs/webms, and now this same maintenance command can also just do thumbs, I would love to let users say 'please regen all thumbs that are the wrong size in the background during idle time, no more than 20mins/day'. There's no reason why this stuff should be only on command for advanced users, so now I need to build the maintenance structure for it.

I'd also like to extend the current thumbnail location override under migrate database to work just like the files, so you can balance thumbnails over multiple weighted locations.

Linux and OS X users: You may not have received an error, but it is decently likely that last week's 'rxx' folder delete did not work for you. This was due to a permissions-setting mistake on my end. I have fixed this permissions code, and on this 346 update, if you still have some rxx folders in your db, the client will attempt to re-do the delete. If it wants to do this, it will give you another popup message as you update, informing you of the situation before it goes ahead.

or search

Several things came together this week, and I managed to get an early prototype of OR search working completely. I am keen to get users' thoughts on how to finish it off.

To start an OR search, hold down shift when you enter a tag in a search page. Rather than adding this tag to a search, it will now place it in a new box just above the input box. Further shift-enter actions will increase the OR 'chain', making 'tag A OR tag B OR tag c', and a final non-shift enter will cap the chain and send the OR search predicate up to the main search. It works! You can even add system predicates!

However, this entry method is not finished. Have a play with it and you will see what I mean. There is no neat way to cap off an OR chain early (i.e. without typing anything new: if you hit a bare enter on it, you'll get system:everything or system:inbox on the end), nor to 'go back one' if you put the wrong tag in. Also, system predicates that need additional panel infomation before 'going through' do not hold on to the shift status. Beyond the bugs, I am interested in what you would like for the workflow here–would you rather the OR-chain under creation be inserted as the top item in the list below, rather than in the new box above? Should it be highlighted as under construction somehow? Should it have a dangling OR while under construction, such as 'blue eyes OR green eyes OR' rather than 'blue eyes OR green eyes'? How should it be cancelled, or undone a step? What would be the best way to say 'ok, I have A OR B OR C entered, and I don't have a D to add, so I want to enter what I have now'?

the rest

I made two more stupid errors last week–one that draws white frames over animations on certain media update events, and one that occaisonally set the 'include current tags' value on search pages to 'false' on reloads. I have fixed both of these issues, and as a one-time for this update, all 'include current tags' values are forced back to true. I am sorry for the inconvenience here. The white frame stuff drove me nuts all week.

Beyond that, I extended the 'manage urls' dialog to support multiple thumbnail selections, which allows for easy mass-deletion of bad URLs. The URL list on this dialog also now supports multi-selection and accepts the delete key. It also supports adding URLs to multiple files, but when opened on multiple files it presents a little red-text warning to remind you that adding a URL to multiple files only makes sense for gallery URLs. I expect to do a little more work on this in future, cleaning the workflow a little more and detecting the url class of urls you attempt to add to multiple files, specifically testing what is and is not appropriate to add. In any case, like everything else this week, please give it ago and let me know how it works for you!

Also, the 'move files now' button on database->migrate database now lets you limit the max time it will run to 10/30/60 minutes! This is a nice stopgap if you, say, have 600,000 thumbnails to move and don't want to do it all at once.

full list

- or search:

- extended the search predicate object to handle more OR stuff

- extended the tag list to handle list objects that have multiple colours

- extended the new OR search predicate to report multiple text-snippet-and-colour pairs based on sub-predicates

- extended tag search input to handle prototype OR predicate creation–hold shift when you enter the tag, and it'll start an OR chain. shift-enter continues the chain, enter alone completes it

- fleshed out the predicate unit tests to cover more of this

- wrote unit tests for OR search predicates. it seems good!

- improved some search logic to apply system preds better in certain edge cases and spend less CPU on OR-search-only searches

- .

- thumbnails:

- thumbnails will now queue for load in a more intelligent order based on estimated difficulty to regenerate, which will tend to put more thumbs on screen faster

- the decision to regenerate a thumbnail from source is now tempered by how different the current thumbnail is from what is desired–the more similar the two sizes, the more (randomly) likely the client will decide to just use the current (resized) this time. this smooths out change-lag while limiting the number of really fuzzy thumbs you get. feedback on how this works IRL would be appreciated–it uses some voodoo distribution polling to figure it out, which I can definitely tweak

- improved visual quality of thumbnail scale-up optimisations

- fixed an issue where a multipage thumbnail grid would incorrectly recalculate the new virtual height after a thumbnail size change event, leading to a bit of invalid extra scrollspace (with noclip rendering errors) at the bottom

- the thumbnail right-click menu's reparse files entry is now extended to a new 'regenerate' submenu with three options: reparse file and regen thumbs (the old action), force regen thumbs, and regen thumbs if wrong size!

- the new 'regen if thumbs wrong size' action sends how many thumbs needed resize up to the popup window, as well

- moved some old thumbnail regen code responsibility out of the db and into the files manager

- cleaned out some old redundant file/thumbnail code

- cleaned and refactored a bunch of general image handling and resizing code

- .

- the rest:

- fixed some bad serialisation code that was making file search objects set their 'include current tags' value to false/true on interleaving loads. on this update, all 'include current tags' values are blanket reset to true

- fixed an issue that was drawing animation canvases pure white on various media update events

- extended manage urls dialog to support multiple files when launched from a selection of thumbnails. there is a warning in this case, noting that only gallery-style urls are appropriate to be added to multiple files

- manage urls dialog now supports multiple selections, including shift-select, and accepts delete key presses for easy mass deletion

- when you ask the database migration dialog to move some files, it now pops up a confirmation dialog that also asks if you would like to limit the max time for the job as 10, 30, or 60 minutes

- improved file permission setting code across the program to be more sensible for non-Windows

- if you are a non-Windows user and were hit with directory permission problems last week on the thumbnail update–which resulted in the rxx directories not being deleted–the update this week will attempt to do the delete again, this time correcting the now missing execute permission bit. if it finds outstanding rxx directories to delete, it will give a popup beforehand summarising the situation and giving you a chance to bail out

- fixed yet another problem that was stopping client api url requests from finding the correct page by name

- when a client api url request includes fixed tags, these tags should now propagate in all scenarios where the single url produces multiple files

- updated sqlite dll and console for windows

- misc fixes and cleanup

next week

Next week is a 'cleanup' week. I'll keep chipping away at the listctrl rewrite and other boring jobs, and I'll see if I can finish up the OR search entry workflow in a sane way.

Thanks everyone!

R: 2 / I: 0 / P: 6 [R] [G] [-]

Release Tomorrow!

I had a good week. I fixed some bugs, extended thumbnail resizing into a much smoother two-stage system, and fleshed out the OR search workflow as I had hoped.

The release should be as normal tomorrow.

R: 12 / I: 2 / P: 6 [R] [G] [-]

Pixiv+Hrdrus

I was thinking since Pixiv has such a weird, complicated, and sometimes awfully designed site, us Pixiv bros should help each other out on figuring out ways to make it easier to use with Hrdrus. Post any tips, scripts, regexes, setups, etc.

R: 26 / I: 4 / P: 7 [R] [G] [-]

Version 345

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v345/Hydrus.Network.345.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v345/Hydrus.Network.345.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v345/Hydrus.Network.345.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v345/Hydrus.Network.345.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v345.tar.gz

I had a good week. The big thumbnail update is done, which means for an important update. Users with smaller databases have less to think about, but anyone with 1,000,000+ files should definitely read all of the following:

thumbnails

This is a great week to make a backup before you update!

Until now, the hydrus client has used two simple thumbnails–one 'full-size', one 'resized'–for each file. Hardware has moved on since then, and so has my code, so it is now feasible to have a single smarter thumbnail that will resize itself on demand. The work for this was done this week. This makes for a simpler and less wasteful storage system and more flexible thumbs all around.

The main thing is that on this update, the client will delete the surplus thumbnails. You will get a popup before it happens to remind you. Specifically, it will delete the 256 'rxx' 'resized' subfolders in your database's file store. If you have moved these folders through the migrate database dialog, this is no problem–the client will delete them wherever they are. Depending on your collection and platform, it could take a few seconds or several minutes. Your new thumbnails may load just slightly slower the first time, but this will return to normal in time.

The best news is that the old 200x200 limit is gone! You can now set your thumbnails' size under options->thumbnails up to 2048x2048! I am now running 360x240 on my 4k machine and like it a lot.

Note that if you previously moved your 'resized' thumbs to an SSD but not your old 'full-size' thumbs, you will want to recheck the migrate database dialog at some point to make sure you move your newly active thumbnail folders to fast storage. This is likely to be a big job, so plan for it.

The migrate database help regarding all this has been updated:

https://hydrusnetwork.github.io/hydrus/help/database_migration.html

Also, be prepared for your next backup run to have significant additional delete overhead as all your backup's 'rxx' folders will be deleted as well. If you have fewer than, say, 250,000 files, I do not think you need to think much of this, but if you have many more, you might like to manually delete these folders from your backup yourself right before your next run. There are 256 of these folders, named in hexadecimal from r00 to rff. Preemptively deleting them will save you backup-scanning time and keep your recycle bin sane (if your backup program deletes to recycle bin). Make sure you do a permanent delete (Shift+Delete on Windows) to skip over the recycle bin yourself.

I tested the new system thoroughly, but as always, some unusual scenario may present problems. The system is more complicated than 'if the thumb is the wrong size, resize it and save it back to disk', so some edge cases may throw an error. Please report any bugs, and I'll try to have them fixed for 346.

full list

- or search:

- set out a plan to achieve some simple conjunctive normal form (e.g. (blue eyes OR green eyes) AND (blonde hair OR red hair)) OR search support

- started work on the object extension and search code to support this search in a very basic (and likely inefficient-for-some-scenarios) way–we'll work on this as we discover the most common inefficiencies

- .

- thumbnails:

- the client no longer uses both 'master' and 'resized' thumbnails–it uses a single, smarter thumbnail

- only the 'txx' thumbnail directories (formerly referred to as full-size) are now used, and the thumbnails inside will regenerate and scale themselves as needed on demand (and will be careful to not save changes to disk when when their source file is non-local)

- the old 'rxx' 'resized' thumbnail directories are no longer referred to anywhere in the code or ui

- the old 'rxx' thumbnails directories will be permanently (i.e. no recycle bin) deleted on update. this is a big job, and you will be prompted on update before it happens

- if you have migrated your db to put 'resized' thumbs on an SSD but not the formerly 'full-size', you will want to recheck the 'migrate database' dialog once you have booted and set a new thumbnail override to move the txx directories over

- due to the smarter thumbnail, 200x200 is no longer the hard limit for hydrus thumbnails! you can now set up to 2048x2048

- all file storage location information is now stored directly in the client db (rather than the options object), which should make for more easily export/importable options in future and improve manual fixing as needed

- added more thumbnail-resizing related popup spam to file report mode

- fixed a windows-only issue that was making the migrate db dialog close after a file move event concluded

- updated database migration help for new concepts and ui

- cleaned up some misc storage code

- .

- the rest:

- fixed a problem in the client api with fetching file identifiers from file_ids

- fleshed out 'help my db is broke.txt' with more specific clone recovery examples

- fixed import support for a variety of single-frame music webms

- fixed an edge-case preview viewer initialisation bug that was trying to draw the canvas before any media was set

- network report mode now states url classes of urls about to be parsed

- misc small fixes and cleanup

next week

I got started on OR search this week. I have a decent plan and feel good. I hope to have some sort of prototype ready for 347. Otherwise I will do some small jobs. I may look into adding multiple-files 'known urls' management.

R: 0 / I: 0 / P: 7 [R] [G] [-]

Release Tomorrow!

I had a great week. With the new simpler thumbnails, I improved how thumbnails load and added some new thumbnail maintenance actions. I also extended the 'manage urls' dialog to work for multiple files and got the very first simple version of OR search working and ready for user testing!

I also fixed the unusual 'include current tags' and white-frame animation bugs.

The release should be as normal tomorrow.

R: 22 / I: 2 / P: 7 [R] [G] [-]

Version 344

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v344/Hydrus.Network.344.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v344/Hydrus.Network.344.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v344/Hydrus.Network.344.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v344/Hydrus.Network.344.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v344.tar.gz

I had a mixed week, but I got some good hydrus work done. It is basically all misc this week.

highlights

The Client API v1.0 got its last polish this week. I fixed some bugs and added a new parameter to control page selection to the /add_urls/add_url command. Future Client API work will be entirely in regular weekly work. I'd like to add wildcard and namespace tag searching, more system predicates, autocomplete tag searches, optional https to encrypt communication outside a LAN, cookie.txt import, and I am sure many other things. The basic bones are decent now though–I now need only hang new things off it.

Animation scanbars (the draggable area just below a gif or webm in the media viewer) now have millisecond-precise x/y progress timestamps! This was fun to do, and it worked out pretty neat–it even works for variable framerate gifs.

I have added barebones .psd (photoshop document) file import support! This is a complicated format that supports image layers and so on, so I cannot provide any native rendering, but it will parse width and height. It presents as 'application/x-photoshop'. PSB files (which support huge 300k width/height) are also recognised, if you happen to have any. Hydrus will treat them as .psd as well. This has been asked for several times, so please let me know if you find useful workflows here. Note that hydrus assumes its file contents will never change, so do not try to load a hydrus-stored .psd in Photoshop and then save any changes back! This is best for 'finished' psds that you are effectively archiving.

There is a new 'open_known_url' shortcut under the 'media' shortcut set. It should work on a focused thumbnail or the media viewer: on activation, if the file has one 'recognised' URL (the ones listed in the top-right of the media viewer), it will launch in your browser, and if it has multiple, you will get a list of all known URLs you can quickly navigate with arrow keys and enter to choose which you want to see.

full list

- final v1.0 client api polish:

- added optional 'show_destination_page' arg to '/add_urls/add_url', defaulting to False, to control whether an URL-add will select (i.e. jump to) the destination page in the ui. this changes the default behaviour for this command

- simplified the routine that finds or creates a watcher or url import page and fixed a bug in the api that was not creating new pages when destination_page_name was specified

- some misc cleanup

- fixed fetching file_metadata by hashes

- fixed the client api help regarding file_metadata response example tags

- client api version is now 5

- .

- the rest:

- psd support added! because of this format's potential multi-layer complexity, it will not render natively, but width and height are parsed. it is treated as 'application/x-photoshop'. PSB is also recognised and treated as psd

- added a 'open_known_url' shortcut to the 'media' shortcut set that lets you quickly open URLs for files. if there is one recognised known url, it will be launched, and if there are multiple, a list with all known urls will appear to select which one you want

- animation scanbars now show an x/y current timestamp! it includes millisecond timings and even works for variable framerate gifs. whether to show a second/shadow caret for timestamp position on variable frame rate is a new discussion to have

- fixed an issue where animations would sometimes not resume animation for several seconds after a big scanbar drag

- when the thumbnail manager cannot produce a thumbnail due to a storage error (like a missing file), it now only puts up a single, more informative error popup on the first problem. subsequent errors are printed silently to the log. (these errors tend to come in en masse, so this cuts down on spam and error-related ui lag that was making loading a bad session difficult)

- improved error reporting when an upload pending command would fail due to service non-functionality–it should now give a popup with error info imediately, rather than obscured through the login system

- added temp_dir parameter to the client and server that will override which temporary directory the program will use

- cleaned up how no_daemons and no_wal mode are handled internally

- no_wal mode now has to be called from the command parameter, the no_wal file hack in the db directory no longer works

- missing ffmpeg errors now prompt the user to check if it is installed

- searching for numerical ratings should now work for files that were rated when the service had a different number of stars (ratings now searches in 'bands' rather than exact values)

- reduced the min height of the new import files frame's list

- doubled the decompression bomb test to permit files up to ~179 megapixel, we'll see how it goes

- misc cleanup

next week

I have some IRL stuff going on atm. I hope it will not affect my hydrus time, but it may steal a day or two from me. If things line up well, the situation should be resolved within a week or two.

With Client API done for real now, I will move completely to OR search. Otherwise, next week will be an 'ongoing work' week. I hope to pull the trigger on eliminating hydrus's double-thumbnail system (i.e. moving to just one thumb per file). The db update next week will thus likely be a big deal, maybe several minutes' of deleting for users with larger clients.

R: 0 / I: 0 / P: 7 [R] [G] [-]

Release Tomorrow!

I had a good week. The big thumbnail improvement is done, making for smarter thumbnail access and storage across the program. Also the max possible size of thumbs, previously 200x200, is now uncapped! The update tomorrow will take a little while, as the now-surplus half of your thumbnails will have to be deleted.

The release should be as normal tomorrow.

R: 7 / I: 1 / P: 7 [R] [G] [-]

Release Tomorrow!

I had a mixed week, but I got some good hydrus work done. I added some neat x/y timestamps to the animation scanbar, made a way to launch file urls with a shortcut, and added barebones .psd import support.

The release should be as normal tomorrow, but perhaps a little late if some IRL stuff comes up.

R: 32 / I: 6 / P: 7 [R] [G] [-]

Version 343

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v343/Hydrus.Network.343.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v343/Hydrus.Network.343.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v343/Hydrus.Network.343.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v343/Hydrus.Network.343.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v343.tar.gz

I had an ok week. The Client API v1.0 is essentially finished, and I did some code cleanup as well.

client api

The Client API now has full 'file search' capability, including fetching file metadata like size and resolution and tags, and also file and thumbnail fetch. As before, the help for these new commands is here:

https://hydrusnetwork.github.io/hydrus/help/client_api.html

But for some tiny last cleanup jobs, the Client API v1.0 is now complete. From now on, I will fit adding additional commands to the API in normal weekly 'small job' work.

I am overall pleased with how this has gone. If you have been waiting for a 'working' version of the API to play with, now is the time. For regular users, there is a link to a browser add-on in that help file, and it seems a phone app is also in the works.

the rest

The media viewer is now smarter about recycling its internal windows, so browsing between video and static image types (in either direction) is now faster and smoother. If you often get the one-frame 'tiny grey square' when transitioning between video and static images, this should now be eliminated.

More types of tiff are now supported, and I fixed an issue that was mis-detecting soundless webms as mkvs.

The list of file paths in the manual file import window is now sortable, and it will preserve this sort order in the subsequent file import or path tagging dialog (useful if you wish to set up sequential number tagging, or you would like for some reason to import by largest file first). It also has a new '#' column to let you return to the original parse sort order.

full list

- client api:

- fixed an int/str type mismatch issue with service_names_to_actions_to_tags in /add_tags/add_tags in the client api that meant that argument was not working

- fixed up some last /get_files/search_files stuff

- added /get_files/file_metadata

- added /get_files/file

- added /get_files/thumbnail

- added help and unit tests to reflect the above

- updated client api version to 4

- .

- the rest:

- the list of paths in the manual file import dialog is now sortable. this order will be preserved in regular and 'add tags' ok events for this dialog. it has a new '#' column so you can return to 'parse' order if desired

- animation and static image windows in the media viewer canvas are now recycled through media type transitions, making for slightly smoother browsing between mixed media

- increased aggression of media viewer image prefetch

- added support for 'MM' Tiffs

- fixed webm mime parsing for webms with no audio (these were falling back to mkv)

- improved the error reports when a serialised png fails to import

- the mass-open-urls popup is now pausable as well as cancellable

- fixed several recently broken ui unit tests

- misc old code cleanup

- some misc test controller/constant refactoring

next week

I'll finish up the last Client API work and move on to tag OR searching in the client as the new 'big job'. My hope is this will not be a gigantic project, but we'll see how it shakes out. Beyond that, I'd like to fit in some small jobs–maybe some time units on the animation scanbar, maybe some new shortcuts.

This week got away from me a bit, so I also have a ton of bug reports and other messages to catch up on.

R: 0 / I: 0 / P: 7 [R] [G] [-]

Release Tomorrow!

I had an ok week. The Client API v1.0 is just about finished, and as hoped I did some long-overdue code cleaning. Also are some tiff/webm detection fixes and slightly smoother media viewer browsing.

The release should be as normal tomorrow.

R: 30 / I: 5 / P: 7 [R] [G] [-]

Version 342

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v342/Hydrus.Network.342.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v342/Hydrus.Network.342.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v342/Hydrus.Network.342.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v342/Hydrus.Network.342.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v342.tar.gz

I had a good week. There are several new ui features and bug fixes, and webp and tiff now have basic support.

tiff and webp

Tiff files are now supported! Tiff is an old format typically used for photographic and scientific uses (think 12,000x12,000px images of the moon). They'll import to hydrus like any other image and will work on the duplicate file system. I tested 24bit and 8bit (monochrome), but could not find a 48bit example–even the easily accessible NASA stuff was all 24bit–so I don't know if high colour-depth tiffs work. If you have an example of one, please send it to me, and if it doesn't work in hydrus, I'll see if I can figure it out. EDIT: Some tiffs are not supported. I know basically what this is and will fix it for next week.

And webp files are now supported! Webp is a newer initiative from Google to replace jpgs and pngs. It has some impressive file size reductions, but hasn't picked up all that much yet. They'll now import to hydrus like any other image and will work on the duplicate file system. WebP also supports image-animations, much like gif, but I haven't written support for this yet. For now, hydrus will assume all webps are static images, and animations will only give you the first frame (just like how apngs used to import when hydrus considered them pngs).

I did not have to do all that much work to get tiff and webp working–I just hooked in my existing image libraries, PIL and OpenCV. If you run from source and have slightly older libraries here, your support may not be complete. I hope to keep working here and try to get animated webps working. If you are enthusiastic about webp, let me know how you get on–they mite b cool for some purposes, and if the library support for saving them is there, I might start using them in hydrus's thumbnail system, including for the long-planned animated thumbnail system.

otherwise all misc this week

Tag autocomplete search should be a bit quicker to update when you temporarily pause on a large search (e.g. if you pause on 'title:' and then type 'gar', the big laggy search for 'title:*' will now be cancelled much faster so the quicker 'title:gar*' can replace it).

Every one of the complicated 'copy' URL thumbnail menu entries is now replicated on the 'open' menu! So, if you select five files that all have twitter URLs, you can hit right-click->known urls->open->these files' twitter tweets to open them all in one go! It will open them one at a time, one per second, so as not to nuke your browser. If asked to open more than a few URLs like this, the client will warn you with a yes/no dialog and throw up a popup that will let you cancel the operation (just in case you accidentally try to open 500 URLs!).

By default, 'system:everything' is now hidden in search contexts with more than 10,000 files! This search predicate is sometimes useful to new users, but experienced users hit it more by accident than not. If you would rather have it anyway, you can force it to appear by hitting the new checkbox under options->default system predicates.

Gallery download pages now have a more informative 'status' column–they'll report gallery fetch status and say 'done!' when all work is complete.

After many attempts and some great help from users, I believe I have finally fixed the Linux 'similar files search with >0 distance' crash. One user and I managed to isolate the program area and figure out a cleaner layout, and it seems to be fixed. If you have had this problem, please give it a hesitant go and let me know what happens.

Also, I cleaned up some video rendering logic and memory management. Videos should render a little smoother and more responsively, and they will clean up after themselves better as well. This may have fixed a long-running but rare all-platforms crash related to launching or closing the media viewer on a playing video.

full list

- added support for webp import. it does not yet support animated webps, which, if the local platform supports, will import like apngs used to: just the first frame

- added support for tiff import. it works ok for 24bit and 8bit (monochrome) tiffs, but I am not sure how well it will do with 48bit

- both webp and tiff should work on the duplicate files system

- improved webm detection to include opus audio (previously, these files were falling back to mkv)

- fixed an issue where unusual formats with duration but no frames or frames but no duration were being sorted and otherwise presented incorrectly

- improved autocomplete job cancelability. this job can now cancel much faster on large jobs, meaning typing searches with large result sets will hit less CPU and return faster on subsequent keystrokes

- _all_ of the complicated 'copy url' commands from the thumbnail right-click->known urls menu are now available on the 'open' submenu! if there is more than one url to open (e.g. 'open all of these files blahbooru post urls' on a selection of 50 files), you will be presented with a yes/no dialog to confirm, and it will open one url in your browser every second (with a cancellable popup if num_urls > 5)

- by default, system:everything is now hidden if its total files is >10k. you can force it to always show under options->default system predicates

- the gallery downloader's list's status column now shows gallery status (deferring to active file status) when appropriate and shows 'done!' when all work is complete

- after working back and forth with a user, I _believe_ the linux similar files >0 distance search crash is finally fixed

- fixed sorting by media views/viewtiming with collections

- a single-selected collection right-click now shows total media views for all files in the collection! you can now see how long you have been viewing an artist!

- fixed an issue that lead to export folders not running on always-on clients as often as they should

- updated the gelbooru 0.2.5 file page parser to pull rating tag from the correct location (previously, it was pulling from what appears to be a site-wide 'mature' browser hint)

- improved memory cleanup stability when animations and other parts of the video rendering pipeline are deleted–this _may_ fix some rare crashes

- increased animation rendering aggression overall and particularly in 'future' of frame buffer

- if a video renderer that is asked to start some way into the video fails to render anything, it will now fall back to trying to render from the beginning. this is slightly hacky atm and leads to out of phase rendering frames, but it is better than an error

- added a '–no_db_temp_files' launch parameter that will force the client or server to return to the recent old behaviour of exclusively using memory for journalling. this is useful if your temp directory is small and/or your available ram is very large. if running in this mode, the client will attempt to check available memory (instead of free space on your temp dir) before performing very large transactions

- with the new lighter-weight update transactions, the client now tests for less free space for journalling before running repository update processing

- added /get_files/search_files to the client api, which does the first half of file searching. it allows tag search (including -tag negation) and system inbox/archive. since the second half, which will fetch file metadata, is not yet in, this can't do anything interesting yet

- updated help and unit tests to support this, client api version is now 3

- some misc refactoring

next week

I got the first half of Client API file search done this week, but it is useless without the other half, so that's the top priority next week.

Also, after thinking about it a bit and talking with some users, I am going to try changing my schedule just a little. I always enjoy stability and cleanup work, and I am happy I found time recently to do some more, but it is often difficult to remember this kind of work with the pressure of a hundred smaller (and often sexier) jobs. With this in mind, I will try having a rotating schedule where I do one week focusing on stability and cleanup (including improving unit tests), one on larger ongoing work (like the recent autocomplete async work and the thumbnail experiment last week), and two on regular smaller jobs (like most of what I did this week). I'll go cleanup, small, ongoing, small. I will still work on bugs and the 'big job' as normal every week, but the rest of my time will be more focused on one thing. This will force me to spend at least some time on each important area every month without turning my week inside out trying to cram a bit of everything in.

Anyway, I'm going to give that a go and see how it feels. I will do cleanup next week. There are a ton of broken unit tests that need catching up to current code, and I'd love to move forward the listctrl object replacement. Both of these are boring as anything and have been put on the back burner, so let's see if I can concentrate next week and move them forward!

R: 46 / I: 3 / P: 7 [R] [G] [-]

Version 341

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v341.tar.gz

I had a great week after being ill at the end of the previous week. There is a bit more than usual in this release because of the extra time. The Client API has some new commands, and there are a variety of fixes and stability improvements.

client api

The Client API now does some fun stuff. If you would like to play with one user's browser extension, which recognises hydrus-compatible URLs–including whether you have the page's file already–and lets you send them straight to hydrus for downloading, please check it out here:

https://gitgud.io/prkc/hydrus-companion

And the Client API help itself is here:

https://hydrusnetwork.github.io/hydrus/help/client_api.html

There is a little info at the top telling you how to turn the API on.

This release brings some more API commands. You can now edit a file's tags, edit a file's URLs, and the 'add_url' command now lets you attach tags for any files coming from the given URL and also lets you determine which page (by page name) the URL will go to (for instance if you have multiple watcher pages working on different check timings for different boards).

the rest

I spent some time this week cleaning things up. Repository update processing uses less memory and now regularly saves its progress on big jobs, reducing transaction bloat, improving error recovery, and giving control back to you much faster when it finishes. Memory use all around should be reduced.

I also replaced several areas of messy UI code, which has resulted in improved stability, particularly for Linux. Some CRITICAL logspam is gone, and I believe I have finally fixed a lingering ui crash on the duplicates processing page (at least for search distance 0–I will still work on >0, which seems to be a db-level problem for linux). Several boot crashes due to incorrect boot-error handling (such as when catching that some client_files directories are missing) are also fixed. I would appreciate your continued feedback here.

Duplicate processing now treats ratings copy/move/merge actions a little different: Before, it would only make a change if the 'destination' file had no rating set–now it will do it if it has no rating or if the rating is smaller than what the 'source' has. For instance, merging two files that have rating 2/5 and 3/5 will result in both files having rating 3/5.

After talking with a user about different ways of storing and presenting thumbnails, I put some work into a thumbnail drawing experiment that only needs one set of thumbs to draw on screen (the current thumbnail system stores two copies for different purposes). If you would like to try it temporarily, please turn it on under help->debug->gui actions->thumbnail experiment mode. Instead of using your 'resized' thumbs, it will only use the 'master'. It makes no changes to your file structure, and you can turn it off at any time. Try loading up some random files in your client and let me know if you get any errors or if any thumbs take an extremely long time to load. I've wanted to drop the two-thumb storage system for some time but never had good numbers to feel good about it, but my tests here are promising–the experiment mode only takes a few hundred µs more per initial thumb load for me, and if I were to switch over to it permanently, I could reduce it even further. Assuming there are no big problems here, I expect to chip away at this problem over the coming weeks and hopefully make a significant simplification in hydrus file storage.

full list

- client api:

- added /add_tags/add_tags, which does several kinds of tag content updates

- added /add_tags/clean_tags, which shows how hydrus will handle potential tags

- added /add_urls/associate_url, which allows you to associate urls with files

- added 'destination_page_name' to /add_urls/add_url, which will choose which destination watcher/url importer to place the url (or create a new one with that name)

- updated client api version to 2

- updated client help and unit tests for the above

- added a linked contents to the client api help

- improved some server error handling, mostly moving 403s to more correct 400s

- improved how missing parameter 400 errors are reported from the server vs deeper keyerrors that should be 500

- .

- the rest:

- tag repository update processing now saves progress to disk every million rows or every minute, whichever comes first. this reduces journaling bloat, improves recovery when the process quits unexpectedly, and makes for significantly faster cancel when requested by the user

- when processing duplicates and copying/merging/moving ratings, the 'source' file will now also overwrite the 'destination' file's rating if that destination rating is lower (previously, the rating would only go over if the dest had no rating set)

- added a new 'thumbnail experiment mode' under help->debug->gui. this will load fullsize thumbs and resize them in memory, please see release post for more details

- reduced menubar replacement flicker while, I believe, keeping and strengthening recent menubar indexing stability improvements

- the tag autocomplete dropdown will now always embed (instead of floating) in non-Windows

- when data seems non-decodable, the fallback encoding format is now that given by chardet, rather than utf-8

- improved serialisability of some pending tag data

- watchers can now hold and pass on fixed pending tag data

- gallery log objects can now hold and pass on fixed pending tag data

- file import objects can now hold and action fixed pending tag data

- hard drive imports now store their paths-to-tags info in this new format, directly in the file import objects

- improved some url-import page drop-target-selection logic

- improved error reporting when dropping/api-adding urls

- adjusted some url import workflow so big 'already in db' download lists should work a bit faster

- attempting to start the program with some external database files but not the main 'client.db/server.db' file will now cause a boot-fail exception with an explanation before any stub db files can be made

- tightened up some hydrus service login-capability-testing code that was previously stopping certain error states from recovering promptly, even on a force account refresh, while the service was maxed on bandwidth

- fixed a source of linux CRITICAL logspam related to several common dialogs

- improved ui stability on boot when file folders are missing (particularly for linux)

- improved stability for the various async tasks on the duplicates processing page, particularly for linux. I am not sure I got everything here, but it is definitely better

- did some more misc stability improvements, particularly in various boot fail scenarios

- completely removed an ancient and janky focus catcher widget from the main gui frame

- now various db caching is improved on the python side, removed a sqlite instruction to force temp information to always stay in memory–hot data returns to staying mostly in memory to start and then spools to disk if the transaction gets too large

- fixed approx bitrate sorting for malformed video files with explicitly '0' duration

- daemon_profile_mode now spams some more info about export folders

- fixed an issue that meant client db maintenance was firing its jobs too aggressively, regardless of idle status

- updated windows build to cv 4.0

- misc refactoring and fixes

next week

We are in the home stretch of the Client API now. I just want to write some simple file searching and it'll be done. I do not expect to be done in one week, but I think I should have something working for 342.

It felt good this week to clear out some long-time bugs doing code cleaning, so I'd like to focus on that as well.

R: 6 / I: 0 / P: 7 [R] [G] [-]

Release Tomorrow!

I had a good week. I fixed several bugs (including I think the Linux >0 distance similar files search crash, finally!), did some neat ui work like better tag autocomplete responsivity, more stable and slightly smoother video rendering, and more 'open urls' options on multiple selections, and added some basic webp and tiff file import support.

The release should be as normal tomorrow.

R: 9 / I: 0 / P: 7 [R] [G] [-]

Release Tomorrow!

I had a great week after being ill at the end of last week. 341 is a big release because of the extra time. There are several new Client API commands, bug fixes and more memory reduction, and some important stability improvements, particularly for Linux.

The release should be as normal tomorrow.

R: 35 / I: 19 / P: 7 [R] [G] [-]

Version 340

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v340.tar.gz

I had a great if busy week. The Client API does more, several bugs are fixed, some new features and options are in, and stability and memory use should be a bit better.

client api

This is still somewhat advanced, but users who are interested may want to start looking in.

The first version of the Client API last week went ok! There were a couple of bugs, but thanks to feedback from some advanced users, I've improved reliability and carved out a better spec. This week brings JSON everywhere, fixes the add_file crash, and adds two neat calls:

/add_urls/get_url_files now looks up which files are attached to a URL

/add_files/add_file lets you import a file, either from a path or raw bytes

Please check the updated help here for more details:

https://hydrusnetwork.github.io/hydrus/help/client_api.html

the rest

I fixed the stupid 'page of pages' closing bug. I apologise if you were hit by this. I got a little blackpilled about all these typo errors not being caught, so I wrote a new testing suite to better test core ui functionality. I will expand this over the coming weeks and hope to completely eliminate the most blockheaded problems.

I have added nijie.info to the downloader defaults. It needs a login to access. I also updated the default danbooru parsers to get 'rating:' tags. I'd like to fold in some more downloaders to the client defaults in the near future. sfw FurAffinity should be doable next week. If you have a simple working downloader that you think would be worth rolling in for all new users, please suggest/submit it.

I added a semi-hacky checkbox to options->files and trash that pauses all new file/thumbnail requests for 15 seconds after your computer wakes from sleep. If you store your client's files over a NAS or other network solution (that may take a couple of seconds to reconnect after wake), give it a go and let me know if you like it.

I did a bunch of little work on stability and memory management and failure recovery this week. If you have had trouble with the menubar or a bloaty client or invalid website encodings or subs that fail to load, see if any of it is better here. OS X users with retina (i.e. high-dpi) screens may also see less blurry ui–I'd particularly like feedback in this case, good or bad.

full list

- client api:

- fixed up some api permissions object stuff so that /verify_access_key response can always serialise correctly

- fixed the 'add_url' api call's instability

- the API will now always return JSON on 200. anything else should be presumed to be raw text

- '/api_version' now returns JSON, and after talking with users, it will now start incrementing with every api change. it remains 1 just for this week

- '/request_access_permissions' now returns JSON

- '/add_url' now results JSON on success with more info, 403 on failure

- '/get_url_info' now returns the 'normalised_url' in the response JSON

- added '/get_url_files', which returns 'url_file_statuses', listing known hashes and file import status for that url

- added '/add_files/add_file', which can import a file from a path or bytes

- added '/add_tags/get_tag_services', which will return info on the client's tag services

- updated client api help to reflect the above changes and fleshed out the intro a bit

- fixed the client api permissions enum values in the help, which I somehow transcribed wrong first time

- updated the client api tests to check the above

- refactored client api tests to be neater and in their own file

- .

- the rest:

- fixed the page of pages close bug

- added a downloader for nijie.info to the client defaults (it needs a login)

- updated danbooru file page parsers to get 'rating' tag

- added gelbooru 0.1.11 parser for future application

- fixed an issue that was stopping advanced content updates from fully copying all the desired mappings in the transaction

- added a semi-hacky checkbox to 'options->files and trash' that will delay all new file/thumb requests for 15s after the computer resumes from sleep (useful if your files are on a NAS that takes a few seconds to reconnect on wake)

- wrote some more graceful fallback decoding handling code that attempts original assumed encoding and 'utf-8' if different and returns the one with the fewest ' ' replacement characters

- the network engine and the ffmpeg info parsing now use this new 'safe' decoding, so even if a site has borked bytes or the video file has unexpected Shift-JIS title metadata, it'll still go through, albeit with some question marks

- moved some more old daemons to the new job scheduler, deleted some old daemon code

- improved some daemon job wake and shutdown code

- wrote a proper upnp manager object and improved all-around reliability of the auto upnp-service-mapping code

- simplified the upnp check code so it now only ever checks/does anything if the respective services actually want upnp mappings. surplus mappings are now wiped immediately on service update

- fixed upnp mapping fetching to cope with ipv6 results

- improved some memory clearing code to deal with some semi-stubborn objects

- improved some 'iterate through this giant list of single numbers from the db without using a lot of memory' code and applied it to the autocomplete cache regeneration routine

- improved menubar stability, both in finding menus and swapping them out

- if a serialised json object fails to load from the db, this is now caught, the bad object deleted and written to a new file in the db dir, and all logging info captured along with an explanatory popup thrown on screen. so, if a subscription fails to load, it will now be extracted so that a subsequent subscription edit/run will work with the remaining good objects. in the case of backed-up objects (gui sessions atm), reattempting the load should restore the next most recent backup

- fixed an issue with login script validation when the given credentials have surplus ( key, value ) pairs to the script's credential definitions

- fixed two login invalid cookie error handling bugs

- maybe made some dupe filter searching more stable

- fixed a py2 datatype issue that made the client unbootable when updating the client from <296

- the client now pauses to nag and moan about backups if you try to update more than 15 versions in one go

- slightly sped up discord bugfix file drag and drops and expanded file limit up to 25 files/200MB

- added experimental secret discord bugfix dnd mode checkbox

- improved how html parsing deals with some unexpected bad tag data

- turned on primitive high-dpi support for OS X. let me know if it fixes any blurry issues on retina displays

- wrote a new 'ui test' under the debug->gui menu to help catch common-action bugs that slipped through weekly work

- improved how the test code does some wx/ui stuff, but also broke some more and ran out of time to clean it up–this is an ongoing project

- improved how some text import line splitting works

- misc fixes

next week

I pushed it too hard this week, and while I am overall happy with the work, I am going to return to normal schedule so I don't burn myself out. I'd like to keep pushing on the Client API, probably getting add_tags done, and otherwise just flesh out the new test code and do some small QoL stuff.

R: 3 / I: 1 / P: 7 [R] [G] [-]

No Release Tomorrow!

I fell ill this week and did not get enough work done to justify a release. I will make some of the time up tomorrow and then continue as normal, aiming for 341 to be out on the 27th.

Thanks everyone!

R: 4 / I: 0 / P: 7 [R] [G] [-]

Any support for AGN.PH?

Does anyone have a working gallery importer for AGN.PH? Gotta collect it all…

R: 27 / I: 6 / P: 7 [R] [G] [-]

Version 339

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v339/Hydrus.Network.339.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v339/Hydrus.Network.339.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v339/Hydrus.Network.339.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v339/Hydrus.Network.339.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v339.tar.gz

I had a good week. I fixed some bugs and finished one part of the Client API.

client api

(This is only for advanced users for now.)

The Client API is now ready for some experimentation! Although most of this work has gone much faster than I expected, the last step of actually writing the server commands along with some half-decent test code and help documentation is proving a little slower. This week rolls out 'set up API access' and 'add urls' functionality. In the coming weeks, I will roll out 'add file', 'add tags', and 'search files' commands.

The basic (under construction) outline is here:

https://hydrusnetwork.github.io/hydrus/help/client_api.html

If you are interested in writing a script to talk to the Client API, please have a play with the new service under manage services and review services and see if you can get some fun going. I am interested to hear your feedback and to help if you run into trouble.

misc

I messed up several things last week, thank you for the quick reports. The bugs with ratings service sorting, the manage import folder dialog, and setting a UPnP port on the local booru/client api are now all fixed.

I also added a button to the ratings' review services panel that allows you to delete ratings en masse based on several file categories. If you have wanted to clear all the ratings for your deleted files, check it out!

full list

- client api:

- wrote some ui to handle client api permissions adding and editing

- wrote a 'catch a permissions request' mini-dialog for external api permissions adding

- wrote api calls:

- GET

- /api_version

- /request_new_permissions

- /verify_access_key

- /add_urls/get_url_info

- POST

- /add_urls/add_url

- and made a new 'client api' help page to describe in detail what these do

- wrote fairly comprehensive unit tests for the new client api

- refactored a bunch of 'hydrus network' specific stuff away from general server code that the client api now uses

- neatened up 401 vs 403 error handling across the program, and replaced some clientside error handling that was inelegantly borrowing these network errors

- deleted very old prototype file/thumbnail client server fetch code, which was no longer in use

- .

- misc and bug fixes:

- added a 'clear ratings' button to the ratings service 'review services' panels. it can clear out ratings for deleted, non-local, or _all_ files

- the '3 downloaders are working, is it ok to close the client?' message is now folded into the 'confirm client exit (auto-yes in 15s)' dialog. this merged dialog will still appear for users who have the regular confirm client exit dialog turned off (and still auto-yeses in 15s)

- the file url downloader now reports 'downloading file' and 'importing url' text status separately

- fixed a typo bug from last week that was breaking asc/desc ratings service sorting

- fixed a typo bug from last week that was stopping manage import folders from opening

- fixed a typo bug from last week that was breaking setting upnp port on the local booru/client api service management panel

- the advanced file reparse-and-re-thumbnailing routine now correctly moves a file to its new extension if its mime changes (e.g. png->apng, or webm/mkv distinctions)

- the client file manager now silently detects and auto-repairs instances of missing files where the file actually does exist, just with the wrong extension

- fixed a url parsing issue that was normalise-mangling url classes with no path but some query parameters

- the network engine now uses utf-8 decoding when no specific encoding is set (previously ISO-8859-1)

- fixed an ffmpeg video parsing bug when the video included Shift-JIS metadata. it should work for other unusual encodings as well

- maybe cleaned up some menubar management code

- the filename tagging dialog now uses a notebook for service choice, like the manage tags dialog, rather than the janky old listbook

- fixed a py2-to-3 issue with the admin-only 'is server currently busy' check while a backup is running

- improved some dialog button event handling. it may completely fix the 'trytoendmodal' issue some users run into

- improved some JSON db serialisation error reporting code, trying to pin down an issue several users have seen with session save

- improved thread-safety of serialisable objects as they serialise

- misc improvements and cleanup

next week

More Client API stuff. I did 'add urls' this week because it was the simplest. I think I can do 'add file' and 'add tags', and we'll see how much of 'search files' I can do as well.

I had meant to try out some high-resolution support for the OS X release this week, but it kept getting put off. I realised I had missed it again just now and tried to slip it in, but my first attempt did not go perfectly. Rather than try to rush it together in the last moments of the week, I have decided to instead give it another go for 340. Sorry for the delay!

R: 0 / I: 0 / P: 8 [R] [G] [-]

Release Tomorrow!

I had a great if busy week. I fixed the stupid page of pages bug (and a whole bunch more), fleshed out the new client api, added some small new options, improved stability, and added several new downloaders to the client defaults.

The release should be as normal tomorrow, maybe a little late.

R: 1 / I: 0 / P: 8 [R] [G] [-]

need the size of the thumbnail more than 200x200.

on 4k TV, they look small

R: 0 / I: 0 / P: 8 [R] [G] [-]

Release Tomorrow!

I had a good week. I fixed a lot of bugs (including a couple of stupid ones I let slip into 338) and finished a first subset of the client api for advanced users to play with.

The release should be as normal tomorrow.

R: 648 / I: 201 / P: 8 [R] [G] [-]

Bugs Thread

Gonna start another thread because >>173 is becoming too big for most people.

In the tag manager, whenever multiple tags are selected for multiple files with different tags, if you press the del key, the program will still ask you if you want to delete them all or put them all in the selected files. This does not make sense, if I'm pressing the del key it's obvious that I want them gone.

R: 25 / I: 4 / P: 8 [R] [G] [-]

Version 338

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v338/Hydrus.Network.338.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v338/Hydrus.Network.338.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v338/Hydrus.Network.338.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v338/Hydrus.Network.338.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v338.tar.gz

The recent v335 was an important update. If you are updating from v334 or earlier, please go read that release post for your special one-time update instructions and update to v335 before you try this week. Once you have updated to v335, the process for updating to anything newer is back to normal.

I had a great week. There's some bug fixes and neat new tag stuff.

tags

After talking to some users this week, I decided to knuckle down and improve autocomplete tag input. This has been in the works for a long time, but I have had trouble just putting the time in.

So, the autocompletes for both file searching (on pages) and tag management (on manage tags dialog) now work asynchronously! They search in the background, letting you continue typing smoothly without hanging you up. Large and laggy searches are cancelled mid-job as needed. Hitting enter before results come in is much smoother and more reliable, so if you just want to add simple tags like 'green eyes' by typing them in and hitting enter, it is much easier and less janky.

I am very happy with this change. Improving the core experience is something I want to do more. That said, I did this in a slightly rough way (I am wrestling with some terrible ancient code here), so there may be some bugs for certain workflows. If you discover you are getting false positives when typing fast or any other strange behaviour, let me know. This new system may want some different fetch timing settings, as well. All feedback welcome, basically.

Also, any 'selection tags' taglist on a regular page now lets you right-click->select all files with (selected tags)! If you select multiple tags for your right-click, you can either select the files with all the tags (AND), or those with any of the tags (OR). This is pretty neat, so give it a go!

misc

Pages now have a 'sort by approx bitrate' in their sort dropdown. This does a very basic (filesize / duration) calculation (putting durationless files on the smaller end) and lets you prioritise bloated gifs and other short, high-filesize (i.e. easy and useful to filter) vids.

Gelbooru changed up how some of their URLs work, so they suddenly started giving 403s for the hydrus default downloader. This release folds in a new parser that works again and a couple of new downloaders for gelbooru user favourites and pools.

I fixed an issue that meant preview view duration was ofter being counted in the new file viewing statistics system even while previews were in hidden pages.

full list

- after talking with some users, put a bit of time into tag autocomplete wolkflow. both read (search pages) and write (manage tags) autocomplete inputs now operate ~asynchronously~, with the tag fetch working on a separate thread. tag jobs can also now cancel at certain checkpoints in the tag search process if overwritten by a new request. therefore, a variety of tag lookup scenarios _should_ be less painful. this change was executed in a semi-inelegant way, so please report if you encounter bugs from fast typing etc…

- I also improved some of the "I hit enter before results were in" code as a result of this. not sure I have it totally nailed, so please give feedback on errors here

- wildcard search tags now have an explicit '(wildcard search)' after their label

- taglists that have an attached page of media (basically the 'selection tags' box and the 'active search predicates' up top) now have a right-click menu entry to 'select files with (all these tags)' and, if more than one tag is selected, 'select files with (any of these tags)'! This is pretty neat in action, so give it a go!

- added a 'sort by approx bitrate' file sorting option that does a basic filesize/duration so you can filter out dense gifs and other short-but-big vids a bit easier. anything without a duration is shoved to the 'smallest' side

- some of the file sort options now default to their respective 'biggest' first, see how you like it

- folded in updated gelbooru file page parser (fixing the 403 errors that just appeared) and added searches for gelbooru user favourites and pools

- if a gallery or watchable url ends up in the file processing queue (and hence fails), it now says the believed-to-be url class name in the error, which should help some false positive url class matching debugging

- fixed a focused-file selection issue that meant preview viewtime was frequently counting in file viewing statistics even while a page was not currently in view

- fixed the local booru review service panel, which wasn't fully deleting shares when the button was clicked

- wrote a service object and basic server skeleton for the client api (basically refactoring the existing local booru code). client api now appears in manage and review services and can boot and present the normal hydrus browser welcome page at '/'

- the client api and the local booru now have an 'allow non-local connections' checkbox! defaulting to off and on respectively

- updated a wx-thread call function to more safely and universally deal with instances where the responsible window died before the call could be made

- removed some old experimental crypto code that isn't used any more–pycryptodome is no longer needed to run the program from source

- improved some misc client service code

- deleted some old unused code

- misc cleanup

- updated to ffmpeg 4.1 on windows release

- updated to sqlite 3.26 on windows release

- updated to wxpython 4.0.4 on all platforms

next week

The Client API continued to go better than expected this week. If you explore manage/review services, you'll see the stubs of the new 'client api' service and can even boot a skeleton server. I'll work on a bit more ui and access code this week, and with a bit of luck, I'll actually have it doing something fun for v339.

R: 9 / I: 0 / P: 8 [R] [G] [-]

中文帖子(Chinese Thread)

你們有甚麼問題也可以在這裡問

https://hydrusnetwork.github.io/hydrus/ (主頁)

https://github.com/hydrusnetwork/hydrus (開源軟件)

https://t.me/hydrus_network (Telegram即時通訊)

https://8ch.net/hydrus/index.html (BBS)

R: 23 / I: 5 / P: 8 [R] [G] [-]

Version 337

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v337/Hydrus.Network.337.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v337/Hydrus.Network.337.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v337/Hydrus.Network.337.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v337/Hydrus.Network.337.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v337.tar.gz

The recent v335 was an important update. If you are updating from v334 or earlier, please go read that release post for your special one-time update instructions and update to v335 before you try this week. Once you have updated to v335, the process for updating to anything newer is back to normal.

I had a good week. A bunch more bugs are fixed, and I added some little things as well.

all misc this week

I fixed a variety of bugs this week. A unicode log printing bug still hanging over from the py3 update, some unusual webms (mostly ones that tend to be a single frame with music), and some network download file size checking.

I also believe I have completely fixed the shutdown memory explosion some users were seeing. I managed to recreate the bug and figured out some bad shutdown scheduling that was causing an event processing routine to go crazy. I cleaned a bunch of other shutdown code as well, including a couple instances of subscription processing not stopping promptly on a client shutdown. Overall shutdown (particularly the last moments just before the splash screen closes) should be a bit smoother for all users.

I also added some little new things. The duplicate system now merges/copies pending tags as well as 'current' tags (and the advanced service-wide content update routine does as well). The 'lain' image on the downloader easy-import page is now clickable to launch a file selection dialog. And the manual export files dialog has a new experimental checkbox for advanced users that tries out exporting symlinks (which requires running as admin mode in Windows, so I don't really recommend it for that). If you try this symlink export, let me know how it goes!

full list

- fixed another couple of unicode encoding problems with the logging and profiling code

- the logger now sticks a unicode BOM at the top of new log files to help text readers guess the utf-8 encoding

- fixed musical webm import when the video stream has no stated duration but the audio stream does and typically stretches out a 'single frame' video

- fixed some 'max size' download file size testing

- 'waiting on bandwidth' statements on network job controls now show the specific network context (like 'web domain: somesite.com') they are waiting on (you might need to hover over to see the tooltip for this)

- the downloader easy-importer lain image is now clickable to launch a file selection dialog

- if you are in advanced mode, the manual file export dialog now lets you export symlinks with a new checkbox. this is experimental, so if you are interested, give it a brief test and let me know how it works for you

- duplicate content merging now applies to pending as well as current tags

- the duplicate filter now counts pending tags when saying which file has more tags

- advanced content updates now _copy_ both current and pending tags. the other actions now state what they do more clearly

- stopped printing long server error text on 304 and 404, where actual response content is uninteresting

- removed wx import that accidentally came into server space due to 2to3 check

- improved the path-fixing code that helps environment construction when launching external programs from non-windows frozen builds

- fixed a critical pubsub processing bug that kicked in at a certain stage of client shutdown. this should stop the post-shutdown-processing memory explosion certain users were seeing and should stop any ui jank in the last 0.2s of the program for everyone else

- improved some other shutdown memory cleanup that was sometimes leading to double-log-printing of exit statements

- did a full pass over the daemon scheduling code. it now reacts more responsively to various shut down situations

- reduced db disk cache aggression significantly and added more memory maintenance to the cache population process

- fixed an issue where subscriptions were not promptly responding to shutdown events

- fixed an issue where some delayed network jobs (e.g. while all network traffic is paused) were also not responding to shutdown events

- added a 'pubsub report mode' debug mode for simpler pubsub review

- the db is now less redundantly spammy on certain behind the scenes update notifications

- wrote a first version of the client api manager and permissions handling objects

- misc fixes, cleaned some shutdown code

next week

I started the Client API for real this week. The first 'manager' component is basically done, and it wasn't as complicated as I feared. I'll get started on some ui code for service review and authorisation management this coming week and otherwise just do some more small work to keep catching up on things.

R: 1 / I: 0 / P: 8 [R] [G] [-]

Release Tomorrow!

I had a great week. As well as fixing a bunch of bugs, I radically improved how tag search works and feels and I added a neat way to select files by their tags. I moved the client api forward more than expected, but it is not ready for use yet.

The release should be as normal tomorrow.

R: 1 / I: 0 / P: 8 [R] [G] [-]

increasing the public tag repo bandwith

Hey i just installed the hydrus client. It's perfect to organize my memes, props to the dev. My problem is that the bandwith volume is too low. I added the public repo mentioned on the github.io site. After 1932 tags downloaded my client bandwith was exceeded. Maybe the bandwith volume so anyone, whos adding the public repo can download anything in on download and not waiting 9 hours to get more volume.

R: 1 / I: 0 / P: 8 [R] [G] [-]

Release Tomorrow!

I had a good week. I fixed several more bugs, including some more unicode issues and shutdown scheduling, and the big shutdown memory explosion some users were seeing, and added a couple of new small features. The client api work started well but is not ready for use yet.

The release should be as normal tomorrow.

R: 18 / I: 1 / P: 8 [R] [G] [-]

Version 336

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v336/Hydrus.Network.336.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v336/Hydrus.Network.336.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v336/Hydrus.Network.336.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v336/Hydrus.Network.336.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v336.tar.gz

Last week, v335, was an important update. If you are updating from v334 or earlier, please go read that release post for your special one-time update instructions and update to v335 before you try this week. Once you have updated to v335, updating to anything newer is back to normal.

I had a good week. A bunch of py3-related bugs are fixed, and I improved some heavy-load throttling.

py3 fixes

Overall, last week's update went well. CPU and memory use and overall jank is down for all users, and afaik there weren't any permanent disasters in the update. This was a big relief for me.

That said, as expected, several unusual bugs related to py3's new unicode handling slipped through. Thank you for the reports. A common one was related to windows logging, another was in pixiv searching with kanji tags, another with videos with unicode metadata. I believe I have cleaned most if not all of it up. There may be a couple more out there, so please let me know if you get anything new. Otherwise, please resume any broken subs or retry any broken webm imports.

I also fixed the numerical tag ui, which was coercing all clicks to to zero or max stars, and which I messed up by accident!

better heavy workflow

The client now has a strict thread-level throttle on the number of importers that can work at once. It seperates the different sub-parts into their own max limits, but the overall total limit is about 20-40. This is in an effort to stop ui hang after accidental heavy import situations such as pasting 100+ queries into a gallery downloader. This is a 1.0 throttle and doesn't have any ui to report 'waiting on a slot' or anything. I don't expect 99% of users will even notice it, but if it affects you seriously, please let me know.

In a related note, for advanced users, gallery downloaders now have the ability to 'bundle' mass-pastes into one downloader, queueing up all the different gallery queries sequentially in the same object (rather than in many in parallel, which is what previously cluttered the ui and boshed the CPU). If you sometimes paste a bunch of single-file md5-queries at once, please check this new check menu item under the gallery downloader's cog menu.

full list

- fixed an issue where the numerical rating control was coercing all clicks to either the minimum or maximum allowable rating (e.g. 3/5 stars wasn't working)

- fixed some text file and process i/o, which was handling some unicode decoding/encoding incorrectly. it now mandates utf-8 in all cases

- fixed a referral url encoding problem that was stopping pixiv from downloading when the gallery page url had kana/kanji characters (from a search term)

- fixed a str vs bytes issue when loading the filename tagging panel

- fixed the delete button on the filename tagging quick namespaces panel (the edit and delete buttons are also now 'live' and will disable when nothing is selected)

- improved some json dump deserialisation code

- fixed a data-sorting issue that would appear with certain parsers in the edit parsers panel

- improved video metadata parsing, fixing an issue when the video has a 'title' row containing inconvenient data

- fixed some hex character processing for system preds

- added an advanced check item to the gallery downloader cog icon menu that will 'bundle' multiple query-pastes to the same single gallery downloader (this is helpful if you are pasting a whole bunch of md5 queries in one go and would rather one downloader work through them sequentially than 50+ separate ones blat your CPU simultaneously)

- the different kinds of importer worker threads now have several limits on the max number that can be working at once, to stop accidental ui overload when a hundred or more are in memory and all want to work at once (like after a big paste event or resuming after computer sleep). during periods of heavy import activity, the importers will now naturally space themselves out to smooth out the spike. the limits are hardcoded for now, let me know if it noticeably bottlenecks your situation

- made some menubar update code a bit less complicated and reduced how often it'll spam during heavy update

- the 'what to do?' buttons that appear in manage tags sometimes on a tag action got a simplification pass and are now on the new dialog system

- simplified my new dialog code significantly, clearing out redundant code and classes and pushing all okable/cancellable/vetoable closing checks through one single method

- wrote some new help.txt in the db dir about hanging startups

next week

I still have a bunch of small stuff going on, but I'd like to start on the Client API, the next 'big thing', as well. I expect this will involve several weeks of ui and clientside server prep before anything exciting happens.

R: 11 / I: 1 / P: 8 [R] [G] [-]

Experimental decentralized booru software

Hello Hydrus Network community.

I am a programmer familiar with booru development. I have written my own private booru software that scrapes public websites for images much like Hyrdrus Network, it currently has 10 million files, but this isn't what I am here to talk about.

I am very interested in Hydrus Network's goals of decentralized zero-authority sharing and collaborative organization of images as developments towards these goals benefits my own booru.

I have recently created a proof of concept decentralized booru called "Decboo", it uses IPFS for image distribution and the Ethereum blockchain for tag distribution.

Decboo is not the booru I spoke of earlier, it is an extremely basic program made only to demonstrate blockchain-based tag distribution, functionality that I hope the Hydrus Network developer(s) will integrate into the Hydrus Network software.

A zip file containing Decboo's software can be downloaded here: https://ipfs.io/ipfs/QmZvYUrXFwpqCognaLm6ecSixQRuXbeXScrZKA71VRUsgE

R: 72 / I: 17 / P: 8 [R] [G] [-]

Version 335

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v335/Hydrus.Network.335.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v335/Hydrus.Network.335.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v335/Hydrus.Network.335.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v335/Hydrus.Network.335.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v335.tar.gz

When I first made this release, Github’s file upload was not working right, and I used Mediafire instead. Github is now working and I have updated the links above.

I had a great four weeks updating hydrus to python 3. The update went well, and the releases today are ready for all users, but there are special update instructions just for this week.

python 3

The client and server now run completely and exclusively on python 3, updating from python 2. The new version has a variety of benefits, mostly in better unicode vs. data handling, but for hydrus it also runs a little faster, uses less idle CPU and significantly less memory, and has a little less ui-jank.

I am pleased with the update. None of it was extremely difficult, but there was a lot to do, a few headaches, and some new stuff to learn. I am glad I took the four weeks. I also appreciate the users who tested the preview releases in the past couple of weeks.

I have squashed a ton of little bugs, and everything normal like file downloading and browsing seems to work completely fine, but there are likely a couple of small issues still out there. If a dialog breaks for you and you get some popups about some datatype being mishandled, please send it to me and I'll get it fixed up for v336!

some notes

Unfortunately, for technically difficult reasons, I could not compile 'debug' versions of the executables, so these are gone this week. I will revisit this, but the original debug builds were a bit hacky and may no longer be practically possible in py3.

Also, I am not certain if the database menu's 'set a password' will have kept correct password data for unusual keyboards, so if you use this function, the client will, just for this v335, forgive incorrect passwords! If this happens to you, the client will give you a popup with more information and tell you how to try to fix it. I would appreciate feedback here, if you encounter it.

Due to a natural library update unrelated to py3, your hydrus sessions will likely be wiped (this also happened to some running-from-source users a little while ago), which means Hydrus will have to re-login to any sites you have logins set up for. If you have special cookies you need to save or re-import from your browser, get set up before you update!

Now, because py2 and py3 are incompatible, the new version cannot be run in a folder that still has old .dll files hanging around. Please follow the following to update:

update instructions for windows installer

Just for this week, I have added a routine to the installer to delete the old files (but obviously saving your db directory where your database and files are stored!), so you shouldn't have to do anything. I nonetheless recommend you still make a backup before you update. Backups are great, and if you don't make one yet, this week is a great time to start.

If you are a particularly long-time user and the installer fails to clear everything out, you may need to delete the old files yourself, like the extract users will have to:

update instructions for windows and linux extract

You will have to perform a clean install, which means deleting everything in your install folder except the db directory before extracting as normal. This is simple, but do not get it wrong. Do not delete your db directory–this is where your database and files are stored.

As always, if you have a recent backup, you don't have to worry about any possible accident, so make sure you have one.

update instructions for os x

Due to technical limitations, the OS X release is now App only. Furthermore, this App release will no longer store the db inside itself! The default location for your db is now ~/Library/Hydrus (i.e. /Users/[you]/Library/Hydrus). This also means that the future update process will be as simple as replacing the existing Hydrus Network App in Applications, just one action. I apologise that this important change has taken so long to come out, but we got there in the end.

If you are updating this week, you will need to make the Hydrus folder under your Library yourself and move your existing db there so the new Hydrus can pick up where you left off. If you use the tar.gz, you'll be moving the contents of your install_dir/db, and if you use the App, you'll want to right-click->Show Package Contents on your old py2 App and navigate to Hydrus Network/Contents/MacOS/db. You want the contents of db, not the db folder itself, so the path to your new client.db file should be ~/Library/Hydrus/client.db, for instance.

If you cannot see your Library folder, check this: https://www.macworld.com/article/2057221/how-to-view-the-library-folder-in-mavericks.html

If you have trouble with this, please let me know and we'll figure it out together.

update instructions for running from source

You'll need to make a new py3 venv and make new shortcuts. I now use pycryptodome instead of pycrypto and dropped some libraries, so I recommend you go to the 'running from source' help page again and just paste the new pip line(s) to get set up.

I don't think 3.4 will work, but 3.5, 3.6, and 3.7 all seem ok. Obviously contact me if you run into trouble. I'm also interested in success stories!

full list

- important:

- hydrus now runs completely and exclusively on python 3!

- for users who are updating, the client has special install instructions for just this week:

- if you are a windows or linux user who extracts to install, you will have to delete your old install's files (but keep your db folder!!!) before installing/extracting the new version so there are no 2/3 dll/so conflicts (don't delete your db folder!)

- if you use the windows installer to install, this v335 installer will do the clean install for you! there is absolutely no way this could go wrong, so no need to make a backup beforehand :^)

- if you are an os x user, I am now only releasing the client in the app. furthermore, the default app db location is now ~/Library/Hydrus (i.e. /Users/[you]/Library/Hydrus). you will have to move your existing db to this location to update, and thereafter you'll just be replacing the app in Applications!

- if you try to boot a non-clean mixed 2/3 install, the client will try to recognise that and give an error and dump out

- please check the release post for more detailed instructions here

- .

- semi-important:

- the db password feature may be one-time broken for unusual keyboard languages, so failures this version will be forgiven with an appropriate error message explaining the situation. feedback from чики брики lads appreciated

- I may have fixed the issue some linux/os x users were having launching external programs, including OS ffmpeg (it was a child process environment issue related to pyinstaller)

- although I did most of my devving here on py 3.6, the client seems to run ok on 3.5. I doubt 3.4 will do it, if you mean to run from source

- I moved from the old pycrypto to the new pycryptodome, so users who run from source will want to get this. I also dropped some libraries

- .

- misc bug fixes:

- fixed the 'load one of the default options' button on manage tag import options when a set of default options is orphaned by a deleted url class

- removed some popup flicker related to long error messages

- fixed some parsing testing ui error handling

- cleared up some bad text ctrl event handling that could sometimes cause a recursive loop

- listctrls should now sort text that includes numbers in the human-friendly 2 < 10 fashion

- cleaned up some bad external process calling code and improved how child process environment is set up

- finally figured out the basic problem of a long-time nested dialog event handling error that could sometimes freeze the ui. I may have fixed it in one case and will keep working on this

- .

- boring details:

- ran 2to3 to auto-convert what could be done

- updated environment to python 3

- went over a whole ton of unicode encoding/decoding manually to update it to python 3

- removed all the old tobytestring/tounicode calls in favour of new python 3 handling

- fixed all the file io to do bytes/str as appropriate

- corrected a bunch of / vs // int/float stuff

- fixed up twisted, which has some str/bytes stuff going on

- fixed all the listctrls to deal with column sorting None values amongst ints/strs

- fixed png export/import, which had some fun bytes/bytearray/int/str issues

- updated the swf header parsing code to py3 (more str/bytes stuff)

- misc float/int fixes

- fixed up some http.cookies handling, which has changed in py3

- improved some ancient loopback connection code that was only really checking to see if ports were in use

- cleaned up a bunch of now-invalid parameter tuples that 2to3 helpfully marked

- numerous misc other refactoring and so on

- updated the new network engine to now decode non-utf-8 responses correctly based on actual response header

- removed some old py2 manual http multipart code

- removed the old py2 'matroska or webm' parsing py, replacing it with some direct ffmpeg format inspection

- replaced all % formatting with the new .format system. I will slowly move to this rather than the current endless concatenation mess

- deleted some more misc old code

- tightened up some spammy network error reporting

- converted all /r/n to /n in my environment project, ha ha ha

- the ui seems to better support rarer unicode characters like

- updated some of the install/update/backup help for all this, and some misc other stuff as well

- fixed misc bugs

next week

A lot of small stuff piled up over the holiday. I will spend a week or two trying to catch up and also planning out the client API, which will be my first big job of the year.

I hope you had a good Christmas and New Year. Mine were great, and I am looking forward to 2019. Let's keep pushing and see if we can do some fun stuff. Thank you for your support!

R: 2 / I: 0 / P: 8 [R] [G] [-]

Release Tomorrow!

I had a good week. I have fixed several bugs in the new py3 release (numerical ratings, some file and network unicode i/o, and some list sorting), cleaned and improved some long-time ui jank-code, and wrote in some limiters and a new workflow to smooth out mass-multiple import jobs.

The release should be as normal tomorrow.

R: 3 / I: 1 / P: 8 [R] [G] [-]

Short Delay

Github is having trouble with my file uploads, so the release is delayed. I’ll give it another go later tonight, and if that fails, I’ll figure out something else tomorrow. Sorry lads!

R: 3 / I: 1 / P: 9 [R] [G] [-]

Important Release Tomorrow!

I had a great four weeks updating the software to use python 3, the new version of the language I write hydrus in. Overall, it went very well. The new client does not look different, but its guts are newer and neater. Some things run faster, and some long-time py2 bugs are gone.

The release will come at the normal time tomorrow, but it will have special one-time update instructions for all existing users. It is important you read the release post.

R: 37 / I: 12 / P: 9 [R] [G] [-]

Version 334

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v334/Hydrus.Network.334.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v334/Hydrus.Network.334.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v334/Hydrus.Network.334.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v334/Hydrus.Network.334.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v334/Hydrus.Network.334.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v334.tar.gz

I had a good week. I mostly fixed bugs in order to make a clean 'final' python 2 release.

all misc this week

I wrote a system: predicate for the new file viewing statistics. It works great!

Also, you can suspend file viewing tracking and clear all records entirely under the new database->file viewing stats menu.

I shuffled some options around in the options dialog and created a new 'gui pages' entry.

The thread watcher should now correctly recognise url and url#12345 are dupes in all cases.

I wrote some login info to the downloader help and completely rewrote the subscription help:

https://hydrusnetwork.github.io/hydrus/help/getting_started_downloading.html

https://hydrusnetwork.github.io/hydrus/help/getting_started_subscriptions.html

full list

- wrote a system:file viewing stats to comprehensively search the new viewing stats–it _should_ also be synced with the exact current values

- but for system:everything, inbox, and archive, which remain where they were, system predicates are now sorted alphabetically!

- added a _database->file viewing stats_ menu that lets you suspend file view tracking and clear all records permanently

- mr. bones now welcomes all users under the help menu

- fixed mr. bones's confusion at those who have yet to board the ride

- also mr. bones now makes sure to get the latest file viewing stats

- moved confirm trash/archive from _options->gui_ to _options->files and trash_

- moved a bunch of 'pages' related stuff from _options->gui_ to the new _options->gui pages_

- added an option to options->gui pages to change the number of session rolling backups

- subscription popups now provide an x/y query progress string in their popup text

- the edit subs/sub panels are now a bit shorter by default and the edit sub has its own frame position data, under 'edit_subscription_dialog', and remembers its last size and position by default

- fixed an issue where some dupe watcher urls (like url and url#12345) were not being correctly merged on a mouse drag and drop watcher-import

- the client will now print up to 512KB of server error info to the log (previously 4KB)

- removed the youtube download prototype–if it returns, we'll do a proper youtube-dl solution. as a result, pafy is no longer needed to run the client

- network report mode now shows more network error information

- gave the 'getting started with subscriptions' help page a complete pass. it now reflects the new system and has up-to-date advice based on my new experience

- wrote a 'logins' section to the bottom of the 'getting started with downloading' help page

- misc fixes

the next four weeks or so

As planned, I will now concentrate on updating the software to python 3. My rough guess has budgeted four weeks for this, but I can't say with great confidence. I will stop putting out normal releases, but I will try to keep up with threads and messages, go to the discord on Saturdays, and make progress updates on Tuesday evenings like my 'release tomorrow' posts. I hope we'll be working on new code and back to normal schedule in the new year.

I had a great 2018, and I am now looking forward to 2019. I am still in a position to work on hydrus, and I still enjoy it. I really appreciate your support–thank you!

R: 3 / I: 1 / P: 9 [R] [G] [-]

Python 3 Update Progress 3

Happy New Year!

I had a great week. The client works well, all big tests passed, and I now have functioning builds for Windows and Linux. I will spend the last week figuring out an OS X environment and doing last tests and cleaning.

I feel great about the schedule. v335 is completely on track for a Jan 9th release.

R: 8 / I: 0 / P: 9 [R] [G] [-]

Python 3 Update Progress 1

I had a good week. I was able to get the client booting and closing cleanly on a new Python 3 environment on Windows. I have changed hundreds of tiny things, and there are still still a bunch more to go, and then I have to hammer out the 'freezing' process to make the executable release, and then figure out the same environment and build for the two other platforms.

I feel good about the schedule. It involves a lot of bashing my head against the keyboard, but I am moving forward.

R: 17 / I: 3 / P: 9 [R] [G] [-]

Database Encrypted Sync

I haven't seen this discussed so I thought I'd make a thread on it.

We are all aware that Hydrus runs fine inside a Truecrypt container, but having one huge file can be a problem.

For instance if you backup your 500GB database inside a truecrypt container and then download even just 1 image you basically need to backup the whole 500GB all over again if you run an automated backup to an external HDD or a NAS. Another problem with Truecrypt/Veracrypt volumes is you have to set the container size at the time of making it, leaving it too big or too small all too easily.

Well I've been experimenting with different crypto programs that feature encrypting directories while keeping files separate. The filenames and contents are unreadable, but they are still their own files so syncing and backup programs know exactly what to copy and what is the same.

The problem is that Hydrus doesn't want to run at all within these encrypted directories, but luckily we can have our installed client separate to the db folder.

>install an encryption program (I used cppcryptfs but there are a bunch for various platforms)

>create a new encrypted folder on any drive

>mount it as a Drive letter (you will need to use the same Drive letter each time)

>Move your Hydrus db folder to that mounted drive (or you can make a new one by ignoring this step)

>create a shortcut to your client.exe adding -d="path to db folder on virtual drive" (e.g "C:\Hydrus Network\client.exe" -d="Z:\db")

>use shortcut

it should find your database and start like normal, with any new files being encrypted as their own files that can be synced individually.

I haven't tried it but theoretically you could even mount a cloud provider as a virtual drive and store your entire database that way, probably stupid but I might try it for fun.

R: 17 / I: 2 / P: 9 [R] [G] [-]

Python 3 Update Progress 2

𝕸𝖊𝖗𝖗𝖞 𝕮𝖍𝖗𝖎𝖘𝖙𝖒𝖆𝖘!

I had another good week. The client works about 95%, and I can get it into a proper executable release that runs fine. I now need to iron out the last issues and sort out Linux and OS X environments.

I feel great about the schedule. I am still aiming for a Jan 9th release for v335.

R: 0 / I: 0 / P: 9 [R] [G] [-]

11111 GET

Hey I heard you guys needed help with the website. Care to give me a shot?

>pic related

R: 1 / I: 0 / P: 9 [R] [G] [-]

hi i've had hydrus a while back and am getting back into it now

i'll have some questions about migrating db's from old to new

but first i wanna ask about what storage hardware is best for hydrus?

because i think i broke me old hdd by using hdyrus (seagate's being prone to crashes)

the tags that i had through the public tag repository should re-appear when i import files, right?

R: 47 / I: 6 / P: 9 [R] [G] [-]

Version 333

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v333/Hydrus.Network.333.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v333/Hydrus.Network.333.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v333/Hydrus.Network.333.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v333/Hydrus.Network.333.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v333/Hydrus.Network.333.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v333.tar.gz

🎉🎉 Merry v333! 🎉🎉

I was slightly short on time, but I still had a great week. There's some fixes and speedup and a bit of fun.

file viewing statistics

The client now records how often (and for how long) each file has been viewed in the preview window and main media viewer! You will see how many times you have viewed a file in its normal thumbnail and media right-click menus. You can customise how this info displays in the menu–including hiding it completely–under options->media.

You can also sort by total views or viewtime under the normal file sort dropdown (which is now itself alphabetical and has improved labels).

This is only a first version, so it isn't perfect. The total viewtime isn't updated until you finish looking at a file, for instance, and counts will sometimes be temporarily desynchronised. I expect to revisit it, maybe adding the stats to the main media viewer's top-right window, and making the current viewtime live, so the seconds count up correctly as you look at something. I'll add a system predicate for it as well, so you'll be able to search for things you've seen at least x times and so on. I will add an option to turn these stats off completely next week.

What do you get here? Do you go back to the same twenty files over and over, or are things more evenly distributed? Once we have fleshed this data out, I could draw some distribution graphs or do some more innovative 'show me some infrequent stuff' searching.

another cache

After the good file and tag work of the past few weeks, I've written a higher-level cache that significantly streamlines how gui-level media is stored in the client. There is now only ever one copy of each media object, with all thumbnails and media viewers pointing at that same one. This reduces total memory usage and CPU in many situations, makes it possible to immediately show content changes after advanced updates like tag repository processing, and it speeds up certain searches as duplicate media objects do not have to be recreated from scratch at the db level.

I am quite pleased with this cache. I have been thinking about it for a long time. Please let me know if you have any problems with it (likely it would be some variation on "I changed a tag/rating/whatever, but the file doesn't show the change, even when I refresh the search").

tumblr madness

If you use tumblr for lewd purposes and missed the news this week, tumblr have gone nuts and decided to ban all nsfw content off their platform on Dec 17th. There's been some corporate drama related to the tumblr app, but no one knows what's really going on with this overbroad new decision–my assumption is Verizon haven't been able to find a workable business model, so they are seizing this chance to reduce liabilities rather than continue throwing money away. Maybe they'll try to find clean ads to run like 4chan are currently doing with 4channel, or maybe they expect to kill the whole thing softly over the next few years. And perhaps the outcry will convince them to reverse the decision, but don't plan on it.

So! If you had plans to download from some nsfw tumblr, get it going now. Go visit the creator's blog and see where they are migrating to so you can figure out new subscriptions.

If you have a big tumblr yourself that is about to get semi-nuked due to your reblogs, there may also be a clever one-time way to get higher quality versions of what you reblogged as well, as per here: >>10905

full list

- added a first version of file viewing statistics! the client db now keeps track of how many times a file is loaded in the preview and full media viewers, and for how long!

- you can see the media and preview stats on any single media right-click menu. there are multiple options for how this displays, including hiding it completely, under options->media

- viewing stats update as they happen! (although viewtime typically only updates on the end of viewing. I'll likely make this more live, especially if I end up showing this info in the media media viewer)

- you can now sort files by total media views/viewtime!

- mr. bones's wild ride continues, as well

- deleted the old 'file list' way of updating in-ui media objects in favour of a long-planned global media cache. there is now only ever one active copy of any particular media, and all data-level updates need only occur once on that single copy. this saves a bunch of CPU, memory, and overall hassle behind the scenes! various search results/lookups for media already loaded elsewhere now load super fast!

- tag siblings refresh is quicker and less memory heavy thanks to this as well

- furthermore, the complicated tag changes from tag repository processing and advanced content updates are now reflected immediately in the gui on the job's completion! (as long as you have fewer than 10k files open, ha ha) previously, these required a search refresh to show the results

- the file sort choice dropdown on all pages is now sorted alphabetically. it has always been a mess picking what you want from here, so let's see if this helps!

- tag and rating sort options are now listed as 'tag:' and 'rating:' respectively

- fixed some misc file sort choice code, which was failing to keep certain defaults in certain situations

- fixed the tag import options' new 'load from defaults' button to correctly load the tag blacklist

- the keyboard icon on the media viewer's top hover window now permits activation of current/default shortcut sets under submenus. it now also omits these entries if no custom shortcut sets exist

- cleaned up some of the hover_window-canvas interaction code

- fixed some long-time sperg-out buffer-drawing when changing position in a long video

- the database->backup actions are now hidden if the current db has non-default file/thumbnail locations. for now, in these cases, only a custom backup is appropriate

- fixed some ancient repository admin code that fetches summary account info given an account key

- the filename tagging dialog now has a much shorter listctrl by default, so should fit better on smaller monitors

- fixed the 'review session cookies' dialog's clear button, which was not deleting sessions after clear. it now also wraps the operation in a yes/no confirmation

next week

I am still planning to take about four weeks over Christmas for a big conversion to python 3. This break starts next week, December 12th. I will use this last week just to tidy up, fix any stupid bugs, and write some more help so there is a clean 'final' v334 python 2 build.

The poll for the 'next big thing' is finished, and prototyping a new client API won out. I will start this in the new year and hope to make it an 8-12 week job before starting up a new poll (or possibly going straight on to OR searching, which was a very close second).

R: 0 / I: 0 / P: 9 [R] [G] [-]

Release Tomorrow!

I had a good week. I mostly fixed bugs and wrote some more help, trying to make a clean 'final' python 2 release.

The release should be as normal tomorrow. It will be the last release of the year, as I will be spending the next few weeks updating to python 3.

R: 33 / I: 9 / P: 9 [R] [G] [-]

Version 332

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v332.tar.gz

I had a great week catching up on a whole bunch of small jobs.

rolling session backups

The program now automaticaly makes up to ten rolling backups (i.e. it keeps the ten newest saves) of your sessions! This includes the 'last session' backup used to restore the program on boot, so if you accidentally close something very important and can't get back to it, or if your session breaks for some reason, you can now rewind a bit further.

You can access the backups under pages->sessions->append session backup. They are listed by the time they were made.

The client also now saves an 'exit session' on a clean client shutdown. This is different to the 'last session', which is updated every few (default 5) minutes, and provides a longer-term backup.

This works for the entire in-client serialisation system, so if this works out, it may make sense to apply it to other objects such as the bandwidth tracking system, just to provide an easy additional layer of safety.

misc

There's a bunch of small stuff this week:

Gallery import pages now have a cog button beside the query input that lets you control if new imports start with files and/or search paused! It remembers its values through a session reload.

Tag import options that can be 'default' now have a button to specifically load a default, so if you want to do a specific download that has exactly the normal default options but one little thing different, you can just load it up and edit it rather than having to recreate the whole thing from scratch.

The new tag id cache is expanded, and I've written a similar cache for file ids. A whole bunch of database operations should be just a bit faster.

Export folders can now be told to delete the exported files from the client after they run. (So you can, for instance, tell your client to send all files with 'nsfw' to an import folder for another client.) If you decide to do this, it'll throw up some warnings just to make sure you are ok with it.

When running database->check->file integrity, the client can now optionally export a .txt file with all the missing files' known urls! This same list can be pasted right back into a new url downloader in the client to try to get them back!

full list

- the client serialisation system now supports multiple rolling backups!

- client sessions (like the 'last session' that typically loads on boot) are now automatically backed up to ten times in rolling backups! you can review and append the backups if you need to recover from _pages->sessions->append session backup_, where they are listed by their timestamp!

- when the client closes, an additional 'exit session' is now saved. this differs to 'last session', which is overwritten every x minutes, and is now available (especially with the new backup) for various error recovery situations

- gallery import pages now have a little cog icon to control if new import queues will start with files and/or galleries paused. these states persist through a session reload

- tag import options that are 'defaultable' now have a button to let you load in a specific default, so you can easily quickly edit in a one-time slight alteration of the default rather than having to create everything from scratch

- under options->downloading, you can now set the 'delay' times on gallery/watcher network error, subscription network error, and subscription other error, now defaulting to 90mins, 12hours, and 36 hours respectively

- attempting to launch a client with db version > software version now spawns a blocking messagebox on pre-launch informing the user of the risks and advising task manager force-kill of the process

- did a little cleanup on the new tag id database cache and merged some other, older semi-laggy tag-fetching code to use the same system

- wrote a similar 'file id' database cache for caching file hashes and also merged some old hash-fetching code into it. a variety of file operations are now significantly faster

- export folders can now delete files from the client after export. the edit panel will warn you on selecting this and oking the panel. it can't be set if the export type is 'synchronise'.

- fixed the edit export folders dialog's old buttons, which were semi-working due to some recent update work

- when checking for file integrity, you can now choose to export a .txt file listing all the missing files' known urls, so you can try to recover by feeding them all back into a new url downloader!

- the physical file deletion process is now a little simpler and deals with larger jobs in smaller batches, no longer hogging the file read lock the whole time. clearing a large trash should no longer hang other media loads

- adding ngugs in the 'export downloaders' dialog now attempts to add the respective gugs as well

- fixed an issue where the listctrl would accept certain kinds of duplicate data and hence confuse its indices

- fixed all the add buttons on the export downloader panel to exclude all items already in the list when figuring out what additional objects to add

- the 'scroll thumbs at this rate per tick' option now _rounds_ the pixel result, rather than always _floor_ing it

- wrote a new mass-selection database routine that should reduce memory footprint of autocomplete regeneration. if it works out, I will use it in some other places

- improved the errors when a network job that cannot wait fails on an invalid login, and added a separate error when it refers to a hydrus service

- fixed a small typo bug when trying to auto-add url classes and parsers

- updated an old deprecated checkboxlist call

- clarified the integrity section of 'help my db is broke.txt'.

- misc fixes

next week

Next week should be more small stuff like this. Maybe some fun/new for 333 if I can fit it in. I mostly want to tidy up for a 'python 2 final' 334 on the 12th Dec. On the 12th, I will break up for Christmas to convert the program to python 3.

R: 5 / I: 1 / P: 9 [R] [G] [-]

Release Tomorrow!

I was a little short on hydrus time, but I otherwise had a great week. As well as some normal bug fixes, and another new performance-increasing cache, I've prototyped a fun new 'viewing stats' system.

The release should be as normal tomorrow.

R: 52 / I: 4 / P: 9 [R] [G] [-]

The poll on the next big thing to work on

Due to some funny voting, I am considering the poll finished earlier than expected and will start work on an API prototype in the new year. (details >>10845) Thank you everyone for voting–I really appreciate the feedback.

Here is the poll if you would like to review what was overall popular and not:

https://www.poll-maker.com/poll2148452x73e94E02-60

This thread remains available for discussion of anything related to the poll.

Thanks everyone!

R: 4 / I: 1 / P: 9 [R] [G] [-]

Running from source

I'm having a lot of trouble getting hydrus to run from source.

Could not import lz4.

I have lz4 installed.

Traceback (most recent call last):
File "H:\Hydrus Network\client.py", line 20, in <module>
from include import ClientController
File "H:\Hydrus Network\include\ClientController.py", line 14, in <module>
import ClientCaches
File "H:\Hydrus Network\include\ClientCaches.py", line 1, in <module>
import ClientDefaults
File "H:\Hydrus Network\include\ClientDefaults.py", line 2, in <module>
import ClientData
File "H:\Hydrus Network\include\ClientData.py", line 3, in <module>
import ClientDownloading
File "H:\Hydrus Network\include\ClientDownloading.py", line 12, in <module>
import pafy
File "C:\Python27\lib\site-packages\pafy\__init__.py", line 7, in <module>
from .pafy import new
File "C:\Python27\lib\site-packages\pafy\pafy.py", line 52, in <module>
"pafy: youtube-dl not found; you can use the internal backend by "
ImportError: pafy: youtube-dl not found; you can use the internal backend by setting the environmental variable PAFY_BACKEND to "internal". It is not enabled by default because it is not as well maintained as the youtube-dl backend.

I have youtube-dl installed as well. Setting the PAFY_BACKEND fixes this.

2018/12/01 21:22:43: Traceback (most recent call last):
File "H:\Hydrus Network\include\ClientController.py", line 1241, in THREADBootEverything
self.InitModel()
File "H:\Hydrus Network\include\ClientController.py", line 602, in InitModel
session_manager = self.Read( 'serialisable', HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_SESSION_MANAGER )
File "H:\Hydrus Network\include\HydrusController.py", line 491, in Read
return self._Read( action, *args, **kwargs )
File "H:\Hydrus Network\include\HydrusController.py", line 177, in _Read
result = self.db.Read( action, HC.HIGH_PRIORITY, *args, **kwargs )
File "H:\Hydrus Network\include\HydrusDB.py", line 861, in Read
return job.GetResult()
File "H:\Hydrus Network\include\HydrusData.py", line 1498, in GetResult
raise e
DBException: ImportError: No module named ordered_dict
Database Traceback (most recent call last):
File "H:\Hydrus Network\include\HydrusDB.py", line 527, in _ProcessJob
result = self._Read( action, *args, **kwargs )
File "H:\Hydrus Network\include\ClientDB.py", line 8656, in _Read
elif action == 'serialisable': result = self._GetJSONDump( *args, **kwargs )
File "H:\Hydrus Network\include\ClientDB.py", line 5479, in _GetJSONDump
return HydrusSerialisable.CreateFromSerialisableTuple( ( dump_type, version, serialisable_info ) )
File "H:\Hydrus Network\include\HydrusSerialisable.py", line 129, in CreateFromSerialisableTuple
obj.InitialiseFromSerialisableInfo( version, serialisable_info )
File "H:\Hydrus Network\include\HydrusSerialisable.py", line 206, in InitialiseFromSerialisableInfo
self._InitialiseFromSerialisableInfo( serialisable_info )
File "H:\Hydrus Network\include\ClientNetworkingSessions.py", line 76, in _InitialiseFromSerialisableInfo
session = cPickle.loads( str( pickled_session ) )
ImportError: No module named ordered_dict

I have no idea what to do here. Any help appreciated.

Thanks.

R: 32 / I: 4 / P: 9 [R] [G] [-]

Version 331

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v331/Hydrus.Network.331.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v331/Hydrus.Network.331.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v331/Hydrus.Network.331.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v331/Hydrus.Network.331.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v331/Hydrus.Network.331.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v331.tar.gz

I had a good week. The login manager is done, I have added proxy support, and some tag stuff is faster!

login manager

The 'manage logins' dialog has some small usability improvements, and it also has a neat 'do login now' button to easily manually attempt logins.

The network->data->review session cookies panel now hides 'empty' sessions by default and allows for cookies.txt drag-and-drop import!

And login scripts can now be rolled into the 'easy import' downloader pngs! They get listed like all the other objects.

With these last jobs, I have finished my last big jobs for the login manager. I would still like to write some decent help, and I'll fold new login scripts into regular updates as users create and share them, but this phase's bulky work is all complete. Sites with captcha and other complicated login systems (like exhentai.org) are not supported in this first version (so you'll have to manage without, or use manual cookies.txt import from your browser), but I am overall really happy with how it has gone.

proxy support

A user reminded me last week to add proxy support back in. This was an important objective of the big network engine rewrite (I had previously removed my tentative support as the old engine became too convoluted to support), and adding it this week couldn't have been smoother–the 'requests' library for python is just great. If you have an http or socks4/socks5 proxy you would like to use, please go to options->connection and fill in the simple new options. It has some text to explain it (including how to add socks support if you are running from source and lack the right library).

This is client-wide for now, so if you set a proxy, all client requests will go through it. Let me know if this causes any problems. I may revisit this and have it work on a per-domain basis, like how bandwidth rules work.

other

Searches that include 'system:num_tags' are now much faster and have accurate tag counts even when multiple services share the same tag! This is still a pretty CPU-expensive predicate, so I recommend you reduce the search domain by including a tag or a simple system predicate like 'system:size', but it should nonetheless run a lot quicker. I wrote a new and unusual query to pull this off, so please let me know if it fails in some situations. If it works well, I will use it elsewhere.

I have also started an extremely basic tag cache for the client db. It makes autocomplete results and file search results appear significantly faster in many situations. Like the new system:num_tags query, I am going to play with it a bit on my IRL client, see if there is any 'I noticed the difference in a good/bad way' feedback, and likely do a similar cache for file data and iterate on them a bit.

There are better and more "show all these things' files" menu entries on the gallery import and thread watcher management panels' lists' right-click menus. You can now choose between showing the 'presented', 'new', and 'all' files, and they now show more quickly in the current page.

I updated last week's e-hentai.org login script to one that works better, and I added a 4channel.org thread url class to permit watching the soon-to-be-separated sfw 4chan boards.

full list

- added a 'do login now' button to the manage logins dialog. it only enables when the selected logins are active and not invalid and so on, and will ok the dialog and queue up some login attempts, which will make report popups as normal

- 'review session cookies' panels now support drag and drop cookies.txt import! cookies.txt importing will also handle errors a bit better and report total number of added cookies

- the 'review session cookies' panel now defaults to not showing sessions with zero cookies. a new checkbox controls this

- login scripts can now be rolled into easy import pngs! should work for export and import just like the other objects (although they won't be auto-added based on domain in export dialog)

- brushed up some of the 'change login script' code–particularly, it now puts login scripts that have matching domains first in the selection list, for easier selection

- after striking a reckless bargain with a daemon from the database-plane, system:num_tags now runs significantly faster and produces accurate tag counts even when searching over multiple tag services that have duplicate tags. if this works out, the immaterial beast promises greater gains for similar jobs with no possibility of anything going wrong

- prototyped a new tag cache in the db that affects (and should speed up) many tag fetching routines. let's see how it goes

- added complete, global proxy support for the new network engine! there are new options under options->connection (with some explanation text) to handle it. if pysocks is installed, socks4/5 proxies are also available!

- updated the e-hentai.org login script to the new one on the github. your existing mappings for e-hentai.org _should_ all be updated right. exhentai.org is likely too difficult to properly support in the current system

- the different panels where you enter system predicate information now all run on the new panel sizing system–if you have had problems with these, please let me know how they size now!

- added a '4channel thread' url class to support watchers for the new 4channel sfw domain. it works for now, but let's see if their api changes when the split actually happens

- the list right-click menu on gallery import and thread watcher panels now has three options to show combined importers' files–presented, new, and all. it also now shows the files (more smoothly) in the same page, clearing any existing highlight.

- misc ui improvements

- updated 'running from source' in help

next week

I have three more weeks until I break up for the holiday and start the big python 3 rewrite. With the login manager work done, I am now just going to catch up on small jobs. I'd like to focus on some code cleanup, particularly, and maybe untangle some hellish db siblings/parents code. If you missed the 'next big thing' poll last week, please check it out here: >>10654

R: 4 / I: 1 / P: 10 [R] [G] [-]

you guys know that there are people who actually use hydrus for searching their files instead of just downloading every trash they can find? so since people exist who want to actually use hydrus, it's way more useful to have or-search instead of a shitty api that only retards would benefit from (that's not true but or-search is easier to implement and IT IS USEFUL.. especially if you use your own tagging style instead of the bs ptr tagging system which is just stupid imo) - end of discussion

R: 0 / I: 0 / P: 10 [R] [G] [-]

Release Tomorrow!

I had a great week catching up on a whole bunch of small work. There's some faster search, better error recovery, a neat new rolling session backup system, and plenty more.

The release should be as normal tomorrow.

R: 26 / I: 5 / P: 10 [R] [G] [-]

Version 330

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v330/Hydrus.Network.330.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v330/Hydrus.Network.330.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v330/Hydrus.Network.330.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v330/Hydrus.Network.330.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v330/Hydrus.Network.330.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v330.tar.gz

I had a great week. There are some more login scripts and a bit of cleanup and speed-up.

The poll for what big thing I will work on next is up! Here is the thread: >>10654

login stuff

The new 'manage logins' dialog is easier to work with. It now shows when it thinks a login will expire, permits you to enter 'empty' credentials if you want to reset/clear a domain, and has a 'scrub invalid' button to reset a login that fails due to server error or similar.

After tweaking for the problem I discovered last week, I was able to write a login script for hentai foundry that uses username and pass. It should inherit the filter settings in your user profile, so you can now easily exclude the things you don't like! (the click-through login, which hydrus has been doing for ages, sets the filters to allow everything every time it works) Just go into manage logins, change the login script for www.hentai-foundry.com to the new login script, and put in some (throwaway) credentials, and you should be good to go.

I am also rolling out login scripts for shimmie, sankaku, and e-hentai, thanks to Cuddlebear (and possibly other users) on the github (which, reminder, is here: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System ).

Pixiv seem to be changing some of their login rules, as many NSFW images now work for a logged-out hydrus client. The pixiv parser handles 'you need to be logged in' failures more gracefully, but I am not sure if that even happens any more! In any case, if you discover some class of pixiv URLs are giving you 'ignored' results because you are not logged in, please let me know the details.

Also, the Deviant Art parser can now fetch a sometimes-there larger version of images and only pulls from the download button (which is the 'true' best, when it is available) if it looks like an image. It should no longer download 140MB zips of brushes!

other stuff

Some kinds of tag searches (usually those on clients with large inboxes) should now be much faster!

Repository processing should also be faster, although I am interested in how it goes for different users. If you are on an HDD or have otherwise seen slow tag rows/s, please let me know if you notice a difference this week, for better or worse. The new system essentially opens the 'new tags m8' firehose pretty wide, but if that pressure is a problem for some people, I'll give it a more adaptable nozzle.

Many of the various 'select from a list of texts' dialogs across the program will now size themselves bigger if they can. This means, for example, that the gallery selector should now show everything in one go! The manage import/export folder dialogs are also moved to the new panel system, so if you have had trouble with these and a small screen, let me know how it looks for you now.

The duplicate filter page now has a button to edit your various duplicate merge options. The small button on the viewer was too-easily missed, so this should make it a bit easier!

full list

- login:

- added a proper username/password login script for hentai foundry–double-check your hf filters are set how you want in your profile, and your hydrus should inherit the same rules

- fixed the gelbooru login script from last week, which typoed safebooru.com instead of .org

- fixed the pixiv login 'link' to correctly say nsfw rather than everything, which wasn't going through last week right

- improved the pixiv file page api parser to veto on 'could not access nsfw due to not logged in' status, although in further testing, this state seems to be rarer than previously/completely gone

- added login scripts from the github for shimmie, sankaku, and e-hentai–thanks to Cuddlebear and any other users who helped put these together

- added safebooru.donmai.us to danbooru login

- improved the deviant art file page parser to get the 'full' embedded image link at higher preference than the standard embed, and only get the 'download' button if it looks like an image (hence, deviant art should stop getting 140MB brush zips!)

- the manage logins panel now says when a login is expected to expire

- the manage logins dialog now has a 'scrub invalidity' button to 'try again' a login that broke due to server error or similar

- entering blank/invalid credentials is now permitted in the manage logins panel, and if entered on an 'active' domain, it will additionally deactivate it automatically

- the manage logins panel is better at figuring out and updating validity after changes

- the 'required cookies' in login scripts and steps now use string match names! hence, dynamically named cookies can now be checked! all existing checks are updated to fixed-string string matches

- improved some cookie lookup code

- improved some login manager script-updating code

- deleted all the old legacy login code

- misc login ui cleanup and fixes

- .

- other:

- sped up tag searches in certain situations (usually huge inbox) by using a different optimisation

- increased the repository mappings processing chunk size from 1k to 50k, which greatly increases processing in certain situations. let's see how it goes for different users–I may revisit the pipeline here to make it more flexible for faster and slower hard drives

- many of the 'select from a list of texts' dialogs–such as when you select a gallery to download from–are now on the new panel system. the list will grow and shrink depending on its length and available screen real estate

- .

- misc:

- extended my new dialog panel code so it can ask a question before an OK happens

- fixed an issue with scanning through videos that have non-integer frame-counts due to previous misparsing

- fixed a issue where file import objects that have been removed from the list but were still lingering on the list ui were not rendering their (invalid) index correctly

- when export folders fail to do their work, the error is now presented in a better way and all export folders are paused

- fixed an issue where the export files dialog could not boot if the most previous export phrase was invalid

- the duplicate filter page now has a button to more easily edit the default merge options

- increased the sibling/parent refresh delay for 1s to 8s

- hydrus repository sync fails due to network login issues or manual network user cancel will now be caught properly and a reasonable delay added

- additional errors on repository sync will cause a reasonable delay on future work but still elevate the error

- converted import folder management ui to the new panel system

- refactored import folder ui code to ClientGUIImport.py

- converted export folder management ui to the new panel system

- refactored export folder ui code to the new ClientGUIExport.py

- refactored manual file export ui code to ClientGUIExport.py

- deleted some very old imageboard dumping management code

- deleted some very old contact management code

- did a little prep work for some 'show background image behind thumbs', including the start of a bitmap manager. I'll give it another go later

next week

I have about eight jobs left on the login manager, which is mostly a manual 'do login now' button on manage logins and some help on how to use and make in the system. I feel good about it overall and am thankful it didn't explode completely. Beyond finishing this off, I plan to continue doing small work like ui improvement and cleanup until the 12th December, when I will take about four weeks off over the holiday to update to python 3. In the new year, I will begin work on what gets voted on in the poll.

R: 0 / I: 0 / P: 10 [R] [G] [-]

Release Tomorrow!

I had a good week. The login manager is finished–with some better login testing and script-sharing–and I significantly sped up some database tag code.

The release should be as normal tomorrow.

R: 28 / I: 4 / P: 10 [R] [G] [-]

Version 328

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v328/Hydrus.Network.328.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v328/Hydrus.Network.328.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v328/Hydrus.Network.328.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v328/Hydrus.Network.328.-.OS.X.-.Extract.only.tar.gz

linux

https://github.com/hydrusnetwork/hydrus/releases/download/v328/Hydrus.Network.328.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v328.tar.gz

I had a good week. The login manager is 95% done, and I got some neat ui stuff finished as well.

I am now considering what 'big thing' to work on next. Please see 'next week' below for more details.

login manager

The login manager is first-draft finished. It works and is turned on, but I have not added any new scripts, and I have yet to remove the hardcoded pixiv and hentai foundry login, so I can do some final testing. I would like advanced users to play with making new scripts and seeing where they fail, as I am not totally confident I have the 'this login messed up, what to do now?' error handling fixed yet.

I will also play with it next week, and assuming I can iron out any big problems, I will remove the hardcoded stuff and start adding login scripts for the existing big sites and some new ones like FurAffinity.

You may get a new error message this week if a HF or pixiv login fails. It may pause a related subscription as part of that. Please let me know how this works for you.

ui stuff

The options dialog now has a 'thumbnails' page, which collects several thumbnail related options. It also now has 'thumbnail border' and 'thumbnail margin'! If you would like a thicker–or thinner; it should even support 0 pixels for both–spacing between your thumbs, please check it out.

The manage tags dialog has had a layout pass. The clunky checkboxes and advanced buttons are collected into a cog icon, and the copy/paste/remove buttons are put into a single line. The whole thing also uses my new sizer, so as well as being cleaner and more compactable, it now expands better as well.

Manage tag siblings and parents now has reason 'suggestion' buttons when you petition a tag repository! Using these pre-defined reasons are helpful for the janitor of the repo (for the PTR, this is me, hydrus dev), as it collects all the same kind of petitions together into a single actionable chunk. If you regularly make sibling and parent petitions to the PTR, please give these buttons a go. It should save everyone time.

The network->pause menu now has some checkbox entries to pause file import queues, gallery searches, and watcher checking globally. This is a little ugly (the downloaders won't say this cleverly in their statuses, and they may take up to 30s to resume after being unpaused from this), but it works well as an temporary emergency stop-gap. I also expect to add an option to gallery and watcher pages in the coming weeks to start file import queues in a paused state by default.

full list

- wrote test ui for edit login script panel

- the login system now works and is turned on, although the legacy hardcoded pixiv and hf logins remain in place. it will not do anything very new this week–it is strictly only for advanced users to experiment with for now

- cleaned up some messy network code

- all subscription and hydrus jobs will no longer wait indefinitely on an invalid login–they will cancel immediately

- network jobs will report a bit more info when they are cancelled

- subscriptions will now attempt to test login validity before and during file downloads and syncing. if they fail, the sub will pause and stop work and a message will be presented to the user

- made a 'thumbnails' options page and moved some things to it

- added thumbnail border and margin to that thumbnails page! you can even set 0 border and/or margin and it works

- fixed up a heap of bad thumbnail drawing code that didn't work with thicker borders

- the tag sibling and parent dialogs now have suggestion buttons in their 'give a reason' dialogs! if your petitions are simple and fit into one of these categories (which is most of them), please use these buttons as they will let janitors (e.g. hydrus dev for the PTR) process them in batches, in fewer clicks

- manage tags dialog's checkboxes and advanced buttons are now wrapped into a cog icon! remove/copy/paste buttons are compacted and put on the same row!

- manage tags dialog's copy button now only copies selected, not all tags

- manage tags dialog now uses the new sizer. some components are smaller by default but will eat up spare pixels better

- misc manage tags code cleanup

- added 'paged file import queues' to the network->pause menu. this will pause any, hdd, url, simple, gallery, or watcher page from processing its file import queue. it is a bit hacky and will take up to 30s to unpause unless you joggle the respective downloader to wake it up–see how it goes!

- added a similar 'gallery searching' to that menu, to pause any url, simple, or gallery page from actioning its gallery-side queue

- and 'watcher checking' for watchers

- fixed a stupid bug that was causing false-positive and _mostly_ harmless errors for certain pixiv and artstation multi-page downloads

- fixed an issue where multi-page data was being mis-metadata'd (mostly, this meant thread watchers were giving the last filename tag to all files, and I think similarly getting the latest source time) due to a mistake in some recent de-duping code

- fixed the new pixiv file page api parser to parse source time, which I must have accidentally deleted previously

- fixed a no-expand bug in my new sizer when in horizontal orientation

- fixed a small bug when making an easy-import downloader png and cancelling the add url class dialog

- misc fixes

next week

The login manager work is basically done. I need to iron out some last error situations so I am happy, and then I can write/integrate some new login scripts in my normal work time. I am thus 'finished' with my big work focus and very happy to start something new. My current plan is to spend mid-December to mid-January doing the python 3 update, which leaves a decent bit of time to have a discussion about what to work on next and put a poll up for voting. Please check out the thread at >>10429 for the current list.

R: 3 / I: 0 / P: 10 [R] [G] [-]

Version list

Hey do we have a compiled list of every version and it's features?

I haven't updated in AGES and I imagine there are a lot of new things, but I don't want to dig through every individual OP to find out.

Can you make a compiled list of versions with major changes, kindof like lists of minecraft updates as an example, where each version will broadly list "bug fixes" or "added desert biome" longer explanations for larger feature updates.

Or even, can that happen going foreword somehow?

R: 42 / I: 12 / P: 10 [R] [G] [-]

Reposting from >>>/tech/457184

Maybe relevant to your interest.

Japan did it.

Automatic anime images tagging with neural networks.

THE FUTURE IS NOW

http://illustration2vec.net/

http://illustration2vec.net/papers/illustration2vec-main.pdf

R: 105 / I: 13 / P: 10 [R] [G] [-]

Big things to work on next

With the download engine and login manager coming to a close, I will need something new to be anxious about and near-overwhelmed by. I will put up a poll in a few weeks for everyone to vote on a big list of possible new features that are too large to fit into my normal weekly work. The poll will allow you to vote on multiple items. I hope to work on the most voted-on item for two to three months before starting the cycle again.

This thread is for discussion of the list, which at current looks like this:

- Just catch up on small work for a couple of months

- Improve tag siblings/parents and tag censorship

- Reduce crashes and ui jitter and hanging by improving ui-db async code

- Speed up tagging workflow and autocomplete results

- Add ways to display files in ways other than thumbnails (like 'details' view in file explorers)

- Add text and html support

- Add Ugoira support (including optional mp4/webm conversion)

- Add CBZ/CBR support (including framework for multi-page format)

- Add import any file support (giving it 'unknown' mime)

- Improve 'known urls' searching and management

- Explore a prototype for neural net auto-tagging

- Add support for playing audio for audio and video files

- Add OR file search logic

- Add an interface for waifu2x and other file converters/processors

- Write some ui to allow selecting thumbnails with a dragged bounding box

- Add popular/favourite tag cloud controls for better 'browsing' search

- Improve the client's local booru

- Improve duplicate db storage and filter workflow (need this first before alternate files support)

- Improve shortcut customisation, including mouse shortcuts

- Import/export ratings, and add 'rating import options' to auto-rate imports

- Add more commands to the undo system

- Improve display of very large/zoomed files in the media viewer

- Set thumbnail border colours on user-editable rating and namespace conditions

- Improve hydrus network encryption with client cert management and associated ui

- Add tag metadata (private sort order, presentation options, tag description/wiki support)

- Write a repository-client refresh/resync routine to clear out junk data and save space

- Prototype a client api for external scripts/programs to access

- Support streaming file search results (rather than loading them all at once once the whole query is done)

- Increase thumbnail size limit (currently 200x200)

- Add an optional system to record why files are being deleted

- Improve file lookup scripts and add mass auto-lookup

- Cleanup code and improve practises

- Add multiple local file services

- Add an incremental number tagging dialog for thumbnails

I am happy to work on any of these items. If you have questions, please ask, and if you have suggestions for new items, go ahead.

R: 30 / I: 5 / P: 10 [R] [G] [-]

Version 329

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v329/Hydrus.Network.329.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v329/Hydrus.Network.329.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v329/Hydrus.Network.329.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v329/Hydrus.Network.329.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v329/Hydrus.Network.329.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v329.tar.gz

I had a great week. The login manager is now ready for everyone, and I did some neat other stuff as well.

login manager

I have fixed up how the login manager deals with various errors and am happy for everyone to give it a go. This release also introduces login scripts for Deviant Art, Danbooru, and the Gelbooru sites. >Please check it all out under network->downloaders->READY FOR CAUTIOUS USE: manage logins. This ui needs a bit more work to make it user-friendly, but you basically enter some credentials (username & password) for a domain and then 'activate' the login script for it. Thereafter, the client will try to log in to that site whenever it needs something from it and doesn't think it is logged in. It will make a little popup window while it logs in just to let you how it is doing. If it fails, it will 'block' that domain, and any network jobs will hang on it and any subs will pause on it. Most logins last from 30 days to five years, so after the initial success, you likely won't see it ever again, particularly if you use subscriptions. It seems to work ok!

That said, feedback would be appreciated, particularly for the individual scripts. If you discover the client does not notice when it fails to log in, or fails to notice when it has naturally been logged out, or keeps reattempting a subscription that has an invalid login or something, please let me know.

Caveats:

As it says on the dialog: With hydrus, use throwaway accounts or accounts you otherwise do not care much about. The credentials are stored in plaintext, and if something goes wrong with the login system or you binge one night and download too much, you don't want to get a black mark on an account you care about.

The deviant art login page only seems to work if your client has some history of downloading from Deviant Art previously. Otherwise, it puts up a 'Are you a robot?' captcha challenge, which this first login manager cannot deal with. If you have been downloading from Deviant Art for a while, you should be fine (otherwise, try downloading a few SFW files first and then try activating the login). The DA parser is also updated to correctly download NSFW images (which you will have access to once logged in).

I don't know when this happened, but it seems that Pixiv now also allow guests to access SFW images. This is a neat development, but it means my login script as-is doesn't detect when it fails to log in (since pixiv now give you a 'guest' login regardless). If you have had trouble getting NSFW pixiv images and can't figure it out, please check your credentials are all correct and try deleting your cookies on the login dialog to try again. I will revisit the script next week to see if I can better detect a failed login.

I looked at FurAffinity this week, since it has been highly requested, but found they require a captcha even on regular login attempts (and not, say, just sign-ups). This throws a wrench in our plans, as this first version cannot deal with captcha. Unless there is a simpler alternate login system, there is likely no good solution here for now, and FA users will have to copy cookies across from their browser. (which I still intend to make more user-friendly and reliable in the coming weeks)

I had hoped to roll out a Hentai Foundry login this week, but it turns out their login cookies is dynamically named, so I will have to tweak the login manager to deal with it. Not a huge issue, but it will be delayed a little. The click-through guest login is now working entirely on the new system and appears to be fine.

other stuff

Texts in the program that ellipsize (Like this…) when they are too short now present tooltips with the full text. This includes the status on the network job control, which will help you review any long login errors.

The manage tags dialog's buttons get a bit more tweaking this week: both the remove and copy buttons now work differently depending on whether any tags above are selected or not: if tags are selected, only those will be affected; if none are selected, all will be affected. Also, the remove button is now wrapped in a yes/no dialog that can be turned off in the cog menu.

The filename tagging options panel now allows you to additionally tag files by their last, second last, and third last directory names!

The export files panel now allows you to delete the files from the client after export. This checkbox value is remembered and presents a WARNING yes/no dialog on export just to make sure.

Manage tag siblings and parents now handle large 'add' batches as one transaction, whether that is by a big .txt/clipboard import or when adding multiple parents. Everything is now bundled into one big list and you are only asked for the pend 'reason' once.

full list

- login:

- the login manager is fully turned on! hentai-foundry click-through and pixiv login now occur fully on the new system

- wrote a Deviant Art login script for NSFW downloading–however, it only seems to work on a client that has done some logged-out downloading first (otherwise it thinks you are a robot)

- updated the DA file page parser to only NSFW-veto if the user is currently logged out

- wrote a danbooru login script for user prefs and special files if you have a gold account

- wrote a gelbooru 0.2.x login script for user prefs

- pixiv recently(?) allowed non-logged in users to see sfw content, so the login script is updated to reflect this. the login script doesn't detect a failed login any more, so I will revisit this

- logging in in the regular order of things now makes a temporary popup message with the overall login status and final result. ~it is cancellable~–and if cancelled, future login attempts will be delayed

- logging in in the regular order of things now prints simple started/result lines to the log

- deleted old network->login menu and related code such as the custom pixiv login management. gdpr click-through is now under downloaders

- subscription login errors will now specify the given login failure reason

- subscription login tests will now occur at a better time, guaranteeing the sub will be correctly saved paused if the test fails

- login errors will now always specify the domain for which they failed

- testing a login script on a fresh edit login script dialog now pre-fills the alphabetically first example domain

- the login script test ui now restores its 'run test' button correctly if the test is abandoned early

- misc improvements to login error handling and reporting

- .

- other:

- any texts across the program that ellipsize when they are too thin to display what they have will now tooltip their text (this most importantly includes the status on the network job control, which will now display full login problem info)

- the copy button on manage tags goes back to copying all if no tags are selected

- the remove button on manage tags now removes only selected if some tags are selected. it still removes all if none are selected

- the remove button on manage tags is now wrapped in a yes/no dialog (as is hitting the delete key on the list's selection). this can be turned off under the cog button

- filename tagging panels now support directory tagging for the last, second last, and third last directories. the related code for handling directory tagging is cleaned up significantly

- the export files panel now lets you delete the files from the client after export. this value will be remembered, and if on will prompt a capital letters warning on export, either via the button or the quick-export shortcut

- in manage tag parents, where there are multiple parents in a pending action (either by importing via clipboard/file or by putting multiple parents in right-hand box), the action will now be treated as one transaction with one 'enter a reason' confirmation!

- in manage tag siblings, when multiple 'better' values are pended in one action via a clipboard/file import, they will now be treated as one transaction with one 'enter a reason' confirmation!

- .

- misc:

- added a new url class that api-links .gifv-style imgur links so they are downloadable like regular imgur single media pages

- the pixiv manga page url class now redirects to the new api, so mode=manga pages should now be drag-and-drop importable and generally downloadable if you have any still hanging around in any queues

- clients now come with an additional danbooru parser that fetches the webm version of ugoiras

- after discovering a pdf that ate indefinite 100% CPU while trying to parse, I have decided to stop pulling num_words for pdfs. it was always a super inaccurate number, so let's wait for a better solution at a later date. hydrus hence no longer requires pypdf2

- fixed an issue with monthly bandwidth estimates rolling over to the new year incorrectly

- in an attempt to chase down a duplicate files content move/copy bug, the duplicate action content updates got a bit of cleanup work. if you have noticed duplicate actions not copying tags/urls, please let me know the exact process in the ui, including services and merge options, you went through

- tag lists should now update their sibling appearance correctly after a tag siblings dialog ok–previously, they were checking for new sibs too early

- tag siblings and parents should now refresh their data more efficiently when sp