[ / / / / / / / / / / / / / ] [ dir / random / 93 / biohzrd / hkacade / hkpnd / tct / utd / uy / yebalnia ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Name
Email
Subject
REC
STOP
Comment *
File
Password (Randomized for file and post deletion; you may also set your own.)
Archive
* = required field[▶Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options

Allowed file types:jpg, jpeg, gif, png, webp,webm, mp4, mov, swf, pdf
Max filesize is16 MB.
Max image dimensions are15000 x15000.
You may upload5 per post.


This board will be deleted next Wednesday. I am moving to a General on 8chan.moe /t/. This board is archived at 8chan.moe /hydrus/!

YouTube embed. Click thumbnail to play.

e5eb10 No.14985

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v421/Hydrus.Network.421.-.Linux.-.Executable.tar.gz

I had a good week. I fixed some important bugs, and cleaned up some core access and maintenance code. This week's changelog gets pretty technical, which you can safely ignore. Database works better™.

highlights

I fixed a problem importing files with 'only add tags that already exist' checked in tag import options. Sorry, this was a stupid typo. I added a unit test here to ensure it doesn't happen again.

When you search for potential duplicates from the duplicates page, this now happens in a non-interrupting popup in the bottom-right. You can keep searching and browsing while it works. All duplicate pages sync with each other better, too, and share more CPU work.

Thanks to a user's contribution and help, the 'discord drag and drop' BUGFIX mode, which is really a 'export files to a temp location before starting the drag and drop' mode, now supports the same 'filename pattern' you see in the export files dialog. Check it under options->gui. This is experimental, but the first step to automatic export with nice filenames. If you care about export filenames, let me know how it works for you, and what sorts of filenames you want to make in future. I really want to update the old 'filename pattern' system to something with conditional 'if' statements.

If you are a new user and had trouble syncing to the PTR in the past week with an error about an invalid tag, you got caught by my new 'invalid tag catching' code being overzealous. This is now fixed, so if your PTR automatically paused due to the error, please hit services->review services->PTR and then unpause it. When a repository gives an invalid tag (we think there are about 17 really broken tags in the PTR atm, from older days), the client now recovers.

I played around with core database modes this week. Default values are now tuned a little better for modern use. Unfortunately, the planned new mode for HDD users did not pan out due to architectural problems, but I think that HDD users will experience better write performance nonetheless. For those who want to experiment more, I have finally properly documented the different launch switches for hydrus here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html

full list

- misc:

- thanks to a user's contribution, added the export 'filename pattern' to the discord drag and drop mode, under _options->gui_. this lets you auto-rename files in this export mode. I like how this works, but the overall pattern-based filename creation system really needs updating. let me know how this works for you, and I'll finally start the job to update filename generation

- fixed a bug when importing files with the 'only add tags that already exist' filter active, and added a unit test so this should not fail due to a typo again

- fixed an issue where ctrl-selecting on taglists was weird, where any mouse movement during ctrl+click would deselect. drag select and deselect can now only start when the drag crosses two indices

- prototyped a basic profile mode for the client api. it is insufficient (due to the asynchronous nature of twisted), but a start

- when the client catches an invalid tag with the new error handling code, when it shows you that bad tag in a popup, it now clips that to 24 characters (some PTR invalid tags are just a few hundred null characters in a row, wew lad)

- the client now recovers from a repository giving it a new invalid tag definition. all such tags are, for now, called 'invalid repository tag'. a plan to auto-hide these tags clientside and fully eliminate them serverside will come later

- the clipboard url watcher settings should stick a bit more firmly. those users who had trouble, please let me know how you get on

- fixed an issue editing duplicate action options when they contained tag or rating preferences for services that no longer exist

- I think I fixed some issues getting autocomplete results when you type the whole namespace before moving on to the subtag. when you hit 'namespace:', it should invalidate the old cache and start a new search

- when the database is given content updates for services that no longer exist, those content updates filtered out of UI update broadcast

- fixed an issue where URL status check could fail when the url map contained orphan hash_ids. proper orphan clearance will come later

- reduced overhead of tag filtering, which should improve display speed of taglist for very large pages

- parents should now work through repository processing faster. periods of 2 rows/s at the end up of updates should be up to 100 times faster

- .

- duplicates search improvements:

- potential duplicate search now works in the background! it will not interrupt you and is easily cancellable. duplicate search pages disable their search buttons while it is going

- the search distance in duplicates pages is now synchronised across all pages–when one updates, they all do

- all the updates to potential search maintenance numbers are now routed through one cached manager. updates here are repeated less often

- misc cleanup for duplicates page

- .

- database modes:

- a new 'program launch arguments' help page now talks about all the available command line switches, here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html

- added the '–db_journal_mode' launch switch to set the SQLite journal mode. default is WAL, permitted values are also TRUNCATE, PERSIST, and MEMORY

- ensured –db_synchronous_override was hooked up correctly

- the old disk cache options under _speed and memory_ are removed, along with various deprecated disk cache load calls and code

- fixed some shutdown maintenance check logic that was saying 'I think a vacuum is due' when it wasn't actually true

- db_journal_mode, synchronous value, and no_db_temp_files is now shown in _help->about_

- .

- technical database nonsense:

- PERSIST is new to hydrus, and _may_ in future versions of SQLite be boost performance for HDD drives with larger databases (e.g. those that sync to the PTR), although unfortunately in our case (which uses multiple ATTACH databases), it seems current SQLite must ultimately treat this as DELETE, as here https://sqlite.org/atomiccommit.html#_clean_up_the_rollback_journals. damn

- hydrus now tries to always trim WAL (and PERSIST, if it worked) journal files down to 1GB after commits (which happen every 30 seconds), so giganto WALs should clear up promptly after big work is done

- hydrus no longer refreshes the database connection every thirty minutes, meaning WAL journal files will persist (and hopefully regularly clip back to 1GB when exceeded), which should improve some elements of long-running write performance, but may result in some surprise memory issues, we'll see

- in lieu of the db connection not refreshing, the memory database now reattaches every ten minutes, which _should_ stop it leaking in certain situations

- when in WAL journal mode, the hydrus db now cleans up any lingering checkpointing work every half hour

- after testing and feedback from users, the database is now default SQLite synchronous 1 (down from 2) when in WAL. the db is still consistent, so sudden program stop (crash, power cut) should not result in software-caused corruption, but the database may lose more than just the last 30 seconds of work. this speeds up tag processing in an SSD test environment by approx 33%

- the 'no_wal' (TRUNCATE) and 'db_memory_journaling' (MEMORY) launch switches remain valid but are now deprecated

- improved launch switch code generally

- boosted cache size for each of the four db files to ~200MB-this will likely become a launch argument in future, along with some other specific db values

- the client and server no longer disconnect from the db to check whether it is possible to vacuum databases

next week

There are two more work weeks in the year. I will continue working on small jobs and cleanup, but the main focus is now to update ancient network and service code to improve client-PTR sync and communications. I'll fix up buggy janitor tools and hopefully add some nice filters so clients and servers can better manage what they sync.

Since a 'next big work' poll is coming up on Christmas, on Saturday I'll post here and in discord about the current big list and ask for more things to add.

____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

10e024 No.14990

Hey, here are the preliminary 'big job poll' items. This poll will go up on December 23rd with my release posts, for the new year. You will be able to vote on multiple items. Please feel free to ask questions and request big jobs be added to it.

As many issues are being handled by the github issue team, here is a neat link that lets you review the larger in more detail:

https://github.com/hydrusnetwork/hydrus/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc+milestone%3A%22Major+jobs%22

The poll:

Add CBZ/CBR support (including framework for multi-page format)

Add Ugoira support (including optional mp4/webm conversion)

Add a carousel for the media viewer to see previous/next files in the current navigation thing

Add an incremental number tagging dialog for thumbnails (for adding page:n etc… to a sequence of files)

Add animated thumbnails for videos (animating on mouseover)

Add blurhash for quick thumbnail presentation

Add duplicate video detection

Add file note parsing to the download system

Add import any file support (giving it 'unknown' mime but preserving file extension)

Add more commands to the undo system

Add multiple local file services (which will enable true nsfw/sfw partition)

Add popular/favourite tag cloud controls for better 'browsing' search

Add ratings import/export, and add 'rating import options' to auto-rate imports

Add tag metadata (private sort order, presentation options, tag description/wiki support)

Add text and html support

Add ui for waifu2x and other file converters/processors

Add version tracking to downloader system objects and explore remote fetching of updates

Add ways to display files in ways other than thumbnails (like 'details' view in file explorers)

Allow for more UI layout customisation

Allow multiple custom 'open externally'-style file launch commands for files

Allow taglists to collapse by namespace

Clean up code and add unit tests

Expand file modified timestamp storage and update

Focus on adding commands to the Client API

Improve 'known urls' searching and management

Improve display of very large/zoomed files in the media viewer

Improve duplicate files display and workflow, and start automatic duplicate resolution

Improve file lookup scripts and add mass auto-lookup

Improve rating workflow by providing score representatives to compare with

Improve shortcuts support

Just catch up on small work for a couple of months

Permit custom ordering of thumbnails, through mouse-dragging or otherwise

Set thumbnail border colours on user-editable rating and namespace conditions

Start on file alternates metadata, including file groups

Unify theming experience

Write an URL Repository so clients can share known url mappings

Write some ui to allow selecting thumbnails with a dragged bounding box

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

2b88bc No.14994

Hi dev.

>Add file note parsing to the download system

Adding to it, would you add the note parsing function when importing local files?

Thanks!

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

92ca14 No.14995

Did this update touch wildcards at all? copying a namespace:*anything* query only copies the namespace again when you fixed this previously, and 'namespace:*' doesnt bring up autocompletion results like I was expecting

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

10e024 No.15002

>>14994

That's the hope, at some point. I want to unify the local and remote file import metadata pipelines more, not only to add features but also to reduce maintenance and expansion work. My dream is that you'll one day be able to apply parsing tech to sidecar files like neighbouring .txt files and just pull whatever the hell data you want out of json or xml files.

I do not know when notes will be able to do this, but first steps are to get .txt working for import/export folders and have the filename->tagging dialog work with the new string converter system instead of my ancient hardcoded regex input fields.

>>14995

I am sorry, 'namespace:*' and friends seem to have been broken significantly as of about v419 when I did some 'optimisation' work to fetch regular results faster. I have fixed it all today and added specific unit tests so these advanced searches can't be lost due to typo or ignorance again. Please let me know how 422 works for you.

Can you say what you want copied when you copy a 'namespace:*anything*'? I am sorry that I have forgotten about this issue.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

92ca14 No.15004

>>15002

No problem; when copying a tag like "character:*anything*" I was expecting to have a "character:*" result I could copy and paste in a different search

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

e2f56d No.15007

>>15004

Thanks, that makes more sense. I'll change it to that and make a note not to change it.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

e2f56d No.15010

I had a great week. I mostly fixed bugs, including the recently broken 'namespace:*' autocomplete lookups, extended database recovery code, improved some quality of life, and added some advanced tools for downloader makers.

The release should be as normal tomorrow.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.



[Return][Go to top][Catalog][Nerve Center][Random][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / / / / / ] [ dir / random / 93 / biohzrd / hkacade / hkpnd / tct / utd / uy / yebalnia ]