[ / / / / / / / / / / / / / ] [ dir / random / 93 / biohzrd / hkacade / hkpnd / tct / utd / uy / yebalnia ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Name
Email
Subject
REC
STOP
Comment *
File
Password (Randomized for file and post deletion; you may also set your own.)
Archive
* = required field[▶Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options

Allowed file types:jpg, jpeg, gif, png, webp,webm, mp4, mov, swf, pdf
Max filesize is16 MB.
Max image dimensions are15000 x15000.
You may upload5 per post.


This board will be deleted next Wednesday. I am moving to a General on 8chan.moe /t/. This board is archived at 8chan.moe /hydrus/!

YouTube embed. Click thumbnail to play.

88ffc2 No.15014

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v422/Hydrus.Network.422.-.Linux.-.Executable.tar.gz

🎉🎉 It was hydrus's birthday this week! 🎉🎉

I had a great week. I mostly fixed bugs and improved quality of life.

tags

It looks like when I optimised tag autocomplete around v419, I accidentally broke the advanced 'character:*'-style lookups (which you can enable under tags->manage tag display and search. I regret this is not the first time these clever queries have been broken by accident. I have fixed them this week and added several sets of unit tests to ensure I do not repeat this mistake.

These expansive searches should also work faster, cancel faster, and there are a few new neat cache optimisations to check when an expensive search's results for 'char' or 'character:' can quickly provide results for a later 'character:samus'. Overall, these queries should be a bit better all around. Let me know if you have any more trouble.

The single-tag right-click menu now always shows sibling and parent data, and for all services. Each service stacks siblings/parents into tall submenus, but the tall menu feels better to me than nested, so we'll see how that works out IRL. You can click any sibling or parent to copy to clipboard, so I have retired the 'copy' menu's older and simpler 'siblings' submenu.

misc

Some websites have a 'redirect' optimisation where if a gallery page has only one file, it moves you straight to the post page for that file. This has been a problem for hydrus for some time, and particularly affected users who were doing md5: queries on certain sites, but I believe the downloader engine can now handle it correctly, forwarding the redirect URL to the file queue. This is working on some slightly shakey tech that I want to improve more in future, but let me know how you get on with it.

The UPnPc executables (miniupnp, here https://miniupnp.tuxfamily.org/) are no longer bundled in the 'bin' directory. These files were a common cause of anti-virus false positives every few months, and are only used by a few advanced users to set up servers and hit network->data->manage upnp, so I have decided that new users will have to install it themselves going forward. Trying to perform a UPnP operation when the exe cannot be found now gives a popup message talking about the situation and pointing to the new readme in the bin directory.

After working with a user, it seems that some clients may not have certain indices that speed up sibling and parent lookups. I am not totally sure if this was due to hard drive damage or broken update logic, but the database now looks for and heals this problem on every boot.

parsing (advanced)

String converters can now encode or decode by 'unicode escape characters' ('\u0394'-to-'Δ') and 'html entities' ('&'-to-'&'). Also, when you tell a json formula to fetch 'json' rather than 'string', it no longer escapes unicode.

The hydrus downloader system no longer needs the borked 'bytes' decode for a 'file hash' content parser! These content parsers now have a 'hex'/'base64' dropdown in their UI, and you just deliver that string. This ugly situation was a legacy artifact of python2, now finally cleared up. Existing string converters now treat 'hex' or 'base64' decode steps as a no-op, and existing 'file hash' content parsers should update correctly to 'hex' or 'base64' based on what their string converters were doing previously. The help is updated to reflect this. hex/base64 encodes are still in as they are used for file lookup script hash initialisation, but they will likely get similar treatment in future.

birthday

🎉🎉🎉🎉🎉

On December 14th, 2011, the first non-experimental beta of hydrus was released. This week marks nine years. It has been a lot of work and a lot of fun.

Looking back on 2020, we converted a regularly buggy and crashy new Qt build to something much faster and nicer than we ever had with wx. Along with that came mpv and smooth video and finally audio playing out of the client. The PTR grew to a billion mappings(!), and with that came many rounds of database optimisation, speeding up many complicated tag and file searches. You can now save and load those searches, and most recently, search predicates are now editable in-place. Siblings and parents were updated to completely undoable virtual systems, resulting in much faster boot time and thumbnail load and greatly improved tag relationship logic. Subscriptions were broken into smaller objects, meaning they load and edit much faster, and several CPU-heavy routines no longer interrupt or judder browsing. And the Client API expanded to allow browsing applications and easier login solutions for difficult sites.

There are still a couple thousand things I would like to do, so I hope to keep going into 2021. I deeply appreciate the feedback, help, and support over the years. Thank you!

If you would like to further support my work and are in a position to do so, my simple no-reward Patreon is here: https://www.patreon.com/hydrus_dev

full list

- advanced tags:

- fixed the search code for various 'total' autocomplete searches like '*' and 'namespace:*', which were broken around v419's optimised regular tag lookups. these search types also have a round of their own search optimisations and improved cancel latency. I am sorry for the trouble here

- expanded the database autocomplete fetch unit tests to handle these total lookups so I do not accidentally kill them due to typo/ignorance again

- updated the autocomplete result cache object to consult a search's advanced search options (as under _tags->manage tag display and search_) to test whether a search cache for 'char' or 'character:' is able to serve results for a later 'character:samus' input

- optimised file and tag search code for cases where someone might somehow sneak an unoptimised raw '*:subtag' or 'namespace:*' search text in

- updated and expanded the autocomplete result cache unit tests to handle the new tested options and the various 'total' tests, so they aren't disabled by accident again

- cancelling a autocomplete query with a gigantic number of results should now cancel much quicker when you have a lot of siblings

- the single-tag right-click menu now shows siblings and parents info for every service, and will work on taglists in the 'all known tags' domain. clicking on any item will copy it to clipboard. this might result in megatall submenus, but we'll see. tall seems easier to use than nested per-service for now

- the more primitive 'siblings' submenu on the taglist 'copy' right-click menu is now removed

- right-click should no longer raise an error on esoteric taglists (such as tag filters and namespace colours). you might get some funky copy strings, which is sort of fun too

- the copy string for the special namespace predicate ('namespace:*anything*') is now 'namespace:*', making it easier to copy/paste this across pages

- .

- misc:

- the thumbnail right-click 'copy/open known urls by url class' commands now exclude those urls that match a more specific url class (e.g. /post/123456 vs /post/123456/image.jpg)

- miniupnpc is no longer bundled in the official builds. this executable is only used by a few advanced users and was a regular cause of anti-virus false positives, so I have decided new users will have to install it manually going forward.

- the client now looks for miniupnpc in more places, including the system path. when missing, its error popups have better explanation, pointing users to a new readme in the bin directory

- UPnP errors now have more explanation for 'No IGD UPnP Device' errortext

- the database's boot-repair function now ensures indices are created for: non-sha256 hashes, sibling and parent lookups, storage tag cache, and display tag cache. some users may be missing indices here for unknown update logic or hard drive damage reasons, and this should speed them right back up. the boot-repair function now broadcasts 'checking database for faults' to the splash, which you will see if it needs some time to work

- the duplicates page once again correctly updates the potential pairs count in the 'filter' tab when potential search finishes or filtering finishes

- added the –boot_debug launch switch, which for now prints additional splash screen texts to the log

- the global pixmaps object is no longer initialised in client model boot, but now on first request

- fixed type of –db_synchronous_override launch parameter, which was throwing type errors

- updated the client file readwrite lock logic and brushed up its unit tests

- improved the error when the client database is asked for the id of an invalid tag that collapses to zero characters

- the qss stylesheet directory is now mapped to the static dir in a way that will follow static directory redirects

- .

- downloaders and parsing (advanced):

- started on better network redirection tech. if a post or gallery URL is 3XX redirected, hydrus now recognises this, and if the redirected url is the same type and parseable, the new url and parser are swapped in. if a gallery url is redirected to a non-gallery url, it will create a new file import object for that URL and say so in its gallery log note. this tentatively solves the 'booru redirects one-file gallery pages to post url' problem, but the whole thing is held together by prayer. I now have a plan to rejigger my pipelines to deal with this situation better, ultimately I will likely expose and log all redirects so we can always see better what is going on behind the scenes

- added 'unicode escape characters' and 'html entities' string converter encode/decode types. the former does '\u0394'-to-'Δ', and the latter does '&'-to-'&'

- improved my string converter unit tests and added the above to them

- in the parsing system, decoding from 'hex' or 'base64' is no longer needed for a 'file hash' content type. these string conversions are now no-ops and can be deleted. they converted to a non-string type, an artifact of the old way python 2 used to handle unicode, and were a sore thumb for a long time in the python 3 parsing system. 'file hash' content types now have a 'hex'/'base64' dropdown, and do decoding to raw bytes at a layer above string parsing. on update, existing file hash content parsers will default to hex and attempt to figure out if they were a base64 (however if the hex fails, base64 will be attempted as well anyway, so it is not critically important here if this update detection is imperfect). the 'hex' and 'base64' _encode_ types remain as they are still used in file lookup script hash initialisation, but they will likely be replaced similarly in future. hex or base64 conversion will return in a purely string-based form as technically needed in future

- updated the make-a-downloader help and some screenshots regarding the new hash decoding

- when the json parsing formula is told to get the 'json' of a parsed node, this no longer encodes unicode with escape characters (\u0394 etc…)

- duplicating or importing nested gallery url generators now refreshes all internal reference ids, which should reduce the liklihood of accidentally linking with related but differently named existing GUGs

- importing GUGs or NGUGs through Lain easy import does the same, ensuring the new objects 'seem' fresh to a client and should not incorrectly link up with renamed versions of related NGUGs or GUGs

- added unit tests for hex and base64 string converter encoding

next week

Last week of the year. I could not find time to do the network updates I wanted to this week, so that would be nice. Otherwise I will try and clean and fix little things before my week off over Christmas. The 'big thing to work on next' poll will go up next week with the 423 release posts.

____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

88ffc2 No.15015

It looks like 'namespace:*' searches, despite now working, may be running super slow or doing too much database work, in IRL PTR-syncing situations. I will look into it next week! If you are hit by this, please turn them off for now under tags->manage tag display and search.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

000000 No.15016

>>15014

Two issues I've noticed network-wise so far.

Issue 1 Disclaimer: no idea if this is a regression - first time I've tried this on any version.

On v421 I'm running into an error when querying for IDs with specific filetypes through the API.

I'm trying to get a list of all files of type webm, wav, etc. but I can't figure out the proper way to do that in the "get_files/search_files" endpoint. Stuff like "meta:webm" works in both the client and API but it's missing a lot of files and seems to be manually assigned; on the other hand, using the query "system: filetype is foo" in the client seems to pull directly from the local DB of files and doesn't miss any but when I hit the API with "?tags=%5B%22system%3A+filetype+is+webm%22%5D" I get no results.

I'm not sure if this is user error, by design/wontfix, or you just haven't gotten around to handling/exposing it yet. Is there currently a way to filter on the filetype over all local files? If not, could you add that to the suggestions you're polling on?

Issue 2 Disclaimer: I have a janky workaround for this

I'm working on a browser-based frontend for Hydrus, and one of the problems I've hit is that safari for iOS (and maybe Mac too, I don't have one to test) won't download media like mp4 files because it tries to sample the file with a range download, and if the remote server doesn't support byte ranges then it bails and the file doesn't load. My current workaround is an nginx reverse proxy with the proxy_force_ranges option set, but it's kind of janky and I can't really ask people to run an nginx server to use the webapp if I ever get it to a releasable state. I'm not sure what library you're using to serve the API but if there's an easy way to enable byte ranges that would be great, and if not no worries, it'll be quite a while before it'll matter.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

4ad377 No.15030

I had a good week making some small fixes and improvements to finish up the year. Autocomplete works a bit faster, and some quality of life is improved.

The release should be as normal tomorrow. It will have the 'next big job' poll and be the last release for the year.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

4ad377 No.15031

>>15016

Hey, unfortunately system predicates do not work on the Client API yet. They aren't text-based in the actual client, so I can't just pull text in the query to accept them (yet). They are top of the list to add for Client API features.

Here's the master job for Client API by the way: https://github.com/hydrusnetwork/hydrus/issues/656

Thank you for the note about file ranges. I have not run into this need yet, so I will research how to do it properly. I use twisted as my main server engine, so I know this is possible, I just think I have to link some things together.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.



[Return][Go to top][Catalog][Nerve Center][Random][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / / / / / ] [ dir / random / 93 / biohzrd / hkacade / hkpnd / tct / utd / uy / yebalnia ]