/hydrus/ -Version 397

Name
Email
Subject	REC STOP
Comment *
File	Select/drop/paste files here
Password	(Randomized for file and post deletion; you may also set your own.)
Archive	Archive [500 char limit]
* = required field	[▶Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field)
Allowed file types:jpg, jpeg, gif, png, webp,webm, mp4, mov, swf, pdf Max filesize is16 MB. Max image dimensions are15000 x15000. You may upload5 per post.

YouTube embed. Click thumbnail to play.

Version 397 Anonymous ## Board Owner 05/13/20 (Wed) 20:17:43 a70092 No.14274

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v397/Hydrus.Network.397.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v397/Hydrus.Network.397.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v397/Hydrus.Network.397.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v397/Hydrus.Network.397.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v397.tar.gz

I had an ok week. I mostly did small fixes and improvements.

highlights

I added 'system:has/has no note with name xxx' to search for files with specific note names. This is bundled into the same dialog as last week's 'number of notes', for the new generic 'system:notes' entry in the normal system predicates list.

When you have many page tabs in a row, they will now middle…elide their name text to try to fit better. You can turn this off if you like under options->gui pages.

When files in the duplicate filter are pixel duplicates, the file size and import time comparison statements now have no 'score' and colour blue. Pixel duplicates are often a different judgment call, so until I can get more flexible scoring options here, I am hardcoding it to be less visually green/red.

If you need to run a gallery download with a search term that includes '#', this should now work.

I updated the IPFS code to work with the new IPFS 5.0 daemon. I think I have things mostly correct, but the new daemon has some new CORS restrictions that may need configuration on the IPFS side, depending on your situation, to give hydrus permission to talk to it.

The left page panel's position should set itself more reliably for users who start the program minimised to system tray. If you have had trouble here, please let me know how you get on. I believe the changes here will also reduce some UI lag for many other page adding and moving operations, particularly for large sessions.

full list

- regular changelog:

- added 'system:has/has no note with name xxx' to search for specific note names

- in the normal system predicate list, the notes pred is now the generic 'system:notes' to launch a combined dialog for both num notes and named notes

- favourite tag suggestions are now sorted in manage tags dialog according to the default tag sort

- page names will now middle…elide when there are too many to fit into a row (and normally left/right buttons would be added). if the elided tabs still do not fit, the buttons will pop up as before. added a checkbox to options->gui pages to turn this text eliding off

- pulled the 'page name' options on that panel into their own box and added some text regarding the 'my big row of import page tabs keeps scrolling weird' issue

- when files are pixel duplicates, the filesize and age comparison statements will now have 0 score and thus be coloured neutral blue

- the standard text entry dialog now always selects any default text it starts with, so you can now type to immediately overwrite. see how you like it and if there are some places where you think an exception should be made

- updated the IPFS interface to work with the new IPFS 5.0. all api requests are now POST so it doesn't 405, and the User-Agent is overridden to one that IPFS will not 403 at, and I fixed a typo the new api is more strict about

- a hack to get page splitters to lay out correctly on session load is rewritten from a hammer to a scalpel. pages now set their splitter positions on their first individual visible selection. this both reduces some minor ui lag on session/page load and improves splitter positions for clients that open minimised to the system tray

- a long-time odd issue where loaded sessions would initially select the top-left-most non-page of pages is fixed. now the bottom-left-most page of any kind is selected

- fixed tag autocomplete selecting the bottom-most pre-loading result. it now correctly selects at the top

- fixed an issue setting certain values (typically loading a default) to a tag import options panel

- the client is now more aggressive about clearing subscriptions from memory when they are finished running

- in windows, the main method that copies files now checks for modified time of the source file. if it is before 1980-01-01 UTC, it does not copy the file metadata, as some Windows has trouble with this lmaoooo

- cleaned up how some thumbnail 'current focus' media determination code works. should have fixed some weird errors when hitting certain shortcuts on collections

- cleaned up basic list/sort code across the program

- the 'queue' and add/edit/delete listboxes now emit change signals when new items are added or imported

- pyparsing, a helper for cloudscraper, is now correctly bundled in the built releases. a new line in help->about displays this

- help->about now lists cloudscraper version

- updated the discord link to the new https://discord.gg/wPHPCUZ

- .

- upcoming string processing changes for advanced users:

- I extended string parsing code this week, but I am not yet ready to turn it on. when it does come on, it will change all formulae from the fixed string match/converter pair a combined general string processing 'script' of n steps

- wrote a new 'string splitter' object that takes one strings and splits it into up to n strings based on a separator phrase (such as ' ,')

- wrote an edit panel for string splitters

- wrote a new 'string processor' object that holds n ordered string match/converter/splitter objects and filters/converts/splits x strings into y strings based on those steps

- wrote an edit panel for string processors. it has a notebook that live updates with test results for each step on every update

- wrote unit tests for string match

- wrote unit tests for string converter

- wrote unit tests for string splitter

- wrote unit tests for string processor

- refactored string conversion edit panels to their own file

- refactored string conversion controls to their own file

- misc string processing cleanup and labelling improvements

- .

- technical url parsing stuff:

- urls are now stripped of leading and trailing whitespace during normalisation, just in case a paste contains some extra whitespace. previously, it would sometimes throw a 'doesn't start with http' error

- the hydrus url normalisation process now normalises the hostname according to the NKFC unicode format, meaning unusual characters like ？and e◌ are now replaced with their normalised visual equivalent ? and é, and hence these urls will no longer throw errors when they are added

- if '?' or '#' end up in a hostname (which are invalid characters), it is now converted to _, just to stop complete parse mangling when weird urls are submitted. this character replacement may become more sophisticated in future

- the hydrus downloader should now support search terms that include '#'

- download query parameters that contain '%23' ('#', encoded) are now not unquoted in url normalisation

next week

Next week is a cleanup week. I will push on more wx->Qt code cleanup. I would also like to try to turn on some of the background text/notes work I have been doing for the past couple of weeks.

I will be taking my week vacation in two weeks, in the 399->400 gap from the 27th to the 3rd.

____________________________

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/14/20 (Thu) 19:39:27 9ecfc5 No.14280

Hey HyDev. About the file modified time, Imgbrd-Grabber and download managers like jdownloader lets you save files with the original modified time, which is stored in the server. Would it be possible also to add this behavior to hydrus? It would be very useful for chronologically ordering pictures from boorus, twitter, etc, without the need to save the id number as a tag, for sorting.

Also, could you add the ability to update the modified time whenever you import the same file with an earlier date, either from the hdd or the web? For example a setting in the "file import options" dialog with options like "Always update the modified time", "Update only if an early date is found", "Never update". Right now the only way to update the modification date is to permanently delete the file from hydrus and import it again. Thanks.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/15/20 (Fri) 07:26:11 0512cc No.14283

>>14274

Checking in to catch up on over 50 versions. Thank you for what you do, Hydev.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/16/20 (Sat) 02:59:37 1e9c58 No.14289

Searching for the note names is a great improvement! Thanks for that!

I hope it will be possible in the future to import notes from text files and set their names using regexes or something. And to search for words in the notes themselves, of course!

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/16/20 (Sat) 10:09:38 4277c1 No.14292

I don't know when this was improved, but manage tags speed is now usable! It used to take like 10 seconds (closer to a minute on an hdd) to complete the search even if I wrote the whole tag, but now it's snappy. Thanks a bunch!

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous ## Board Owner 05/16/20 (Sat) 21:44:14 4321d3 No.14293

>>14280

Yeah, I really want this. I have a whole plan for it, but I haven't found the time for it. Even when a site doesn't give a modified time, I can cobble together something from any 'post time' the site parser got.

I had been grappling with how to best handle modified time conflicts/updates. Figuring out decent values and tacking them on top file import options for more customisation is a great solution, thank you.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous ## Board Owner 05/16/20 (Sat) 21:47:34 4321d3 No.14294

>>14283

Thanks m8, keep on pushing.

>>14289

Yeah, I have search and import/export+client_api in my big notes job. The import/export will have some note renaming capability, so when we come to it, please let me know if that does what you want, or if you absolutely need regex capability. It might be that is easier to do with a client api script than writing it into the program.

I don't know when I will get all this done, but I hope to keep moving forward a step at a time in normal weekly work.

>>14292

Hey, great, thanks for letting me know. I reworked the whole pipeline a couple weeks ago and have been cleaning it up since, so it times queries and caches results much better.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/17/20 (Sun) 16:41:52 f47ee7 No.14311

>>14293

Seconding modified time features, would be really useful. Would also be nice to be able to store different times from different sources and use the earliest by default.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/18/20 (Mon) 11:45:54 f7ad7c No.14318

File: 72966c73496e09b⋯.png (392.14 KB,1500x1500,1:1,21589266.png)

>>14280

For what it's worth I wrote a Go program to convert file system metadata into hydrus tag sidecar files a few years ago when I wanted to import my stuff and have an exit path to export them back out if I needed to. Maybe it's useful to someone in the short term.

https://ipfs.io/ipfs/QmYpD3zqGkcyjogy9mCt2yYqZbL7tR8Tf9ZVMtmprHX5Me

If I run

hydrus_meta "look a subdirectory"

from the "hydrus_meta" directory, it will make a sidecar for files in it that looks like this

`look a subdirectory\Mio\21589266.png.txt`


Original Filename:21589266.png
Original FullPath:C:\Users\UserNameRedacted\hydrus_meta\look a subdirectory\Mio
Original ShortPath:Mio
Original Relative Path:look a subdirectory\Mio
Original CreationTime:2020.05.18-07.14.17
Original ModTime:2014.03.03-05.33.15

Creation time only works on Windows, otherwise it uses the modtime for both but it's real easy to change that and the tag format/which tags you want. I also censor the username by default in case they were to accidentally go to the PTR but you can remove that easily too.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Release Tomorrow! Anonymous ## Board Owner 05/20/20 (Wed) 04:16:30 4321d3 No.14324

File: daa7d64a9c63342⋯.jpg (2.75 MB,3086x3803,3086:3803,daa7d64a9c63342c1bb1c388bc….jpg)

I had a good work week. I added some tag autocomplete search options, so you can now enable '*', 'namespace:*', and '"ser" matches "ser*:*"' wildcard searches for particular services, expanded the text processing tools for the users who make downloaders, and also did a little quality of life.

The release should be as normal tomorrow.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous ## Board Owner 05/20/20 (Wed) 04:33:21 4321d3 No.14325

>>14311

Thanks. I am thinking about it. I don't really want to add the additional data structure of a per-source 'posted time' timestamp, since it is a bunch of extra overhead, but it does make it easier to present a modified date.

Also, if I were to build this sort of thing and it turned out most boorus had upload times within a few days, it might not actually prove that useful. I like this idea of bumping the decision to file import options. If a merged modified date fails to account for an interesting situation where different sites' modified dates differ greatly, maybe it could be extended.

I am also thinking about the issue that most of us have huge collections with fairly bad modified dates already, and going back to sites to reparse this data would be a big old waste of bandwidth. I'm thinking of writing a small import/export system for modified dates, so users can share this info. I'll get good new modified dates going first though, and we'll solve the retroactive problem later. Just need to find time for it.

>>14318

This is cool!

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.