/hydrus/ -Version 345

Name
Email
Subject	REC STOP
Comment *
File	Select/drop/paste files here
Password	(Randomized for file and post deletion; you may also set your own.)
Archive	Archive [500 char limit]
* = required field	[▶Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field)
Allowed file types:jpg, jpeg, gif, png, webp,webm, mp4, mov, swf, pdf Max filesize is16 MB. Max image dimensions are15000 x15000. You may upload5 per post.

YouTube embed. Click thumbnail to play.

Version 345 hydrus_dev ## Board Owner 03/27/19 (Wed) 22:29:56 b920fe No.12021

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v345/Hydrus.Network.345.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v345/Hydrus.Network.345.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v345/Hydrus.Network.345.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v345/Hydrus.Network.345.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v345.tar.gz

I had a good week. The big thumbnail update is done, which means for an important update. Users with smaller databases have less to think about, but anyone with 1,000,000+ files should definitely read all of the following:

thumbnails

This is a great week to make a backup before you update!

Until now, the hydrus client has used two simple thumbnails–one 'full-size', one 'resized'–for each file. Hardware has moved on since then, and so has my code, so it is now feasible to have a single smarter thumbnail that will resize itself on demand. The work for this was done this week. This makes for a simpler and less wasteful storage system and more flexible thumbs all around.

The main thing is that on this update, the client will delete the surplus thumbnails. You will get a popup before it happens to remind you. Specifically, it will delete the 256 'rxx' 'resized' subfolders in your database's file store. If you have moved these folders through the migrate database dialog, this is no problem–the client will delete them wherever they are. Depending on your collection and platform, it could take a few seconds or several minutes. Your new thumbnails may load just slightly slower the first time, but this will return to normal in time.

The best news is that the old 200x200 limit is gone! You can now set your thumbnails' size under options->thumbnails up to 2048x2048! I am now running 360x240 on my 4k machine and like it a lot.

Note that if you previously moved your 'resized' thumbs to an SSD but not your old 'full-size' thumbs, you will want to recheck the migrate database dialog at some point to make sure you move your newly active thumbnail folders to fast storage. This is likely to be a big job, so plan for it.

The migrate database help regarding all this has been updated:

https://hydrusnetwork.github.io/hydrus/help/database_migration.html

Also, be prepared for your next backup run to have significant additional delete overhead as all your backup's 'rxx' folders will be deleted as well. If you have fewer than, say, 250,000 files, I do not think you need to think much of this, but if you have many more, you might like to manually delete these folders from your backup yourself right before your next run. There are 256 of these folders, named in hexadecimal from r00 to rff. Preemptively deleting them will save you backup-scanning time and keep your recycle bin sane (if your backup program deletes to recycle bin). Make sure you do a permanent delete (Shift+Delete on Windows) to skip over the recycle bin yourself.

I tested the new system thoroughly, but as always, some unusual scenario may present problems. The system is more complicated than 'if the thumb is the wrong size, resize it and save it back to disk', so some edge cases may throw an error. Please report any bugs, and I'll try to have them fixed for 346.

full list

- or search:

- set out a plan to achieve some simple conjunctive normal form (e.g. (blue eyes OR green eyes) AND (blonde hair OR red hair)) OR search support

- started work on the object extension and search code to support this search in a very basic (and likely inefficient-for-some-scenarios) way–we'll work on this as we discover the most common inefficiencies

- .

- thumbnails:

- the client no longer uses both 'master' and 'resized' thumbnails–it uses a single, smarter thumbnail

- only the 'txx' thumbnail directories (formerly referred to as full-size) are now used, and the thumbnails inside will regenerate and scale themselves as needed on demand (and will be careful to not save changes to disk when when their source file is non-local)

- the old 'rxx' 'resized' thumbnail directories are no longer referred to anywhere in the code or ui

- the old 'rxx' thumbnails directories will be permanently (i.e. no recycle bin) deleted on update. this is a big job, and you will be prompted on update before it happens

- if you have migrated your db to put 'resized' thumbs on an SSD but not the formerly 'full-size', you will want to recheck the 'migrate database' dialog once you have booted and set a new thumbnail override to move the txx directories over

- due to the smarter thumbnail, 200x200 is no longer the hard limit for hydrus thumbnails! you can now set up to 2048x2048

- all file storage location information is now stored directly in the client db (rather than the options object), which should make for more easily export/importable options in future and improve manual fixing as needed

- added more thumbnail-resizing related popup spam to file report mode

- fixed a windows-only issue that was making the migrate db dialog close after a file move event concluded

- updated database migration help for new concepts and ui

- cleaned up some misc storage code

- .

- the rest:

- fixed a problem in the client api with fetching file identifiers from file_ids

- fleshed out 'help my db is broke.txt' with more specific clone recovery examples

- fixed import support for a variety of single-frame music webms

- fixed an edge-case preview viewer initialisation bug that was trying to draw the canvas before any media was set

- network report mode now states url classes of urls about to be parsed

- misc small fixes and cleanup

next week

I got started on OR search this week. I have a decent plan and feel good. I hope to have some sort of prototype ready for 347. Otherwise I will do some small jobs. I may look into adding multiple-files 'known urls' management.

____________________________

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 03/27/19 (Wed) 23:23:29 b920fe No.12022

>>12021

Thinking about it, I'll add an optional timer to the 'move files now' button on migrate database dialog for 346. You'll be able to say 'move files for 10 minutes' in order to stop some of the ui hanging this action brings.

So, if you want to move all your HDD thumbs to your SSD, please feel free to wait another week while I make that big job more easy to do in small parts.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 03/27/19 (Wed) 23:39:04 150527 No.12023

File: 57bb3ff1222ca22⋯.png (65.88 KB,628x532,157:133,ClipboardImage.png)

>>12021

>- set out a plan to achieve some simple conjunctive normal form (e.g. (blue eyes OR green eyes) AND (blonde hair OR red hair)) OR search support

Fucking nice.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 03/29/19 (Fri) 05:33:56 d30f92 No.12029

Does this change anything about how thumbnails are served via api client?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 03/29/19 (Fri) 13:13:19 e86b39 No.12030

>>12021

>The best news is that the old 200x200 limit is gone! You can now set your thumbnails' size under options->thumbnails up to 2048x2048! I am now running 360x240 on my 4k machine and like it a lot.

Nice.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 03/29/19 (Fri) 14:24:27 b920fe No.12031

There are a couple of issues in this 345: The 'include current tags' search value is sometimes switching to false on load, and animation windows are drawing white frames on media update events. I will have these both fixed for 346. I am sorry for the inconvenience.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 03/29/19 (Fri) 23:26:57 7b6658 No.12033

I regenerated all thumbnails through the database > regenerate menu and it gave a bunch of errors for some mkv files. I'm thinking this is related to the error a while ago where Hydrus thought some webm files were mkv or whatever and couldn't find the files. I've tried looking up some of these files in Hydrus and they work fine otherwise, both the thumbnails and looking at them in the media viewer. I can even reparse and regenerate thumbnails manually just fine.

https://pastebin.com/1pdyBkAs

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 03/30/19 (Sat) 01:04:54 a2cae3 No.12034

"rely on the appropriate default tag options at the time of import" didn't import any rule34.xxx tags, did I fuck something up or is it not supposed to for some reason?

I'm still using v321 if that helps, trying to pare down my duplicates before I backup/update

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 03/30/19 (Sat) 15:28:31 50c05f No.12037

I have recently been importing some big artist galleries into Hydrus. These have often had a bunch of sketchs, that I consider of little value to keep around. Would it be possible to add some sort of lightness sorting option/system tag, so these appear separately, and thus easier to sort through?

One would think that artists would choose a greyscale format, to minimize image size, but that appears to be a rarity. The system would probably need to count the lightness itself.

Related to above, it would be nice if Hydrus had some built in (non-lossy) image optimizations, like converting greyscale images to a greyscale format. (This obviously crashes with the hashing, but I'm guessing the similarity stuff could come in use.)

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 03/30/19 (Sat) 19:34:07 2c0083 No.12038

>>12021

Apart from having to delete the old thumb folders myself due to odd permission errors (which I can't explain at this point), the thumbnail change worked out great.

Thanks, based dev! This was a very good change!

>>12021

If Hydrus itself optimized files, it'd know what the hash of the original file was.

But unless you use powerful but still rare formats like flif and heif in the situations where they work or only apply optimizations on png where there are lossless optimizers working on palettes and such, you'll be in a bit of trouble with this plan.

It's actually not THAT easy with existing tooling, for instance I've accidentally wrecked quite a lot of ProPhoto colorspace tiffs with full –lossless settings on multiple encoders, because they -without warning- wouldn't actually be lossless for the colorspace conversion step.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 03/30/19 (Sat) 21:15:27 b920fe No.12046

>>12033

Thank you for this excellent report. Yeah, I think it is getting confused about file extension at some point. The reparse code and the regen thumb maintenance code are separate so I'll see what they are doing different. I will try to have a look at this this week.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 03/30/19 (Sat) 21:19:24 b920fe No.12047

>>12034

Thank you for this report. 321 should be good for this afaik. What are the default tag options set as? I forget where it is under 321, but it is now network->downloaders->manage default tag import options. Did you set to parse tags for all 'file posts' under there, or 'rule34.xxx file page' url class specifically?

By default, the client parses nothing.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 03/30/19 (Sat) 21:29:03 b920fe No.12048

>>12037

Thanks. Yeah, it is a long-term plan to add colour-awareness to hydrus, both for searching and sorting. It is not aware of colour at all atm, unfortunately.

I vacillate on converting/optimising files in hydrus. I currently do bmp->png on import, but I hesitate going further. Maybe one day I will (optionally) convert everything to webp or flif or something, but for now there is less pressure on reducing hard drive space–which is tbh atm fairly easily solved, and at the current rate of hard drive growth less and less important–and more on cleaning up my messy code.

If I ever did optimisation tech, even completely lossless, it would always be optional and default off.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 03/30/19 (Sat) 21:31:32 b920fe No.12049

>>12038

Thank you, I have received this error report elsewhere as well. I messed up some of my permissions settings for Linux (while trying to ensure the directories were writeable for the delete, I accidentally removed execute permission, fugg, making them inaccessible). I will fix this issue and give the thumb dirs another pass for 346 for users who missed this.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 03/31/19 (Sun) 16:35:39 7001ce No.12057

File: 823c72d1689784b⋯.png (175.4 KB,1560x775,312:155,client_2019-03-31_11-09-30.png)

Ok hdev, I need some help

I am one of the people with a problem with deleting things because I will see the previously deleted notice and have to check said image, When I see that it is a perfectly fine image, I then have to re import it.

so with this problem in mind, I have asked about this section here (see image) containing data about erasure.

Is there anything I can do that would set up for when this is implemented?

I know I asked for hotkeys or a potential fill in the blank area when deleting an image to facilitate faster removal, I know that at some point when I asked for this was around the time file note were added, the obvious place for it.

so I have to ask, is it possible to link this area with files note's?

My pressing issue is that I can't go through files till something happens, due to my autistic need to know what the hell was deleted I can't leave well enough alone even though logically I should be able to, I am eating around 5-25gb per week of storage space, and i'm coming up on my last 100gb before a new hdd needs to be procured. I have been taking an almost entirely shotgun approach to images, and if push come to shove could very easily remove likely half of what I currently have, I just need some way at all to mark an image with the reason why its being removed so they don't get required.

with duplicates, what I honestly find the easiest to deal with, is it possible to add a tag/note/rating to the worse file? This way if file notes does get linked there, it will be able to tell me why it was removed. Also is it possible to edit multiple files notes at once? if file notes gets linked there, i'm more than capable of adding notes manually till something happens, however I can't feasibly do it manually for every single file, I would need to edit them in bulk.

Anything helps, I'm trying to avoid a new hdd purchase till later in the year if even this year at all.

tldr

I have about 1.5 months left, 2 at most of hdd space left without culling images

I'm ok with doing shit manually, I just have to know what i'm doing will work, and won't be in vane

If what I think may work will work, I need to be able to edit many files at once

And if what I think will work works, I need a way to add the notes through dup, or add a specific tag so I can search them in the trash to add the notes later.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/01/19 (Mon) 20:38:32 335f6d No.12068

Hello, new to Hydrus, when I view an image with Hydrus's image own image viewer it displays the wrong color gamut as my irfranview, Firefox, Chrome and Photoshop. How do I gain access to the image viewer's color management settings?

Irfanview has Properties/settings > Zoom / Color Management > Color management options

Anything Hydrus has similar to this?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/02/19 (Tue) 01:10:11 335f6d No.12070

File: 032d8fc9abf1bdb⋯.png (58.87 KB,1237x517,1237:517,Untitled-3.png)

>>12068

Also I notice Windows own photo thumbnails are much sharper/higher quality compared to the one hydrus uses. Will there be improvements here?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/02/19 (Tue) 10:01:55 335f6d No.12071

>>12070

When I open images with externally and move from one image to another it doesn't go to the next image within hyrus's client but the within the folder of the OS.

With Hydrus's image viewer are there options in Windowed mode and full screen that all images are fit to your native screen size if larger than it, and if it isn't larger than your native screen then there will be no resample and it will just display the original image size?

Also how can I make it so the Hydrus Client remembers the width size I set to the left panel of the GUI?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 04/02/19 (Tue) 23:16:22 b920fe No.12074

>>12057

I am sorry you are still having trouble here. I will attempt this: For 347, I will make file deletions from different locations across the client record a brief reason, such as 'Deleted via archive/delete filter.' or 'Deleted due to being worse of a better/worse duplicate pair.' I will retrieve this line of text in the deleted notes of file imports. It will be slightly useful for everyone to see these brief reasons, and obviously more important to you. Please let me know how it works for you.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 04/02/19 (Tue) 23:32:20 b920fe No.12075

>>12068

Thank you for this report. I generally convert images to sRGB as soon as I load them as this is generally what my ui library works in. My program does not have options for colour management yet. Can you expand on the specific color gamut you are expecting, and what you seem to be getting, and can you post an example file so I can try it on my end?

>>12070

Thumbnail generation code is slightly in flux at the moment. That blurry thumb may have been badly generated from another thumb. I will be rolling out some better thumbnail regen tools (from the thumbnail right-click menu) tomorrow. Please run a force-regen call on that blurry thumb in 346 when it is convenient and see if it neatens up. If it is blurry even when you regenerate it with the new code, please submit the file and the thumbnail resolution you have, and I will have a look on my end.

>>12071

Unfortunately, external programs are not aware of how hydrus stores files, so preserving ordered file lists through an 'open externally' call is not currently possible. Perhaps in future I will add some sort of way to set up 'open playlist' calls, for those programs that support them, but this is not available now.

The options for how the media viewer sizes media are under file->options->media. I apologise for the debug-tier ui here, but if you double-click on a particulary mimetype, like 'image/jpg', you can set a whole bunch of 'if the media is larger than the canvas, scale it down'-type options. I think you will be able to set what you want here.

The width of the left panel should be saved for the currently viewed page on client exit. If you would like to have more control, please check pages->management and preview panels. You can force a save of the currently viewed page's value, and then propagate that value across all current pages.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/02/19 (Tue) 23:54:56 335f6d No.12076

File: ab968e14c2aecc4⋯.png (74.92 KB,1031x707,1031:707,1.PNG)

>>12075

The Hydrus Image viewer doesn't seem to be using my System's sRGB color space profile in the drivers, and instead uses something else. I would like to point so the color management uses this ICC profile.

My firefox, Chrome and Photoshop automatically uses this profile without any tinkering so they all display the same calibrated profile, but image viewers doesn't so I have to point them to this ICC file.

Thanks for your help.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/03/19 (Wed) 00:02:55 335f6d No.12077

>>12076

My monitor is color calibrated with DisplayCAL Loader, I think the image viewers are pointing to use the loaded ICC profile in that managment program, so I have to manually point them back to the right ICC profile otherwise they'll end up double calibrating themselves, Photoshop and the web browsers seem to ignore this and stay on the defaulted sRGB ICC profile.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/03/19 (Wed) 01:39:12 19b270 No.12079

>no next release announcement thread

is hydrus dev okay?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/03/19 (Wed) 01:39:46 19b270 No.12080

forget it

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/03/19 (Wed) 17:30:40 335f6d No.12081

is it possible to add in a option for color managment just to point to an ICC Profile? Would help greatly for people using an ICC profile on Windows color managment or or programs like displaycal.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 04/03/19 (Wed) 18:59:25 b920fe No.12082

>>12076

>>12077

>>12081

Hey, I apologise, this ICC profile tech is completely new to me. I will make a job to look into this, specifically how much control I have over it with wx or OpenCV/PIL, for next week.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 04/03/19 (Wed) 19:01:24 b920fe No.12083

>>12079

>>12080

I took some extra time yesterday evening (and just now) to finish some bugs and cleaning up thumbnail code. Going to test it a bit more, so release might be delayed a little bit. I feel good about it though, so no chance of anything going wrong™.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.