/hydrus/ -Version 349

Name
Email
Subject	REC STOP
Comment *
File	Select/drop/paste files here
Password	(Randomized for file and post deletion; you may also set your own.)
Archive	Archive [500 char limit]
* = required field	[▶Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field)
Allowed file types:jpg, jpeg, gif, png, webp,webm, mp4, mov, swf, pdf Max filesize is16 MB. Max image dimensions are15000 x15000. You may upload5 per post.

YouTube embed. Click thumbnail to play.

Version 349 hydrus_dev ## Board Owner 04/24/19 (Wed) 22:34:18 4e1764 No.12361

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v349/Hydrus.Network.349.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v349/Hydrus.Network.349.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v349/Hydrus.Network.349.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v349/Hydrus.Network.349.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v349.tar.gz

I had a great week. The duplicate filter work went really well, and the manage tags dialog has a neat new button for fixing siblings and parents.

The poll for the next 'big job' is live here >>12358 ! The direct link is https://www.poll-maker.com/poll2331269x9ae447d5-67

duplicate filter

The duplicate filter page has its old 'file domain' button swapped for a full file search context. This allows you to see 'potential pair' counts, show some random pairs, and launch the full duplicate filter on just a subset of your dupes! For instance, you might want only to filter from a certain creator, or only on very small jpgs. You can also check a box to set whether both of the pair must match your search or only one.

The duplicate page has a significant relayout as well. It now has two tabs–one for the maintenance/search side, and one for the actual filtering–and shows its data in a clearer and less laggy way. The new search data is saved with your page, and on a per-page basis, so if you like you can set up several duplicate processing pages for your different queues.

This system works faster and better than I had expected. Please check it out and let me know what you think. It is usually fairly fast unless you give it a very slow search to work with.

One bug I just noticed with my IRL client is if you give it a specific search with more than 10,000 files (like for me system:archive), the duplicate count will be too low and will change slightly on refreshes. This is because of the 'implicit' system:limit on all queries, which caps search results to 10k files (or whatever you have it set to in the options), and causes the duplicate filter to instead be sampling the potential search domain on each query. This is not true in this case for system:everything, which here uses a search optimisation to get the full count. So, I suggest you throw in a <10k tag for this week, and I will see if I can tackle this problem for 350.

I will update the help for this once I know the ui is settled.

manage tags

The manage tags dialog has a new 'siblings and parents' button that auto-hard-replaces siblings and adds missing parents! It even works on multiple file selections, so if you have a bunch you want to fix, you can just ctrl+a and hit the button and it will sort it out for each file. It gives you a little summary yes/no dialog before it fires just to make sure everything looks sensible.

Beyond some additional sibling/parent code cleanup on that dialog, there is also a new 'cog' option to stop triggering 'remove' actions if you 'enter' a tag that already exists. If you turn this on, only a double-click or delete-key press on the list will make for a 'remove' result.

the rest

I spent some time cleaning up file import and thumbnail generation, particularly for videos. Videos have improved metadata parsing, and thumbnail generation should be much faster. A variety of unusual webms should now be importable and have correct frame counts. Thumbnail right-click->regen commands also work faster and update thumbnail metadata more cleanly and reliably.

If you would like pages to always focus their 'input' boxes when you switch to them, there is now an option for this under the 'gui pages' options panel.

full list

- duplicate filter:

- the duplicate filter page now has a full-on real-deal file search object to narrow down the duplicate filter, potential duplicate count, and 'show some random dupes' search domains! it also has a 'both files match' checkbox that determines if one of both files of the potential pairs should match the search!

- the duplicate filter page has multiple layout changes as a result:

- the main management area is now split into two pages–'preparation', for doing maintenance and discovery work, and 'filtering', for actioning the potential dupe pairs found

- the 'filtering' page will select by default, but if 'preparation' needs work, its name will become 'preparation (needs work)'

- the 'filtering' page now has file search ui and the 'both files' checkbox instead of the file domain button. this search data is saved on a per-page basis

- the two pages' status texts are now updated on separate calls which have been rewritten to be asynchronous (with 'updating…' text while they work). both now have explicit refresh buttons to force them to update

- the additional non-unknown pair counts listed on the filter area, which were irrelevant to filtering and sometimes confusing, are now gone. it only lists the 'unknown' pair number

- the duplicate filter page's help button no longer has the awful 'simple help' entry. the full html help will get a pass in the coming weeks to reflect the new search changes

- the duplicate file db code received significant refactoring and improvement to support searching the potential dupe space while cross-referencing the new file search context (and still falling back to the fast code when the search is just blank/system:everything)

- misc duplicate file db code cleanup and refactoring

- while in advanced mode, you can no longer select 'all known files' file domain for an export folder (and now the duplicate filter page) search context

- making a file delete action in the duplicate filter (by hitting delete key or the button on the top hover window, which both trigger a dialog asking to delete one or both) now auto-skips the current pair

- .

- manage tags:

- the manage tags has a new 'siblings and parents' button that will auto-replace incorrect siblings and auto-add missing parents! it works on multi-file selections as well! it gives you a summary yes/no dialog before it fires

- the manage tags dialog has a little logic cleanup r.e. siblings and parents and their cog auto-apply options. the auto-application now only applies on add/pend actions

- the manage tags dialog has a new cog button option to not trigger 'remove' actions from an autocomplete dropdown or suggested tag input action when the tag already exists

- .

- the rest:

- gave video metadata parsing another pass–it now detects 'hidden' incorrect framerates due to advanced 'skip frame' codec settings and is more accurate at determining frame count and duration, including some fixed offset calculations that was sometimes adding or discounting a few frames

- manual video frame count, when needed, is now faster and produces better error text

- fixed a critical bug in thumbnail regen that was sometimes potentially looping regen on files with unusual rotation exif information

- significant improvements to how the client file manager handles thumbnail identifier information, saving a great deal of time for file import and thumbnail regeneration code of videos

- fixed an issue where regenerated file metadata was not propagating up to the ui level in real time

- cleaned up some thumbnail cache initialisation code

- the 'generate video thumbs this % in' option is moved from the 'media' to 'thumbnails' options page

- to simplify code, and in prep for the idle-maintenance-rewrite of this system, the database->regen->thumbnails call is now removed

- all three fields of text on serialised pngs now wrap, and they pad a little better as well

- added a new option to the 'gui pages' options page to force input text box focus on page changes

- fixed a small type issue with the server's session cookie code and some new library versions

next week

I've felt increasingly ill this week, so I am going to take a day or two completely off. Otherwise, next week will be catch-up for smaller jobs and bug reports and other messages, which I have fallen behind on. Please have a think about the big jobs poll, and let me know if you have any questions. Thank you for your support!

____________________________

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/25/19 (Thu) 00:23:04 05b5ca No.12364

>>12361

The duplicate view changes to a "normal" view are a large improvement.

Can I lock the zoom level and relative position while switching between files? I sometimes need this when I'm trying to get a close-up look at details to figure out which file is better

Thanks for the release and get well soon!

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/25/19 (Thu) 00:39:34 793b72 No.12365

>>12364

get a magnifying program, the only time I need to zoom in anymore is when files are over 4000x4000 or there around, and even then, I don't need to half the time too. not ideal but its the best we currently got.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/25/19 (Thu) 00:44:52 793b72 No.12366

just a heads up, after filtering some images, I went to perma delete them and the program hung for 30+ minutes before I killed the process

The program forgot I deleted the images, and on refreshing trash, they were still 'there'

the files were gone, but the note that they were in the db was, so it was tossing back errors.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/25/19 (Thu) 00:49:44 793b72 No.12367

File: ecf786a1069ab3a⋯.png (244.25 KB,412x622,206:311,client_2019-04-24_19-46-22.png)

oh, one more thing I forgot to mention when duplicates were getting tagged for why they are deleted, is there any way to fix this? long time ago my db fucked itself, and in fixing it, broke duplicates going backwards. now that I can go though duplicates, is there something I can do?

right now I have 400k~ images to parse, so its not pressing, but this is now an issue that if it can get resolved, that would be fantastic.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/25/19 (Thu) 02:18:15 793b72 No.12368

>>12366

Update, got another hang thats so far been going on for 5 minutes, i'm thinking it wasnt me deleting a file that made it hang last time, it may have been clicking show some random pairs that did it.

Its possible last time WAS deleting images that caused the hang, I just assumed because I let it hang for a while came back, and the images where still in the db but not in the db at the same time.

giving this 20 minutes before I force close it

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/25/19 (Thu) 02:18:58 793b72 No.12369

File: 719763908a48a0a⋯.png (16.31 KB,1116x462,186:77,dwm_2019-04-24_21-16-53.png)

>>12368

forgot the image if it matters, this is what I searched when I got this hang

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/25/19 (Thu) 02:41:59 793b72 No.12370

>>12369

just got hang number 3 that forced a shutdown from trying to show random potential pairs from the exact same search.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/25/19 (Thu) 07:45:46 bab0fb No.12372

File: 1c658192d0f4b92⋯.png (59.33 KB,175x149,175:149,ClipboardImage.png)

>>12361

>The duplicate filter page has its old 'file domain' button swapped for a full file search context. This allows you to see 'potential pair' counts, show some random pairs, and launch the full duplicate filter on just a subset of your dupes! For instance, you might want only to filter from a certain creator, or only on very small jpgs. You can also check a box to set whether both of the pair must match your search or only one.

Nice.

Now I won't accidentally a chunk of my files again.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 04/25/19 (Thu) 13:05:47 4e1764 No.12373

Looks like there is more dupe work to do. Just a quick thing, I noticed myself that some dupes queues have a blank/ghost dupe pair at the end (at count 1) that does not display. I think this is actually an old issue brought to the fore with smaller queues. I can reproduce it and will try to fix it for 350.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/26/19 (Fri) 00:46:15 793b72 No.12378

have a question, think the delete reason will get in this week? just ran across an image archive for an artist who usually doesn't have torrents, he makes art I like, but for every good piece, there are 3-5 edits with shit. If I know it's coming this week I can get the torrent now, but if not i think i'm holding off for a week.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/26/19 (Fri) 14:51:30 bab0fb No.12384

>>12378

Not dev, but get the torrent either way.

I've often been fucked out of files because fags stopped seeding a torrent.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/26/19 (Fri) 23:09:49 793b72 No.12390

>>12384

nah, the place it is is semi private, the issue is that if a torrent comes out now, it likely won't get an update for a year or so despite more art. right now is a current as of 4/25 torrent, it will be there for at least a few months if not year due to the way the system rewards people

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/27/19 (Sat) 04:13:22 181156 No.12395

File: b1f655d77b365c8⋯.gif (877.17 KB,498x498,1:1,thup.gif)

Thanks for all the neat new things you added this version, hydrus_dev!

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 04/27/19 (Sat) 22:30:48 4e1764 No.12404

>>12364

Great.

I think it locks zoom and position right now if the files have the same resolution ratio. I can't think of a clever solution off the top of my head for when the files have different ratio–maybe just match same width size, or the same (smaller dimension), and offset position from the top-left?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 04/27/19 (Sat) 22:32:37 4e1764 No.12405

I am thinking of moving the duplicate metadata info (and probably the action buttons too), like 'this file has higher res' to an always-on-top hover window in the middle-right. It is annoying to always be moving my mouse up to the top to see the differences between the two files, and I think a big part of improving the workflow here will be to highlight how different the files are on a technical level.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 04/27/19 (Sat) 22:36:03 4e1764 No.12406

Thank you for this report. Do you have the implicit system:limit under options->speed and memory set as 'no limit'? There seem to be a couple of bugs in the dupe search code overall, which this new queuing system is exposing more often. 30 min waiting for a search to complete is surely not related to search size, but if you set that value back to the default of 10,000, does it clear up?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 04/27/19 (Sat) 22:37:17 4e1764 No.12407

>>12367

I am sorry, I do not understand this post. Can you explain the problem a different way?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 04/27/19 (Sat) 22:40:21 4e1764 No.12408

>>12378

Yeah, I hope to have the advanced delete file dialog in this week.

>>12395

Thanks m8, let me know if any of it gives you trouble. The dupe system still has some bugs in it. The big poll though is pushing towards duplicate db work atm, which I will be very happy to work on.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/28/19 (Sun) 01:16:52 5f127f No.12416

>>12361

Question, is the twitter downloader not usable as a subscription? It insta-fails when I try. I thought I might bypass the whole "gives up after 5 retweets" thing by just setting a sub.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/28/19 (Sun) 08:00:21 793b72 No.12421

File: f00edef019f339e⋯.png (95.77 KB,1064x734,532:367,client_2019-04-28_02-52-43.png)

>>12406

here is my ram options, I think I touched them at some point, it would have been quite a long time ago, I think I know the setting you are talking about but to be sure.

>>12407

2.4 million files in dup filter

3.8 million files total

A while back I had a power out, it fucked my db, and when we got it working again, the dup detection was not working correctly only going forward.

I think at the time I said you had better things to do because for me dup detection was a novelty because I couldn't delete things yet, now that i'm able to delete things from dup, this is a concern… but I have 400k potential pairs to go through before it becomes a real problem for me.

If the solution isn't simple, I still think you have better things to do, but if it is simple, then I would appreciate the help.

>>12408

Good to know, will be downloading it now, this is going to make a great test case for it.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 04/28/19 (Sun) 18:09:03 4e1764 No.12428

>>12416

If it works ok in the gallery downloader page, it should be ok in a subscription. Things seem ok here on my dev machine, so can you give an example of a twitter username that is not subbing right for you, so I can test it on my end? And how is it failing for you–does it give an error, or just run and seem to cancel early? If it is the latter, if you go in and edit the sub and drill down to a (presumably 'dead') query, does the 'gallery log' button of the query have a failed/ignored entry, and if so, is there an attached error note? If so, can you post it?

In case I am misunderstanding, do you know the retweet issue is part of it? Is a 'page' of tweet results being truncated to '0 results' because retweets are being culled or something, do you know? If so, I think this could, yes, cause an initial subscription run to think it had hit the end of results early. There is not an easy fix for this atm, unfortunately.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 04/28/19 (Sun) 18:15:38 4e1764 No.12429

>>12421

Yeah, at the bottom of that page, there is a 'Forced system:limit for all searches'. You might like to turn this back on just to speed up some things if you occasionally accidentally hit a bare 'system:inbox' search and the client tries to build a multi-million file result. The default is 10,000.

For the damaged dupe detection bug, I am not sure if there is (still) a problem here. I am afraid I cannot remember the exact details of what you encountered, so I cannot speak cleverly about it. If your dupe filter is lining up pairs that seem correct, I think you are good. If you are worried that 2.4M < 3.8M, you can feel better–the duplicate filter only works for still images like jpgs and pngs right now, so if you have a whole bunch of webms and so on, these will not be processed by the duplicate system (yet) and aren't counted.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/28/19 (Sun) 18:23:13 05b5ca No.12430

>>12404

> I think it locks zoom and position right now if the files have the same resolution ratio.

Seems like I have a lot of images that aren't. But that explains what is happening.

> I can't think of a clever solution off the top of my head for when the files have different ratio–maybe just match same width size, or the same (smaller dimension), and offset position from the top-left?

Hm, not exactly sure what you meant.

I believe centering on the same relative percentage of distance from the side and top and equal zoom level is the best simple (not image content aware) method?

As in, if the user zoomed in and scrolled vertically to a third from the top of the image, you do the same on the other image - a third from the top of that image, same zoom level. Basically, you assume they're probably scaled images

I'm pretty sure this needs to have a toggle button for this that is defaulted to on but can be turned off as-needed for images that happen to not be just scaled.

That's the most needed change IMO. There are a bunch of options that might be interesting to have on top of that, like having two buttons to only fix zoom or scrolling individually, or to have a setting that disables this toggle button the first time after you picked an area to focus on [at the time where you first switch to the other image in the group, after having zoomed/scrolled - this is for collections with a lot of duplicates with borders and stuff where it only approximately works and to actually compare the features you zoomed in on you need to scroll and zoom the other image manually].

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/29/19 (Mon) 23:19:16 5f127f No.12436

>>12428

Strange, now it's working fine. I must've screwed something up during creation. When I last tried it, it insta-404'd

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/30/19 (Tue) 03:30:12 fc0939 No.12438

KILL URSELF NIGGER BABY

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 04/30/19 (Tue) 03:30:31 fc0939 No.12439

HYRDUR SSUCKS BAN ME

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.