[ / / / / / / / / / / / / / ] [ dir / random / 93 / biohzrd / hkacade / hkpnd / tct / utd / uy / yebalnia ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog  Archive

Name
Email
Subject
REC
STOP
Comment *
File
Password (Randomized for file and post deletion; you may also set your own.)
Archive
* = required field[▶Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options

Allowed file types:jpg, jpeg, gif, png, webp,webm, mp4, mov, swf, pdf
Max filesize is16 MB.
Max image dimensions are15000 x15000.
You may upload5 per post.


This board will be deleted next Wednesday. I am moving to a General on 8chan.moe /t/. This board is archived at 8chan.moe /hydrus/!

YouTube embed. Click thumbnail to play.

21e110 No.13500 [Open thread]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v379/Hydrus.Network.379.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v379/Hydrus.Network.379.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v379/Hydrus.Network.379.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v379/Hydrus.Network.379.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v379.tar.gz

Happy New Year! Although I have been ill, I had a great week, mostly working on a variety of small jobs. Search is faster, there's some new UI, and m4a files are now supported.

search

As hoped, I have completed and extended the search optimisations from v378. Searches for tags, namespaces, wildcards, or known urls, particularly if they are mixed with other search predicates, should now be faster and less prone to spikes in complicated situations. These speed improvements are most significant on large clients with hundreds of thousands or millions of files.

Also, like how system:inbox and system:archive 'cancel' each other out, a few more kinds of search predicate will remove mutually exclusive or redundantPost too long. Click here to view the full text.

18 posts and 3 image replies omitted. Click [Open thread] to view. ____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

423042 No.13527

File: 1000e7cd4f4bd52⋯.png (755.81 KB,1284x768,107:64,Untitled.png)

>>13519

No no, it turns out that I'm the stupid one here. When I used Pages>New Download Page>Watcher to download from 8kun, it'd say that the thread died and label it "unknown subject" like I said before. So I tried the other download page options and Simple Downloader with the "8chan thread (all linked files) forumula works for it. My bad.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

7f0121 No.13530

>>13527

Does it work for videos? I'm using 378 and the url downloader just skips videos since it grabs them by the filename rather than hash.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a8078e No.13533

File: 6ed44fe3c6360c7⋯.jpg (221.83 KB,1000x1300,10:13,6ed44fe3c6360c7d3e84e13e85….jpg)

I had a couple of difficult weeks, but I got some good work done. A prototype for an mpv video player embed is ready for advanced users to test! This new player renders video smoothly and finally adds audio support to hydrus. I have also fixed a heap of bugs, added some new imageboard downloaders, and improved the quality of life of a variety of UI.

I still have some things to do, and plenty to test, and messages to catch up on, so the release is likely to be late tomorrow.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a8078e No.13534

>>13523

Thank you. I have just fixed this now, let me know how it works for you. There are some other fixes in this area in today's release as well–please check out the changelog and let me know what else I need to catch up on.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a8078e No.13535

>>13530

>>13527

No worries. I am rolling out 8kun thread url recognition and parsing today, so you should be able to paste 8kun threads into a watcher page and it'll all work, videos and everything, in 380.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.



c3c7f9 No.13528 [Open thread]

Let's get this back going!

____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.


YouTube embed. Click thumbnail to play.

1ceb9a No.13478 [Open thread]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v378/Hydrus.Network.378.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v378/Hydrus.Network.378.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v378/Hydrus.Network.378.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v378/Hydrus.Network.378.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v378.tar.gz

I had a great, simple week. Searches are less likely to be very slow, and system:limit searches now sort.

all misc this week

I identified a database access routine that was sometimes not taking an optimal route. Normally it was fine, but with certain sizes or types of query, it could take a very long time to complete. This mostly affected multi-predicate searches that included certain tags or system:duration and system:known urls, but the routine was used in about 60 different places across the program, including tag and duplicate files processing. I have rewritten this access routine to work in a more 'flat' way that will ensure it is not so 'spiky'.

Also in searching, I managed to push all the 'simple' fPost too long. Click here to view the full text.

____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

bd6d2b No.13498

I had a great week. I focused on a variety of smaller jobs: better downloader UI, nicer search workflow, m4a audio file support, better select/remove menus, and more file search speed optimisations.

I caught the coughing cold that is going around, so my schedule is a bit out. The release will like be quite late tomorrow.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.



7d21b5 No.13482 [Open thread]

did the board go back in time?

____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.


YouTube embed. Click thumbnail to play.

69f3b8 No.13453 [Open thread]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v377/Hydrus.Network.377.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v377/Hydrus.Network.377.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v377/Hydrus.Network.377.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v377/Hydrus.Network.377.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v377.tar.gz

I had a good week. I mostly caught up with my smaller jobs queue, just pushing on work that had piled up.

Qt

Thank you for the continuing Qt bug reports. I fixed a variety of typo-broken buttons this week, mostly buried in less-used UI, and if you have set a browser launch path override in the options, hyperlinks across the program should obey that again. I also believe I fixed the annoying issue where media viewer hover windows that needed to shrink (because of switching to a new media that had fewer info lines or known urls) would linger too tall for one frame. EDIT: I'm still having slight hover window resize flicker in my IRL client when I keep my mouse over the top-right hover, I'll give it another go next week.

Post too long. Click here to view the full text.
7 posts and 3 image replies omitted. Click [Open thread] to view. ____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

c07151 No.13472

>>13469

>>13470

Thank you for the follow-ups. I believe this is fixed for 378. Thankfully, your sessions are not deleted from the db, they were just not showing in the menu.

A recent change in how the 'pages' menu is generated was relying on the 'backups' list for the main 'session names' list, but in the case where a session only had one recorded save, no backups, it was not included in the backups list. This would particularly hit older sessions that were saved before the backup system was established. I have fixed the way these lists are generated to check both areas for session names.

>>13469

For your time-skip issue, I think this is the fix:

Same deal with the sqlite3 executable console as in >>13461 above. We will rewind all your subs a bit just to make sure a new session gets saved on top rather than be discarded as the new 'oldest' one. First, we need to pick a number of seconds to rewind. If you reckon a couple of months difference, let's try rewinding them six months, or 86400 * 30 * 6 = 15500000:

.open client.db
UPDATE json_dumps_named SET timestamp = timestamp - 15500000 WHERE dump_type = 13;
.exit

Now any new session (with a timestamp of 'now', 1576526960 or so) should save on top of the backup stack. I will fix this issue this week and make sure that even if current backups appear to have been saved in the future, any new save will fast-forward. If you need to do some more clock-changing in future, I'll still advise you do it while the client is off, as all downloader/subscription/thread watcher timings are based on current time and may go haywire.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

c07151 No.13473

>>13468

Hey, I am sorry you have run into this. I believe you got hit by the tricky 335 update, which was the jump from python 2 to python 3. Your db is safe.

Hydrus needs a 'clean' install to clear out old python 2 dlls that conflict with py3 stuff. Forgive the tumblr URL, but here is the release post from then with full instructions:

https://hydrus.tumblr.com/post/181885931124/version-335

The basic thing is 'don't delete the db directory', since that is where all your stuff is stored. Delete everything else, reinstall/reextract, and you should be good. Please let me know if you still have any trouble. A big jump of 40-odd versions may have other issues, so you might like to try 345 or so first, boot it once, then 355, 365, 377.

Please also bear in mind that a few weeks ago a complete overhaul of the UI engine happened, wx to Qt. It is as significant a change as the py2->py3 step was. Hydrus looks a little subtly different now, but performance is significantly better. There are no special install instructions for this wx->Qt stuff.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

7b9344 No.13474

>>13472

Thank you so much, that fixed it! I don't expect to do any clock-changing for the foreseeable future, but I'll follow your advice if that changes.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

2856c2 No.13476

>>13473

Thank you very much, Hydrus dev!

I followed your instructions, cleaning out the old program and reinstalling thru 335 to 377 every 10 or so updates, and now the latest version is running well so far. I encountered 2 minor hurdles in the form of a "missing cache to rebuild" and the migration to the new PTR, both done smoothly.

If I encounter any unusual, less-than-optimal events I'll be back with details.

Thank you again for your help!

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

c07151 No.13477

File: f976e21aab1682a⋯.jpg (1.03 MB,1433x1900,1433:1900,f976e21aab1682aa1d9ee4f581….jpg)

I had a great, simple week. I mostly concentrated on code cleanup and database optimisation. A number of unusually very slow searches and routines (sparse system:known url searches particularly) should now be significantly faster. I also managed to push all the simple file sorts into system:limit queries, so now if you search with sort:filesize:descending and system:limit=256, you'll get the 256 biggest files of the whole sort, not a sample.

The release should be as normal tomorrow.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.



120bcd No.13456 [Open thread]

thank you mr. hydrus for the software

now i can download my furry porn from 8kun without having to fix my old scraper

____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

120bcd No.13457

ok nevermind it turns out that it does not work with 8kun

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

30b621 No.13462

>>13457

Yeah, unfortunately the DDoS protection is a block for now. You may have luck using Hydrus Companion to copy your browser's cookies across to hydrus, but I don't know if this will work for 8kun specifically.

https://gitgud.io/prkc/hydrus-companion

Once the whole situation calms down, I will roll out what I can to get hydrus working again with 8kun and the popular bunkers.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.



File: b3c853761fa1749⋯.jpg (626.58 KB,1690x1080,169:108,b3c853761fa17492fad032b195….jpg)

08b31d No.13452 [Open thread]

I had a good week. I focused on catching up on smaller work. More bugs are fixed, there's some neat new shortcuts to pan media around, the UI should be smoother when importing files and saving sessions, and the system:known urls predicates now run much faster.

The release should be as tomorrow.

____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.


YouTube embed. Click thumbnail to play.

a40db4 No.13444 [Open thread]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v376/Hydrus.Network.376.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v376/Hydrus.Network.376.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v376/Hydrus.Network.376.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v376/Hydrus.Network.376.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v376.tar.gz

I had an ok to good week. Subscriptions run better and there are more UI fixes and basic theming for Qt.

subscriptions

Subscriptions have always been resource heavy. As last year's downloader overhaul extended their capabilities, we saw increased CPU lag with many subscription operations. I had hoped to completely fix this this week, even permitting subs with >250,000 items cached, but the surgery needed to figure this out was more complicated than I expected. While I otherwise prepped for this next step, I also tightened up subscription timing code and wrote a new scheduler that reduces overall subscription resource needs.

Subscriptions will now try to load when they are due, and only then. So, if a quPost too long. Click here to view the full text.

____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.


YouTube embed. Click thumbnail to play.

94a0a3 No.13437 [Open thread]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v375/Hydrus.Network.375.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v375/Hydrus.Network.375.-.Windows.-.Installer.exe

macOS

app: https://github.com/hydrusnetwork/hydrus/releases/download/v375/Hydrus.Network.375.-.macOS.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v375/Hydrus.Network.375.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v375.tar.gz

I had a great week. There are a bunch more Qt fixes, and a few other things as well.

Qt

I have fixed a bunch more bugs in the Qt code. We are getting to the end now–this is mostly smaller stuff like an unusual dialog button not working, but I have fixed another important memory leak that was causing some backend not to be deleted correctly when a media viewer closed on a video. This should radically reduce memory use for some heavily used clients.

Some windows that were large, or could expand to be, like the options dialog on some pages, were sizing off the edge of the screen. This should be fixed now, and a variety of child-wPost too long. Click here to view the full text.

1 post omitted. Click [Open thread] to view. ____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

94a0a3 No.13439

- .

- new file maintenance jobs:

- added a new 'check for missing files' file maintenance job, where if the file is missing and has urls, those urls will be queued up in a new url downloader for redownload. the file record is not removed, preserving archive/inbox and import time

- added a new 'check for invalid files' file maintenance job that does the same deal as above with an additional expensive byte-for-byte content check if the file is not missing

- added a new 'check for invalid files' file maintenance job that only cares about invalidity–if the file is present and invalid, it is moved out but the file record is not removed

- .

- the rest:

- network jobs that receive low-bandwidth error codes from the server now use a separate wait routine (previously, they piggybacked on the connection fail retry system). they have a separate cog-menu action to override these waits

- the time delay multiple for connection errors and serverside bandwidth problems are now editable under options->connection. old default was 10 seconds base, now 15 and 60 seconds respectively

- updated the danbooru login script

- improved the precision of the thumbnail size estimate in database migration

- the alphabetisation of a url class's GET paramaters on normalise is now optional. it is a new checkbox on the url class edit panel

- when a default object fails to load from a png path, a simple error is now written to the log

- misc cleanup

next week

I got hit by some IRL stuff right at the end of this week, including some Thanksgiving surprises, so I couldn't fit in the new downloader fixes I wanted to, but I have some fixes for pixiv tag search and twitter video download waiting to be added. I also couldn't get to Qt theming again, nor macOS tab DnD! So, I'll focus on those first thing so they definitely get done.

Otherwise, next week is scheduled to be a 'medium size'Post too long. Click here to view the full text.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

94a0a3 No.13440

test BO trip

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

502ec9 No.13441

Curiously this thread only shows up in the catalog. At least for me.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

d48852 No.13442

>>13441

Yeah, I have had to force-refresh a couple of threads on the site to get new posts, and a couple of the BO things didn't work immediately after migration. And things are obviously sluggish. Seems to be getting better though!

My only real peeve is that file downloads now don't have the content-length header sent or something, so my web browser doesn't have a nice progress bar as they download. And the anti-DDoS blocks hydrus html for now unless, I assume, you send the vanwatech cookies with Hydrus Companion or similar. JSON API seems to work, so maybe that is better.

I keep meaning to do this job but keep not getting around to it, but I'll roll out new thread watcher support for the big bunkers, and 8kun, in a release soon.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

7a4da2 No.13443

File: df048ec65ccb387⋯.jpg (85.56 KB,496x768,31:48,df048ec65ccb3878338698ca38….jpg)

I had an ok week. Subscriptions now operate more efficiently, using less resources and choosing their sync times more precisely. Some simple Qt theming support is added for experimentation by advanced users, and a number of bugs are fixed, including page tab drag and drop for macOS users and anyone else with a center-aligned tab bar.

I have a busy day, so the release will be quite late tomorrow.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.



File: 52b73b28ef460e8⋯.jpg (160.95 KB,1920x1080,16:9,52b73b28ef460e8ffbbd05ff52….jpg)

9b0adc No.13436 [Open thread]

I had a great week. I have fixed some more Qt problems, including some bad window positioning, thin popup message widths, a video memory leak, and another round of improvements for high-dpi displays. Also are some improvements to the file maintenance system for recovering (including auto-redownloading) missing files, some better network retry timing code, and new downloaders for some broken sites.

The release should be as normal tomorrow. I will be making a release post here as well as Endchan.

____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.


YouTube embed. Click thumbnail to play.

bf06a9 No.13387 [Open thread]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v362/Hydrus.Network.362.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v362/Hydrus.Network.362.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v362/Hydrus.Network.362.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v362/Hydrus.Network.362.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v362.tar.gz

I had a mixed week. The duplicates overhaul work is finished.

duplicates work finished

The duplicates storage overhaul is done! Everything is now on the new storage system, the duplicate filter has had some quality of life attention, and now there is some updated help:

https://hydrusnetwork.github.io/hydrus/help/duplicates.html

If you have used hydrus for a bit but haven't checked out the duplicate system yet, this is a good time.

Post too long. Click here to view the full text.
12 posts and 6 image replies omitted. Click [Open thread] to view. ____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

14b112 No.13427

>>13399

Not yet. I had a tentative job to explore post-duplicate metadata merge/permanent synchronisation in this iteration, but adding that system was bigger than I had time to do well. It'll be slightly complicated, and I don't want to mess it up with a quick hack.

So just to be clear, for now the metadata merge only works on the pairs you see in the dupe filter. But since both representatives from the dupe filter are the Kings of their respective groups, and therefore 'good' data travels up from groups to kings, you can generally assume that Kings will have fairly 'complete' metadata based on previous decisions for now.

Since duplicate records are permanent, once I do add something like this, we'll be able to retroactively apply whatever merge settings you prefer, so non-Kings of groups will all sync too in an ongoing basis.

I suspect when I do this I'll have different or better default duplicate merge options here. Some tags aren't appropriate to merge backwards, like 'high resolution' or 'webm', and it'd be nice to have examples of those as default, or easily selectable, rather than making users awkwardly figure it out on their own.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

14b112 No.13428

>>13408

I am not certain. I think I remember reading about one user getting it to run on FreeBSD some time ago, but I can't be sure. The program is devved on and works best for Windows, and I build the Linux release on Ubuntu, so some more esoteric flavours of Linux and their Window Managers can have trouble with my build. Your best bet is probably running from source, for which I have some help here:

https://hydrusnetwork.github.io/hydrus/help/running_from_source.html

There's an Arch package maintained by a user, but I don't know enough about FreeBSD to say what is compatible:

https://aur.archlinux.org/packages/hydrus/

I assume that wouldn't work for you, right?

If you have a Windows machine, I recommend you try hydrus on that just to see if you like it first, rather than going through the rigmarole of getting it going on your FreeBSD machine and finding it isn't what you like. If you do give it a go, let me know how you get on!

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

14b112 No.13429

>>13417

Not yet, but as it happens I am on the verge of doing some heavy work on local tags and I hope to add it then. I'll do a couple of weeks on some audio metadata stuff and then start on all that.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

5e6830 No.13433

>>13425

I had added the lookup script for Deep Danbooru but couldn't get it to work; it would get stuck in the uploading stage and quietly fail out. But the errors didn't pop up till maybe 20 minutes after my various attempts so I wasn't sure if they were related or not.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

b5f746 No.13434

>>13426

Like I said I think, the reason this one works for the jpeg being larger is due to the text, at least that what I think. to get clean text on a png, it's more or less par for the coarse and doesn't eat much space, but for a jpeg, to keep it without any kind of flaw, at least visually, it would need a stupid amount of resources. edge cases like this, or potentially others like this where a jpeg is made, and my best guess is one of the joi threads on 4chan had someone blanket convert with the highest quality settings on them, hit this image and it tossed out a larger file then put in. i'm guessing that 2/3 or above of the image being something jpeg does not handle well had a part in why this exists the way it does.

as for an auto decision, I don't see how the difference between 'the pixels are the same choose jpeg' or 'the pixels are the same choose lowest file size' either way, 99%+ of the time it will go jpeg anyway, this would just introduce a safety net for the sub 1% of the time jpeg isn't the smallest choice.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.



File: 48cb0ba54a838aa⋯.gif (171.58 KB,500x500,1:1,48cb0ba54a838aa77fac4f1bd7….gif)

2e1689 No.13278 [Open thread]

I had an ok week. I did some of the last duplicates work, including matching zooms for files with different resolution ratios and adding a neat pixel-for-pixel duplicate detector, polished some of the logic for the new tag autocomplete search, and cleaned and fixed a bunch of code.

The release should be as normal tomorrow.

11 posts and 1 image reply omitted. Click [Open thread] to view. ____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

8eec11 No.13348

>>13332

This is stupid but my laptop only has 4GB ram, and normally I'm able to keep my browser and Hydrus open at the same time, and Hydrus is often frozen but still usable sometimes. But in trying to load a tab with 139,198 images at once, Hydrus remains frozen for 40+ minutes at a time for me, and I had to sit there watching it in my peripheral vision to catch it for the seconds in a row it was unfrozen.

I managed to load the images, highlight them all, press f3 to tag, and paste the desired tag in from a text document where I manually typed the tag when Hydrus was frozen. I even seemingly successfully pressed accept, because a subscription window partially loaded, signifying I returned to the main client window. But on this partially-loaded subscription window on the main client window, it's been frozen for like 10 hours now.

My measly 4GB ram never dips below 95% usage wih only Hydrus open in this state, with Hydrus using from 3 mil to 3.2 mil. So I was just wondering if there is a way to tag without loading images… like if I could perform the same search with arbitrary exclusions if possible, but without the burden of the client loading it all, or any of it in one tab.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

8eec11 No.13354

>>13317

>Then I suggest you open a big search for all files with a pixiv url but no id tag and then ctrl+a and right-click->known urls->copy->pixiv file page urls, then open a new 'url downloader' page and set the tag import options to get tags even if urls recognised and file and in db, and then paste all those urls in. It should catch you up with the missing tags without trying to download the actual files again.

I just did this, and the number it shows is 0/12,025, when my search of "r-18" and "-pixiv work:* only showed 8,499 results. So I was worried it would only fix the r-18 tagged images, but it must've gotten everything instead. So I think it works, thanks.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

b2bbc6 No.13367

>>13329

Great. Although the new downloader system is amazing compared to the old rubbish I had, it is not always pleasant to work with with the duplicate objects and so on. The next iteration here will have versioning or something and auto-updates.

>>13331

>>13332

I hope to add namespace siblings in some tag repository work that will start in the next 3-5 weeks! This has long been requested.

>>13348

Yeah, hydrus performance bombs out at about 30-50k thumbs in a single page. And manage tags doesn't work well for selections above about 8k thumbs. I am afraid the code just can't handle this stuff yet. Next step here is to improve manage tags so it does big operations asynchronously with a nice cancel button, and then yeah, I'd like some sort of faster tech for various large operations overall. The main problem for manage tags is when you try to add a tag, it needs to check all the files in the selection if they already have the tag in order to present you with 'pend to 3 files/petition from 2 files' kind of decisions. It does not scale well (and I think the final 'approve' stop is O(n²)), so for 10k files it just takes an unreasonable amount of time. As well as code optimisations, some CPU-faster entry actions (like 'woah lad, there's a lot of files here, let's make some add-only decisions beforehand') will come at some point.

>>13354

Great!

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

8eec11 No.13372

>>13367

Thanks for taking the time to read and respond to my random feedback on trying to add "pixiv work" tags to those that missed them the first time around.

I wanted to say too though that actually I was mistaken- the "url downloader" page only added "pixiv work" tags to the images tagged "r-18".

I was able to tell as much because I found a western artist who used to tag his images "r-18g", which is the gore tag, rather than mere "r-18" to mean adult. So every one of his images before he started using the "r-18" tag still didn't have a "pixiv work" tag.

I think I may not have a meaningful way to add "pixiv work" tags to every image that failed to get it the first time, in the end. Because even if I manage to do the "url downloader" for as many popular tags as I can, there will always be images that weren't tagged correctly.

But the only reason anything even has an "r-18" tag is because I used a "gallery" tab and put in a pixiv member id. I thought I could use the fact I've done that to fill in anything missing a "pixiv work" tag somehow.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a4df90 No.13386

File: 02db0e20484d1d2⋯.png (149.02 KB,1293x1392,431:464,pixiv.png)

>>13372

Hmm, that's odd. I just did a test for this:

https://www.pixiv.net/member_illust.php?mode=medium&illust_id=75961157

As in pic related, and it looked ok. The actual json parsing rules for pixiv work tag look sensible and don't seem to rely on some clever lookup path that goes through r-18 gubbins anywhere.

I am not sure. I am not a big pixiv guy and didn't write the new parser, so I can't talk too cleverly.

This is a small thought, but is there any chance some of those files that couldn't take a 'pixiv work' tag actually had it deleted in the PTR (if you are adding to PTR)? If you hit F3 on one of them that you know didn't work and then hit the cog icon on manage tags and tell it to show deleted, is there 'pixiv work:123456 (X1)'? I think I have deleted some before after some large petitions came up on the PTR regarding some of this 'meta'-style content.

The 'what to do when a tag is deleted' workflow has always been a bit shaky. I'd like to clear some of it up when I move onto this PTR work in 2-4 weeks with better overall local control of what tags you see from difference sources, moving solutions to 'I like this/don't like this' preferences away from the existing 'this is correct/incorrect' workflows.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.



YouTube embed. Click thumbnail to play.

a692a4 No.13293 [Open thread]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v361/Hydrus.Network.361.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v361/Hydrus.Network.361.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v361/Hydrus.Network.361.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v361/Hydrus.Network.361.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v361.tar.gz

I had an ok week. Some final duplicates work is done, and there is some polishing to the new tag autocomplete.

duplicates

The duplicates filter can now detect if static images with the same resolution are pixel-for-pixel duplicates! If they are, it gives one of the standard 'comparison' statements on the right hover window. Furthermore, if one file is a png and the other not, this statement will colour green/red and bias heavily to the non-png, since the png is likely a bloated 'clipboard' duplicate that you don't want. Pixel summary data is not cached long-term, so this routine takes a bit of extra CPU. It only kicks in if both files are images with the same res, but nonetheless please let me kPost too long. Click here to view the full text.

17 posts and 2 image replies omitted. Click [Open thread] to view. ____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

ce7dcd No.13362

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

e11d3d No.13368

>>13349

That resolution stuff is an interesting thought. I think trying to solve the problem of what is a useful resolution is beyond the time I have left in this current big job, and I should leave it to human eyes for now. I can add some common resolutions for now, though, and I can detect odd-numbers vs even-numbers easy, and we can iterate on that later if we see other easy answers. I'll make a job for nice resolution ratios, but it may come later.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

780cd7 No.13373

is there a way to filter 'pixel for pixel' duplicates?

would love to be able to get those hammered out and could do that very fast too.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

89b8d8 No.13378

>>13373

Not yet–this data is currently generated on the fly in the duplicate filter. But I think it is very worth caching it in the db at some point. I can search it incredibly quickly to find pixel dupes and get started on some auto-dupe-resolve rules like 'if two files have the same pixels and one is a png and the other is not, delete the png' so we can finally and automatically clear out Clipboard.png trash dupes. It would be fast enough that I could even add a hook for it on import!

This would be optional and probably default on. More complicated stuff like 'if two jpegs are the same, keep the one with smaller/larger filesize' would be default off and up to the user.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

780cd7 No.13384

>>13378

I would set both rules to delete the larger file size, it's very rare, but I have come across some jpegs were saving as png did lower file size.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.



File: 3b0690bc96c6a5d⋯.jpg (88.43 KB,720x725,144:145,3b0690bc96c6a5da52cbec5f5f….jpg)

f7bf46 No.13380 [Open thread]

I had a mixed week. The duplicates overhaul is done, with some final duplicate filter improvements, advanced thumbnail commands, and some updated duplicates help, and I fixed a heap of bugs.

The release should be as normal tomorrow.

____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

339ff7 No.13382

File: e941a0c31d0039f⋯.png (963.13 KB,1500x793,1500:793,e941a0c31d0039fc2cd193cfbf….png)

>>13380

I'm always looking forward to duplicate filter improvements.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.



YouTube embed. Click thumbnail to play.

ad39a7 No.13223 [Open thread]

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v360/Hydrus.Network.360.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v360/Hydrus.Network.360.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v360/Hydrus.Network.360.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v360/Hydrus.Network.360.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v360.tar.gz

I had a great week. There are a bunch of little things and an important speed overhaul to tag autocomplete.

tag autocomplete

I have never really liked the tag autocomplete workflow. It once blocked the UI completely, and some unusual timing options were needed to make it even useable, but it still often lagged out or just responded with gigantic lists in judders. After chipping away at the problem, this week I am finally updating it to what I really wanted.

So, the main change is that autocomplete results are now fetched as soon as you type. It responds very quickly and overall, I think, feels great, particularly once you have put fourPost too long. Click here to view the full text.

33 posts and 10 image replies omitted. Click [Open thread] to view. ____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

9ecf9b No.13321

>>13259

Thank you for this feedback. That's definitely the direction I'd like to go. The file maintenance system has proved a good step forward, so now I'd like to do similar for db-wide actions like analyze and repo processing. Get everything working on the same pipeline and timers and checks, and then I can neaten up the user feedback of 'hey, I need to do x, y, z big job, is this a good time?' into one system rather than the current cobbled-together system that consults several different maintenance clocks all with their own settings.

As a side note: standard python is restricted to using one core as a language. I can bump up CPU if I go into low-level C++ hard math stuff, or by calling other processes like ffmpeg, but up in my high-end, the language has a Global Interpreter Lock (GIL) that I can't get around. I have threads and everything, but none of them actually execute simultaneously on different cores.

The biggest sources of lag and latency in hydrus are due to my shit code rather than hardware/language limitations though. I just need to put the work in to make it neatly async, like the recent autocomplete improvements.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

9ecf9b No.13322

>>13260

Unfortunately, the current downloader system doesn't support the idea of 'removed' results. It always treats a new gallery request (or thread check) as an 'append any new stuff' action and doesn't take into account urls that have dropped off (which in the gallery context could have just cycled out to the next page). I think any clever additions to the downloader system will have to wait for the next big iteration of work on it.

I like the idea of marking threads in some way for later processing. I had always wanted to make the individual thread items drag-and-droppable so you could migrate them to other/new watcher pages, but I ran out of time last time. My best suggestion for now is to multi-select and right-click to say 'show all selecteds' presented files' or whatever, which will show all the files for several at once. If it all looks legit, and the threads are DEAD, I then recommend you drag and drop the big selection of files to a new processing page or give them a later-filtering-lookup rating/tag, and then clear out the DEAD threads.

>>13266

The new 361 should let you add specific potential pairs from the right-click menu. I am vacillating on whether I should/can easily add something like a 'launch dupe filter from here' command from a thumb selection. I think adding a temp tag/rating you can find again in the dupe filter is a good idea, or just picking your favourite from the thumbs in front of you may also just be easier, at least for smaller alternates groups. A quick browse through ten files in a row and choosing 'yeah, the one where all the girls have a benis and runny makeup is the best' when they are all in front of you and just deleting the others may be faster than going through them by pairs later on.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

9ecf9b No.13323

>>13268

Hey, just fyi: the 'speculative' search and the other options are inclusive of the more narrow choices. Speculative is really just 'hamming distance <=8', so it includes the others, which are 4, 2, and 0.

When I eventually add videos, I'll add gifs as well. It'll work on frames, so your pokemon gif should match enough of the still images to get linked up with the eventual dupe/alternate groups you create here, and will appear in 'speculative' searches and so on.

For finding things in trash, if you change the search domain, either on the duplicate files page or a page initialised with a 'similar files' system pred, from 'my files' to 'all local files', that should include trash. You may need to be in help->advanced mode to see this option.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

9ecf9b No.13324

>>13271

>>13270

Hey, you should be able to do this from manage tags right now, although it is advanced and debug-tier UI.

First, make sure help->advanced mode is on. Then select your thumbs and open manage tags. Hit the cog icon and then 'advanced operation'. It'll load the same mass-copy dialog you can do service-wide from review services, just limited to that selection. Double-check you have everything set how you want, and then click Go!. Make a backup before you get into this, as it is a powerful tool and if it goes wrong, there is no easy undo.

This dialog will get a pass in the upcoming PTR/tag services work I am planning. I'd like to add the tag filter object to it so you can more finely determine which tags are moved around.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

9ecf9b No.13325

>>13285

>>13286

Yeah, the current duplicate files search is a big atomic job, and if it goes on too long with a manually started job, the UI gets caught up until it eventually finishes. I don't like it, and as I move it to this new unified maintenance pipeline, I'd love to have it work in smaller async chunks like the file maintenance stuff.

Since you have a big db, I recommend you have it work in idle time (the cog icon on the duplicates page should allow this) and let it catch up in that system, which has improved cancellability.

I generally recommend users start with 'exact match' (i.e. 0 hamming distance) search to start with and work that queue completely before going for larger distances. This is much faster than the other searches and gives you easy decisions first. Because of the internal logic of the duplicates system, easy/actual duplicate decisions are more powerful/useful than alternate decisions–they eliminate the queued pair count faster.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.



Delete Post [ ]
[]
[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]
| Catalog | Nerve Center | Random
[ / / / / / / / / / / / / / ] [ dir / random / 93 / biohzrd / hkacade / hkpnd / tct / utd / uy / yebalnia ]