[ / / / / / / / / / / / / / ] [ dir / random / 93 / biohzrd / hkacade / hkpnd / tct / utd / uy / yebalnia ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.
Name
Email
Subject
REC
STOP
Comment *
File
Password (Randomized for file and post deletion; you may also set your own.)
Archive
* = required field[▶Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options

Allowed file types:jpg, jpeg, gif, png, webp,webm, mp4, mov, swf, pdf
Max filesize is16 MB.
Max image dimensions are15000 x15000.
You may upload5 per post.


This board will be deleted next Wednesday. I am moving to a General on 8chan.moe /t/. This board is archived at 8chan.moe /hydrus/!

YouTube embed. Click thumbnail to play.

a74b8e No.10810

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v332/Hydrus.Network.332.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v332.tar.gz

I had a great week catching up on a whole bunch of small jobs.

rolling session backups

The program now automaticaly makes up to ten rolling backups (i.e. it keeps the ten newest saves) of your sessions! This includes the 'last session' backup used to restore the program on boot, so if you accidentally close something very important and can't get back to it, or if your session breaks for some reason, you can now rewind a bit further.

You can access the backups under pages->sessions->append session backup. They are listed by the time they were made.

The client also now saves an 'exit session' on a clean client shutdown. This is different to the 'last session', which is updated every few (default 5) minutes, and provides a longer-term backup.

This works for the entire in-client serialisation system, so if this works out, it may make sense to apply it to other objects such as the bandwidth tracking system, just to provide an easy additional layer of safety.

misc

There's a bunch of small stuff this week:

Gallery import pages now have a cog button beside the query input that lets you control if new imports start with files and/or search paused! It remembers its values through a session reload.

Tag import options that can be 'default' now have a button to specifically load a default, so if you want to do a specific download that has exactly the normal default options but one little thing different, you can just load it up and edit it rather than having to recreate the whole thing from scratch.

The new tag id cache is expanded, and I've written a similar cache for file ids. A whole bunch of database operations should be just a bit faster.

Export folders can now be told to delete the exported files from the client after they run. (So you can, for instance, tell your client to send all files with 'nsfw' to an import folder for another client.) If you decide to do this, it'll throw up some warnings just to make sure you are ok with it.

When running database->check->file integrity, the client can now optionally export a .txt file with all the missing files' known urls! This same list can be pasted right back into a new url downloader in the client to try to get them back!

full list

- the client serialisation system now supports multiple rolling backups!

- client sessions (like the 'last session' that typically loads on boot) are now automatically backed up to ten times in rolling backups! you can review and append the backups if you need to recover from _pages->sessions->append session backup_, where they are listed by their timestamp!

- when the client closes, an additional 'exit session' is now saved. this differs to 'last session', which is overwritten every x minutes, and is now available (especially with the new backup) for various error recovery situations

- gallery import pages now have a little cog icon to control if new import queues will start with files and/or galleries paused. these states persist through a session reload

- tag import options that are 'defaultable' now have a button to let you load in a specific default, so you can easily quickly edit in a one-time slight alteration of the default rather than having to create everything from scratch

- under options->downloading, you can now set the 'delay' times on gallery/watcher network error, subscription network error, and subscription other error, now defaulting to 90mins, 12hours, and 36 hours respectively

- attempting to launch a client with db version > software version now spawns a blocking messagebox on pre-launch informing the user of the risks and advising task manager force-kill of the process

- did a little cleanup on the new tag id database cache and merged some other, older semi-laggy tag-fetching code to use the same system

- wrote a similar 'file id' database cache for caching file hashes and also merged some old hash-fetching code into it. a variety of file operations are now significantly faster

- export folders can now delete files from the client after export. the edit panel will warn you on selecting this and oking the panel. it can't be set if the export type is 'synchronise'.

- fixed the edit export folders dialog's old buttons, which were semi-working due to some recent update work

- when checking for file integrity, you can now choose to export a .txt file listing all the missing files' known urls, so you can try to recover by feeding them all back into a new url downloader!

- the physical file deletion process is now a little simpler and deals with larger jobs in smaller batches, no longer hogging the file read lock the whole time. clearing a large trash should no longer hang other media loads

- adding ngugs in the 'export downloaders' dialog now attempts to add the respective gugs as well

- fixed an issue where the listctrl would accept certain kinds of duplicate data and hence confuse its indices

- fixed all the add buttons on the export downloader panel to exclude all items already in the list when figuring out what additional objects to add

- the 'scroll thumbs at this rate per tick' option now _rounds_ the pixel result, rather than always _floor_ing it

- wrote a new mass-selection database routine that should reduce memory footprint of autocomplete regeneration. if it works out, I will use it in some other places

- improved the errors when a network job that cannot wait fails on an invalid login, and added a separate error when it refers to a hydrus service

- fixed a small typo bug when trying to auto-add url classes and parsers

- updated an old deprecated checkboxlist call

- clarified the integrity section of 'help my db is broke.txt'.

- misc fixes

next week

Next week should be more small stuff like this. Maybe some fun/new for 333 if I can fit it in. I mostly want to tidy up for a 'python 2 final' 334 on the 12th Dec. On the 12th, I will break up for Christmas to convert the program to python 3.

____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

50c7ae No.10813

>>10806

that solved the issue though it also takes away 10 potential button presses, I don't think it matters much as I can change out the shortcuts i'm using.

with that said, in the custom shortcuts, would it be possible to have a drop down menu for them and a check mark next to them when selected, kind of like the pause menu stuff, at least this would be a nice in ui way to switch to what you are using without going full screen, loading your short cuts, and needing to do this everything you exit full screen.

granted just a note this isn't something I would use immediately, but it would be a nice feature when I finally get to a second phase of rating images into categories.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

50c7ae No.10814

I have one issue with saves, a while back you found out they caused a crap ton of hang for me, I have also noticed that when the client goes into what I think is a save, it eats 4-6gb of ram bringing the running ram use of hydrus from 12 to 18

not sure if there is much you can do, but a less ram intensive way to save would be nice.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

976f7c No.10815

Any chance you could raise the thumbnail width and height limits? 200x200 is too small for my eyesight.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

50c7ae No.10816

File: 89f94cecdab3217⋯.jpg (99.83 KB,1117x400,1117:400,client_2018-11-28_20-59-31.jpg)

>>10815

i'm wondering this too, but not an out right 200x200+ thumbnail, but a scaled up thumbnail where its still 200x200 but I could have it display as 300x300. at least the way I have things set up, I have a very large preview window, nearly 900x900, this serves almost perfectly for my needs but if I just want to sort via thumbnails, being able to scale 50-100% bigger would come in handy, sure at 50% I may only have 5x6 or at 100% i would have 4x5 at current hydrus size, but this would also likely be faster as I don't have to take as much time sorting,

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

50c7ae No.10817

File: 37b3b9e16b6ee58⋯.png (10.67 KB,578x408,17:12,taskmgr_2018-11-29_01-01-4….png)

here is an image of the save issue where it takes a crap load of ram, at least I believe its saving. for the entirety of that section, the program is locked up.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

354fcd No.10818

I never got around to testing the new login scripts for rule34hentai back when it was added to the new login scripts so I'm not sure if this was recently or always been this way but the downloader for that site still doesn't pick up hidden images where you need an account to see. Login works fine I guess, it knows I'm logged in. I'm just not sure if its the downloader or the login script that's causing Hydrus to ignore hidden images when downloading from rule34hentai.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

50c7ae No.10824

Ok, while its not as big of a problem as it use to be, tabs are moving around fairly regularly in the program.

The biggest problem for me is that all my top level things are page of pages, and those are starting to move around, making it difficult to put the program back in order when things want to move.

Is it possible to get tabs to either lock adding new things to it or lock tab positions?

Hell, if you have a tab/thing I can put in that just acts as a spacer between tabs, that would help immensely with re organizing because i'm not putting shit into tabs when trying to put everything back in place.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

65a947 No.10825

Can you implement a setting on how much the main thumbnail view scrolls with one tick of the scroll wheel? Because the setting I'm using on my mouse now which works well for most other programs scrolls the thumbnails view way too much. One tick scrolls about 2 pages right now. Ideally I'd like one page at most.

I just updated to Win10 from Win7 and it wasn't a problem there.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

190dfa No.10834

An issue I have now is with the PTR sync. It recently got stuck for some reason. So I decided to reset it, which worked. But it's now so very slow at re-processing the updates It's only working at a bunch of dozen rows per second but especially when it seems to have done one update (say, 880/1621) it just gets stuck at the end for minutes to half an hour - often it seemed like it just wouldn't continue.

What's going on there? The database should be stored on a relatively fast HDD on XFS. I recall various Java databases (h2, hqsqldb, cassandra) and other databases (rocksdb and so on) were able to insert k-v or full tuples in the order of quite a few thousands to tens of thousands per second with no weird hangups.

>>10814

I was also having some issues with Hydrus' RAM usage. Maybe it was also mainly during saving, though I also couldn't explain why it was using x GB RAM (can you tell why yours is using 12GB RAM?).

It doesn't seem to keep all the full sized images and videos loaded in RAM, nor should it have to, right?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

50c7ae No.10839

File: 5ec9871ce1c7743⋯.png (7.02 KB,401x173,401:173,client_2018-12-02_03-18-52.png)

Ok, in making an effort to process my images, I halve the window

as in I select half, and I drag and drip them to another page of pages nested inside each other till they are a reasonable sub 1000 size, see image for example.

However in doing this I found out that moving one to another has started to just combine the files back into one bundle. its not a pressing bug to me to fix, but its something that I think needs to be addressed because I think this will cause problems in the future.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

e1629a No.10841

VERSION 333 - SOON TO BE WITH THEE

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a74b8e No.10854

>>10813

Sure, this is a good idea.

>>10814

That seems pretty huge! I assume this will blow up your client.db size as well, if up to twenty copies (last_session and exit_session) of your main session build up. If the db size becomes a problem for you, I can add an option to change the number of rolling backups kept.

This answer is going to suck, but the best way to reduce memory usage is to reduce the size of your session. I do not recommend having more than 50k thumbnails/urls total in your session, so if you have 800k or something, that is obviously going to add up when it comes time to serialise it all and save it to db.

I explored a 'what's in memory' debug system a while ago, but didn't have luck getting a stable solution. I am going to try again in python3 in the new year. If I can see what is actually eating that memory, we'll be able to figure out how to reduce it.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a74b8e No.10855

>>10815

>>10816

Due to some technical bullshit, I can't do this by just increasing the number. But I would like to do something here. I am afraid it did not score highly on the 'big things' poll, so we may have to settle for a dirty solution rather than a beautiful one.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a74b8e No.10856

>>10818

I have not used it myself, and I don't think I wrote the shimmie login, so I can't comment cleverly. If it has logged in truly correctly, it should show what your regular browser login shows. Maybe try going manage logins and then hit 'reset login', to force it to try again.

If it is still broken then, let me know and I'll dig a little deeper. It might be r34 is returning some specific error and the script isn't picking up on it, thinking everything went ok.

You might like to try help->debug->network actions->fetch an url and put in a url you expect to have access to. That should let you save the html, so you can scan it in an editor or load it in your browser and see if it looks right or if it says 'not logged in m8'. You can pastebin the file to me, but be careful if it has private info like your username in it.

Let me know what you find out.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a74b8e No.10858

>>10824

>>10839

Thanks. I think the nice answer here is to improve the ui code so it doesn't stutter and hence interpret some clicks as tab drags.

But maybe in the meantime I can just have an option to disable them, or yeah, pin them. I like the idea of pinning most, I think. Pin against close and pin against move.

If it helps, and you haven't seen, you can right-click those page tabs to get a bunch of different move/collect options. This might be more reliable than DnD for you.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a74b8e No.10859

>>10825

Try options->thumbnails->rate per scroll tick. This is a little experimental. Typically, for every physical scroll, the OS delivers 3 scroll events to hydrus, so a default client tends to scroll three thumb heights per physical scroll. Two pages at once seems high. How high are your pages, approx, in thumb heights?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a74b8e No.10860

>>10834

Check your options->speed and memory to juggle the size of the caches in the client. It usually reserves a few hundred MB for things like image and thumb caches. Most of the size of giganto clients is in huge sessions that have hundreds of thousands of import objects or thumbnails. I recommend clients not have more than 50k thumbnails/urls total in their main session.

For your PTR sync, the main bottleneck on HDD clients is random access read latency. You can improve this a bit by increasing the db disk cache under than same options panel. Since your reset tables may be new and un-analyzed, you might like to try database->maintenance->analyze->force as well. Failing that, here's some help to help get me more info:

https://hydrusnetwork.github.io/hydrus/help/reducing_lag.html

The best solution is to move your db to an SSD though, where processing time typically increases to 20-70k rows/s. Putting the db on SSD and files on HDD works great:

https://hydrusnetwork.github.io/hydrus/help/database_migration.html

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a74b8e No.10874

File: c44291cc0020dcb⋯.gif (1.99 MB,332x215,332:215,c44291cc0020dcb0d9b5e52c70….gif)

File: 75ef90c76ef991a⋯.gif (147.14 KB,517x469,517:469,75ef90c76ef991a0aabf32dc8d….gif)

File: a5c6975fc6d5f7b⋯.png (232.91 KB,500x447,500:447,a5c6975fc6d5f7bf1f07b90291….png)

File: d68b759e4c9d51e⋯.jpg (142.99 KB,578x600,289:300,d68b759e4c9d51e37d50779b57….jpg)

File: 3f166a4498f7ffd⋯.png (219.37 KB,450x450,1:1,3f166a4498f7ffd6c67546ebb3….png)

https://tumblr.zendesk.com/hc/en-us/articles/231885248-Sensitive-content

>Starting Dec 17, adult content will not allowed on Tumblr

>will not allowed

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

50c7ae No.10876

>>10854

the number you are looking at is 40k before the client starts to have lag spikes, with everything split into sub 1000 image tabs, its going to be alot easier to just go through it now, but it will take some time.

as for db size, its gone from 28gb to 37gb this version. granted this is more an issue when backing up then it is an issue on the nvme

>>10855

for me that's in the would be nice category

"Add an optional system to record why files are being deleted"

Is in the critical one, though its not doing a whole hell of alot better. I have a deadline where this goes from would be really nice to have to shit needs to be there to use the program at all and that's 721gb, so possibly another year and some change before the 3tb drive runs out of space.

Like I said a while ago, user notes remembers on delete, if user notes would just be linked to the downloaded files I could at the very least set up some macros for common things and just add them to the files notes, I would see them on re import. any system you make here doesn't need to be a forever system, it just needs to be a system that won't lose its data when the forever system becomes a thing.

>>10874

Here's hoping that they don't move to fucking twitter or some other shit and go to an actual porn hosting site like hentai foundry. will definitely miss short stack approved though.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

295ce3 No.10877

The tag import copy from defaults does not copy blacklists.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a55e7b No.10878

>>10874

SHUT THE FUCK OUT KID DO YOU KNOW HOW MANY DOWNLOADS I STILL HAVE

At least I'll have less filter-related dupes after this.

>>10876

They're already moving to Twitter like the Japs. Some to Newgrounds too, of all places, because they gave a timely reminder on Twitter. Shit's fucked, all I can say is: at least they're not on DA!

I'd wish for Pixiv but they have almost the same policy, just shittier enforcement.

At this of all times, I'd put together something and try to get advertisers to pay me bitcoin to avoid payment processors altogether, but I don't know how to get artists to actually use it or how to get exposure. Do you think ad revenue sharing would work enough to get traction?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

190dfa No.10879

>>10860

>It usually reserves a few hundred MB for things like image and thumb caches.

I'd expect that. Not sure how it gets up to multiple GB of memory used over time

> I recommend clients not have more than 50k thumbnails/urls total in their main session.

The URLs are a bit surprising… do subscriptions count somehow?

> the main bottleneck on HDD clients is random access read latency

It's not merely very slow even compared to my experience value for various HDD-hosted databases. It gets stuck for hours. Maybe indefinitely now, nothing happened over 6 hours.

> Since your reset tables may be new and un-analyzed, you might like to try database->maintenance->analyze->force

No result, unfortunately.

> The best solution is to move your db to an SSD though

I'm considering it. Not terribly easy to do in this machine, but I guess it's an option if I pay enough to replace an existing SSD with a much bigger one.

OTOH again, the HDD storage isn't that slow and it can have a bunch of hundred MB of RAM to do it's job if it must- it worked on other databases with dozens to hundreds of millions of entries before. Never had them lock up weirdly like hydrus' GUI suggests its DB does; they'd generally drop down to a bunch of hundred rows per second on more complex queries that edit data inside tuples.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

fdee05 No.10880

>>10874

This has to be the one of the most retarded decisions I came across that a company could have made.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

354fcd No.10884

>>10883

Crap meant to reply to >>10856

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

da63be No.10885

File: 78aedcf50a720b5⋯.png (36.62 KB,256x280,32:35,CANTWAKEUP.png)

>>10841

>halfway to version 666

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a74b8e No.10886

>>10877

Thank you for this report. This is fixed for tomorrow.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

fdee05 No.10901

>>10874

Apparently, someone on Danbooru found a way to download raw images from Tumblr that was recently removed. I don't think this can be integrated to Hydrus, especially not before December 17 but it may be a good idea to download all the stuff manually before Tumblr nukes all the artworks, so I decided to let people know. Here's the method:

>I think it's still possible to download raw sources from Tumblr, just very inconvenient. I don't think it's intended so it might be fixed, but with the upcoming ban on adult content I figured it's now or never. If you export a blog you run (Blog Settings -> Export) then you get seemingly full size images even from reblogs; I've received 20MB PNGs and JPGs from this export. I last tested this on Nov 24 as all my blogs are 10+GB and I'm on mobile currently. If there are any Tumblr artists who don't post to a proper art sharing site like Pixiv this may be useful.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a74b8e No.10905

>>10876

Thanks. Yeah, if your db is that huge, and grew that much, I guess your session is about 500MB. I recommend you cull as many download queues as you can and put off the processing tabs with 'x to process'-type local tags so you can find those files again later.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a74b8e No.10906

>>10876

>>10878

>>10880

>>10901

I'm still laughing/in disbelief about tumblr's decision. I thought after the uproar they would swiftly reverse the decision, but I guess they don't want to subsidize free porn bandwidth any more. The only conclusion I can make is Verizon just don't have a real business plan for it. No idea why they took it on in the Yahoo purchase. What a way to kill a brand.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

006ef8 No.10917

Is their a way to search through Notes or see which images have Notes written on them?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

65a947 No.10933

>>10859

Thanks, 0.8 works better, it scrolls a full page per tick now. I have my scroll length set to 5. One page is about 5½ thumbs high.

I think this might also be related to the fact that I have Windows 10 set to 125% scale, Hydrus doesn't seem to handle that quite correctly (for example when resizing the width of the tagging panel to the left of the thumbnail view).

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a74b8e No.10937

>>10917

Not yet, but I'd like to present that information better in several ways. Adding search would also be useful.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

a74b8e No.10938

>>10933

I hope to check out high-dpi stuff in the new year. The new wx is supposed to be able to handle it better, so I'll have a look and see if I can figure out what flag I need to set or whatever. I have a 4k monitor now, so I can test this better.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.



[Return][Go to top][Catalog][Nerve Center][Random][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / / / / / ] [ dir / random / 93 / biohzrd / hkacade / hkpnd / tct / utd / uy / yebalnia ]