[ / / / / / / / / / / / / / ] [ dir / random / 93 / biohzrd / hkacade / hkpnd / tct / utd / uy / yebalnia ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.
Name
Email
Subject
REC
STOP
Comment *
File
Password (Randomized for file and post deletion; you may also set your own.)
Archive
* = required field[▶Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options

Allowed file types:jpg, jpeg, gif, png, webp,webm, mp4, mov, swf, pdf
Max filesize is16 MB.
Max image dimensions are15000 x15000.
You may upload5 per post.

This board will be deleted next Wednesday. I am moving to a General on 8chan.moe /t/. This board is archived at 8chan.moe /hydrus/!
YouTube embed. Click thumbnail to play.

066d9e No.11606

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v340/Hydrus.Network.340.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v340.tar.gz

I had a great if busy week. The Client API does more, several bugs are fixed, some new features and options are in, and stability and memory use should be a bit better.

client api

This is still somewhat advanced, but users who are interested may want to start looking in.

The first version of the Client API last week went ok! There were a couple of bugs, but thanks to feedback from some advanced users, I've improved reliability and carved out a better spec. This week brings JSON everywhere, fixes the add_file crash, and adds two neat calls:

/add_urls/get_url_files now looks up which files are attached to a URL

/add_files/add_file lets you import a file, either from a path or raw bytes

Please check the updated help here for more details:

https://hydrusnetwork.github.io/hydrus/help/client_api.html

the rest

I fixed the stupid 'page of pages' closing bug. I apologise if you were hit by this. I got a little blackpilled about all these typo errors not being caught, so I wrote a new testing suite to better test core ui functionality. I will expand this over the coming weeks and hope to completely eliminate the most blockheaded problems.

I have added nijie.info to the downloader defaults. It needs a login to access. I also updated the default danbooru parsers to get 'rating:' tags. I'd like to fold in some more downloaders to the client defaults in the near future. sfw FurAffinity should be doable next week. If you have a simple working downloader that you think would be worth rolling in for all new users, please suggest/submit it.

I added a semi-hacky checkbox to options->files and trash that pauses all new file/thumbnail requests for 15 seconds after your computer wakes from sleep. If you store your client's files over a NAS or other network solution (that may take a couple of seconds to reconnect after wake), give it a go and let me know if you like it.

I did a bunch of little work on stability and memory management and failure recovery this week. If you have had trouble with the menubar or a bloaty client or invalid website encodings or subs that fail to load, see if any of it is better here. OS X users with retina (i.e. high-dpi) screens may also see less blurry ui–I'd particularly like feedback in this case, good or bad.

full list

- client api:

- fixed up some api permissions object stuff so that /verify_access_key response can always serialise correctly

- fixed the 'add_url' api call's instability

- the API will now always return JSON on 200. anything else should be presumed to be raw text

- '/api_version' now returns JSON, and after talking with users, it will now start incrementing with every api change. it remains 1 just for this week

- '/request_access_permissions' now returns JSON

- '/add_url' now results JSON on success with more info, 403 on failure

- '/get_url_info' now returns the 'normalised_url' in the response JSON

- added '/get_url_files', which returns 'url_file_statuses', listing known hashes and file import status for that url

- added '/add_files/add_file', which can import a file from a path or bytes

- added '/add_tags/get_tag_services', which will return info on the client's tag services

- updated client api help to reflect the above changes and fleshed out the intro a bit

- fixed the client api permissions enum values in the help, which I somehow transcribed wrong first time

- updated the client api tests to check the above

- refactored client api tests to be neater and in their own file

- .

- the rest:

- fixed the page of pages close bug

- added a downloader for nijie.info to the client defaults (it needs a login)

- updated danbooru file page parsers to get 'rating' tag

- added gelbooru 0.1.11 parser for future application

- fixed an issue that was stopping advanced content updates from fully copying all the desired mappings in the transaction

- added a semi-hacky checkbox to 'options->files and trash' that will delay all new file/thumb requests for 15s after the computer resumes from sleep (useful if your files are on a NAS that takes a few seconds to reconnect on wake)

- wrote some more graceful fallback decoding handling code that attempts original assumed encoding and 'utf-8' if different and returns the one with the fewest ' ' replacement characters

- the network engine and the ffmpeg info parsing now use this new 'safe' decoding, so even if a site has borked bytes or the video file has unexpected Shift-JIS title metadata, it'll still go through, albeit with some question marks

- moved some more old daemons to the new job scheduler, deleted some old daemon code

- improved some daemon job wake and shutdown code

- wrote a proper upnp manager object and improved all-around reliability of the auto upnp-service-mapping code

- simplified the upnp check code so it now only ever checks/does anything if the respective services actually want upnp mappings. surplus mappings are now wiped immediately on service update

- fixed upnp mapping fetching to cope with ipv6 results

- improved some memory clearing code to deal with some semi-stubborn objects

- improved some 'iterate through this giant list of single numbers from the db without using a lot of memory' code and applied it to the autocomplete cache regeneration routine

- improved menubar stability, both in finding menus and swapping them out

- if a serialised json object fails to load from the db, this is now caught, the bad object deleted and written to a new file in the db dir, and all logging info captured along with an explanatory popup thrown on screen. so, if a subscription fails to load, it will now be extracted so that a subsequent subscription edit/run will work with the remaining good objects. in the case of backed-up objects (gui sessions atm), reattempting the load should restore the next most recent backup

- fixed an issue with login script validation when the given credentials have surplus ( key, value ) pairs to the script's credential definitions

- fixed two login invalid cookie error handling bugs

- maybe made some dupe filter searching more stable

- fixed a py2 datatype issue that made the client unbootable when updating the client from <296

- the client now pauses to nag and moan about backups if you try to update more than 15 versions in one go

- slightly sped up discord bugfix file drag and drops and expanded file limit up to 25 files/200MB

- added experimental secret discord bugfix dnd mode checkbox

- improved how html parsing deals with some unexpected bad tag data

- turned on primitive high-dpi support for OS X. let me know if it fixes any blurry issues on retina displays

- wrote a new 'ui test' under the debug->gui menu to help catch common-action bugs that slipped through weekly work

- improved how the test code does some wx/ui stuff, but also broke some more and ran out of time to clean it up–this is an ongoing project

- improved how some text import line splitting works

- misc fixes

next week

I pushed it too hard this week, and while I am overall happy with the work, I am going to return to normal schedule so I don't burn myself out. I'd like to keep pushing on the Client API, probably getting add_tags done, and otherwise just flesh out the new test code and do some small QoL stuff.

____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

60c4ac No.11608

File: 4b9c89079f84560⋯.png (6.96 KB,626x54,313:27,ClipboardImage.png)

>>11606

I appreciate the work, but since 339 and now 340 Hydrus has taken a nose dive in terms of stability. It's back with another memory leak after lockup after lockup afer lock up with idling just like a few versions back.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

44bc16 No.11611

>>11606

hey, I'm the guy who was trying to move my dbs to different locations and load one from an external drive. I got it working, but for some reason with the external drive I had to remove the quotes around the db path for hydrus to open, but I can either use or not use the quotes for the db on my local drive. weird.

I've run into another problem though. I can't load webms on either db, I get this error:

FileNotFoundError

[Errno 2] No such file or directory: 'ffmpeg': 'ffmpeg'

Traceback (most recent call last):

File "Hydrus/include/HydrusThreading.py", line 342, in run

File "Hydrus/include/ClientRendering.py", line 314, in THREADRender

File "Hydrus/include/HydrusVideoHandling.py", line 626, in init

File "Hydrus/include/HydrusVideoHandling.py", line 694, in initialize

File "subprocess.py", line 775, in init

File "subprocess.py", line 1522, in _execute_child

FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg': 'ffmpeg'

And when I try to download webms they get ignored and I'm told they are an unknown file type. Advice? I'm using v340

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

ad3f02 No.11612

>>11606

Based and blackpilled

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

97d621 No.11613

File: 75d2a340d58759c⋯.png (148.4 KB,974x490,487:245,ClipboardImage.png)

>you can copy the hashes of multiple files

That's pretty nifty, since when was that in?

Time to fucking crash my hydrus.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

97d621 No.11614

>>11613

>6 minutes including sorting the hashes

I expected it to fuck up more.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

46aebf No.11617

File: 67933065229ed8f⋯.png (463.97 KB,1318x699,1318:699,chrome_2019-02-14_15-34-21.png)

>>11613

able to share these images?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

97d621 No.11621

File: 35d78d2c7dc6ca0⋯.png (3.32 KB,260x500,13:25,35d78d2c7dc6ca062ee3ddd0e9….png)

File: bc33a1bcf606d62⋯.png (3.37 KB,260x500,13:25,bc33a1bcf606d62da2737cc905….png)

File: bfeba95cd34fad8⋯.png (2.92 KB,260x500,13:25,bfeba95cd34fad89744e1a220e….png)

>>11617

>asking for the untagged pics as well

Anon, why?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

97d621 No.11622

File: 3262173a27a509d⋯.png (1.69 KB,80x128,5:8,3262173a27a509dc4409fd3745….png)

File: 5aa695bc5440dfd⋯.png (1.72 KB,120x244,30:61,5aa695bc5440dfd97dc0b985d9….png)

File: 504386cbbe78fb6⋯.gif (3.44 KB,65x213,65:213,504386cbbe78fb68b6745de088….gif)

>>11617

>>11621

Surprisingly easy due to the filesize, I thought it'd be here for longer.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

97d621 No.11623

File: debcbd7f100eae8⋯.gif (Spoiler Image,24.58 KB,320x160,2:1,debcbd7f100eae870df90db22b….gif)

File: 6a28262e29dedf2⋯.gif (Spoiler Image,25.28 KB,640x320,2:1,6a28262e29dedf2ed989706ef0….gif)

File: 1adb01e89e0f031⋯.png (Spoiler Image,29.33 KB,640x320,2:1,1adb01e89e0f031a0442ae1e15….png)

File: 0398bdb3f36751b⋯.png (Spoiler Image,31.12 KB,360x160,9:4,0398bdb3f36751b5a1056b08c5….png)

File: 631350375d5484f⋯.gif (Spoiler Image,42.52 KB,544x296,68:37,631350375d5484f2ec119a9b77….gif)

>>11617

>>11621

Also, the first three are from a game by fuugetsuin, he does some great sprite porn on those.

There's a thread over at /hgg/ up with all the games in a mega.

>>>/hgg/8386

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

97d621 No.11628

File: 5d4a20f88e08c45⋯.gif (1.45 KB,41x81,41:81,5d4a20f88e08c450c3f72aff8c….gif)

forgot to post the viv

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

44bc16 No.11630

>>11611

btw I also get this error when I try to import pngs, but I can still view pngs already in my db unlike my webms. And again, I am on osx if that's useful to know at all.

FileNotFoundError

[Errno 2] No such file or directory: 'ffmpeg': 'ffmpeg'

Traceback (most recent call last):

File "Hydrus/include/HydrusThreading.py", line 342, in run

File "Hydrus/include/ClientGUIDialogs.py", line 1021, in THREADParseImportablePaths

File "Hydrus/include/HydrusFileHandling.py", line 315, in GetMime

File "Hydrus/include/HydrusVideoHandling.py", line 309, in HasVideoStream

File "Hydrus/include/HydrusVideoHandling.py", line 116, in GetFFMPEGInfoLines

File "subprocess.py", line 775, in init

File "subprocess.py", line 1522, in _execute_child

FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg': 'ffmpeg'

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

60c4ac No.11632

>>11608

I think i've cleared up the constant crashes and lockups, havent had any for a couple days now after disabling idle processing in the background and also pausing the PTR sync. not sure which is the culprit though.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

ed45d3 No.11635

Can resized and/or full sized thumbnails be disabled in Hydrus? I'm rather convinced I don't need both (maybe not even either), and they store pretty inefficiently on disk.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

ec7425 No.11636

The new asynch tag lookup is fantastic. However, sometimes it is very slow. I think maybe it is looking up the results of some tag I wrote but I have already started writing the next tag and waiting for the results on that one. Can this be improved? In some cases there are significantly longer wait times than before.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

066d9e No.11638

>>11632

>>11608

Thank you for this report. I am sorry you are having trouble.

Can you describe your client situation a little more? Do you have a lot of downloaders or other importers open, typically? How many total thumbnails (+pending urls in downloaders) are open in your client in a normal session? How many total files do you have in your client, and what is the rough total size of the four client*.db files in your install_dir/db folder?

Are you regularly opening new pages or media viewers? What sorts of media do you look at?

When you first boot the client, what's its memory usage? And when it bloats up a bit, if you run help->debug->data actions->run slow memory maintenance, does that kick it down a significant amount or not at all? How about 'clear image rendering cache' on the same menu?

I believe there is memory explosion during PTR processing, although it _should_ settle back down within 5 mins of processing finishing. I am going to keep working on it.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

066d9e No.11639

>>11630

>>11611

Thank you for this report. I think you have the same as >>11554 . Can you do the same as my post here: >>11594 ? Try the new subprocess report mode and let me know how it goes.

The png thing btw is it now needs ffmpeg to figure out apngs from pngs.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

066d9e No.11640

>>11613

A little while, though it was hidden by advanced mode for a little while, I think. Maybe it still is now.

I'll see if I can wrap that into a larger single 'fetch hashes for these files' call, rather than a million 'fetch hash for this file' ones. It should speed that mass-lookup up.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

066d9e No.11641

>>11635

Not now. My rough tests a while ago suggested resized thumbs are usually about 1.2% of total file size and fullsize are 1.6%. This bloats a little of course with 4KB hard disk sector size and on MFT. I am generally ok with this, although I've wanted to eliminate resized thumbs for a while as I am not happy with two copies.

Would you propose creating (and caching in memory) thumbnails from source files every time you loaded them? I feel this would be very slow, maybe 0.2s+ per file and much more for high res images and videos. Maybe I am thinking about it wrong. Even if you were willing to swallow that delay, I am not sure how simple it would be to change my thumbnail and file storage workflow to handle it. Maybe it would be doable, but I'd need more data first that it was actually feasible.

You know what–thinking about this, I will put a bit of time this week to seeing if dynamic thumbnail resizing from master is doable. If I can load the fullsize thumb and scale it in a reasonable real time, that will eliminate half of this problem.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

066d9e No.11643

>>11636

Yeah, the new async system is still running on the old pessimistic 'it was xms since you typed in >=y characters' timings under options->speed and memory. I think it would be appropriate to rewrite the rules of when to fetch results now we know they don't block. If you haven't played with those ms timings, please do and let me know what you think would be good new defaults and if you think there should be fundamental changes to how it works.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

ed45d3 No.11663

File: f7ff7f62f4f5d2c⋯.jpg (44.93 KB,500x750,2:3,f7ff7f62f4f5d2c6bca3b8f6bb….jpg)

>>11641

>This bloats a little of course with 4KB hard disk sector size and on MFT

On f2fs and other filesystems, it bloats roughly by a factor of 2.4 vs the thumbnail's own size.

> I feel this would be very slow, maybe 0.2s+ per file and much more for high res images and videos.

Here, Hydrus isn't actually very appreciably faster for images than an image viewer or file manager without cached thumbnails, even on HDD.

On videos, cached thumbnails are noticeably faster, but actually I never felt like I really needed the cache for these either - they still generate fast enough overall. I'd possibly still disable the cache for these or just generate small thumbnails if I could.

> Would you propose creating (and caching in memory) thumbnails from source files every time you loaded them?

Yes. I'd propose it simply does that if thumbnails are disabled by the user. I think it will work okay.

If larger thumbnails are enabled, maybe allow disabling the small thumbnails and generate the downsized versions from the larger thumbnails? This should be speedy even if you don't use something as optimized for this use-case as the flif format (which can be "be partially decoded to yield a lower resolution image" directly).

> You know what–thinking about this, I will put a bit of time this week to seeing if dynamic thumbnail resizing from master is doable.

Thank you very much for that!

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

d658f0 No.11665

>>11594

>>11639

I tried adding ffmpeg to the bin folder but hydrus still produced the same error. I didn't have time to install 340 yet, so that's still 339.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

46aebf No.11669

>>11628

>>11623

>>11622

>>11621

bit late in saying it but thanks

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

ab4bb7 No.11685

>>11608

>>11632

Not sure if this is related since I'm on 337 but I too would have random instances of memory leaks slowing down either hydrus or my PC and the client locking up. Its been happening ever since I've been on 337 I believe but it sounds like its still been happening in the later updates. I haven't been using PTR though.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

46aebf No.11687

>>11606

Hey hdev, just had an idea

so im over on exhentai, looking at the new archives and finding artists, come across one that I have downloaded before, but they updated.

could there be a single button way to relook up artists in the gallery downloader

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

46aebf No.11688

>>11606

1319954

found that pivix aritist, downlaoded from them before, but went to update by re importing them, have 307 new files, but only 21 display

have 2033 files from them, but only 1731 will show

it looks like some of the links are showing files, but not downloading them, I think this may be an issue, weather its a the program isn't downloading them or not aside the its registering the links as successful downloads and labeling them files while not having them is the problem, a future update may grab the files, but will it if it thinks they are already downloaded?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

066d9e No.11689

>>11663

Thanks, this is interesting info. Due to me being ill last week I am a bit crushed up with todo atm, but I have work here as a priority. I'll see about adding more ways of loading thumbs and let you turn them on as experimental options over the coming weeks to see how they perform. Please give them a go and let me know how it works out.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

60c4ac No.11699

>>11638

So it's been a week and nothing's changed, still no crashes and I havent gotten the wx menu bug since making the changes I stated. I started to use page of pages as well but havent notice any difference in stability. Next week I'll try turning one of the options back on and see if anything goes to hell.

I typically leave downloaders open with them running or not. Since the switch to combine them into the list with only one displaying at a time was made a while ago. I normally download directly from boorus, three of the most popular I use as danbooru/gelbooru, e621 and derpibooru. Some of these don't come with default tag scraping, so I went and enabled that a while ago. This is why my pending tags can get huge, more rarely but still on occasion I scrape from websites directly like an old 4chan archive, sometimes deviantart's gallery or twitter.

Gallery Downloaders, usually downloading with no-limit option ticked I download the full artist's gallery then sort after the fact. If a tag has something that's under 1k total files (checked on a booru first) then i'll download that one I search for too.

Thumbnails for each page can range from 10 to 1000+, usually hovering around a couple hundred or so since it's entire artist gallery. Currently, I added them up and have about 2k thumbnails that are loaded with no issue. Unless there's a big problem I usually just tell Hydrus to load last session every single time on a crash. Then I might go in and close some of the pages to reduce the strain. I usually just use downloaders instead of subscriptions or watchers.

I do regularly open and close the mediaviewer all the time, flipping through or bringing up the right click context menu, sometimes to open it in irfanview for an easier zoom. The media I look at is jpg/pngs mainly but occasionally scraping video files and zips from 8chan or yiffparty respectively. Normal RAM usage is 300-400MB on loading it up. I havent tried the slow mem maintenence but maybe I will next time. Also, havent tried the clear image rendering cache either but I'll keep it in mind for next week when I turn things back on.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

60c4ac No.11700

File: 444ea0096b22dec⋯.png (131.88 KB,607x841,607:841,ClipboardImage.png)

>>11638

Cont. since Hydrus is done vacuming

The total number of files I have is abour 380k, 90% of them are still in the 'inbox' but I dont see much use in the 'archive' for my daily use. I just use the 'trash' instead to get rid of everything I dont want and leave the rest as 'inbox'.

I believe pic is what you're asking for for the 4 client files? Let's just call it roughly 15 GB all together.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

066d9e No.11702

>>11685

Thanks. Should be a bit better in new versions, and moreso in upcoming 341. Please keep me updated.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

066d9e No.11703

>>11687

Does this fit into the subscription system, or is it a different kind of workflow? I am not very familiar with exhentai.

If you haven't tried it before, subscriptions are a great way to keep up with artists:

https://hydrusnetwork.github.io/hydrus/help/getting_started_subscriptions.html

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

066d9e No.11704

File: 4e55e5c587407d0⋯.png (27.1 KB,1135x188,1135:188,p.png)

>>11688

Hey, Pixiv is a slightly odd site, so there are two caveats here:

1) Ugoiras are not supported, so they will most likely get 'ignored' status.

2) Any 'manga' page will branch out to create new file import entries in the queue, leaving a stub entry.

Pic related has an example of 2. It is an unfortunate artifact of how Pixiv have decided that some 'mode=medium' URLs refer to one file while others refer to multiple. Although the initial https://www.pixiv.net/member_illust.php?illust_id=73135351&mode=medium url here gets 'successful', meaning it was a queue item processed successfully, it doesn't have any files itself.

Could the manga surplus explain your bad file counts here? It sounds maybe reasonable for the 2033 vs 1731, but not for the 307 to 21. Can you tell me more about what that 307 means? Where did you see it, and where did the 21 come from?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

066d9e No.11705

>>11699

>>11700

Thank you. Your situation does not sound crazy–2k thumbs is fine, for example. I have now done more for this for 341. In particular the db will now spool temp data to disk if it gets too big. I expect many users to see significantly less memory use during PTR work.

Please keep me updated here.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

46aebf No.11711

File: 4722a45faf3ad08⋯.png (45.6 KB,1081x532,1081:532,client_2019-02-25_16-07-41.png)

>>11704

Its possible that its manga and ugoria, but the problem I have is that it says there are 307 new images, but when I tell it to present new, only 21 show up.

my concern is that it's filing things that may, in the future, be parseable under an I have it, and when it rechecks it thinks I have it and skips over it.

>>11703

for this, sorry, im not getting them from exhentai, im finding new artists, usually in a gallery for an artist they will post where you can find their work directly, and I plug it in to download.

effectively, I download a gallery of something in the gallery downloader, and a few weeks later I see an update to it, so I regrab it in the gallery, the best way to put it is these are impulse downloads, not something I want to set a subscription up for. I think this shows off what I would like to avoid fairly well, if there was a way to recheck without adding the same search again, that would be great.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

066d9e No.11712

>>11663

>>11636

I did some work on this today and it went well! I have a 'thumbnail experiment mode' in for v341 under help->debug->gui so you can try it out yourself. It loads the fullsize and resizes in memory. There is not a human-noticeable difference in speed. On my dev machine here, which is a few years old, I added some timers and got (once the file data was in memory) approx 500µs to load a resized thumb and 1.8ms to load a fullsize thumb and scale it, which I was impressed by. I have a Nvidia 1030 here to drive my 4k monitor, so perhaps that is accelerating this.

I am willing to experiment more here, so I will mention it in my release post and see what you and other users find.

After looking at the code, I think that in exchange for simplifying the file system by only having one set of thumbs, I could justify making that single thumb more intelligent in how it swap outs bad-size thumbs when needed (i.e. basically removing the master rather than the resized). So, I think we can get the best of both worlds here, saving space and keeping things fast. I have a job set out for it now, so I'll try to chip away at this in the coming weeks. Thank you for bringing this up.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

60c4ac No.11713

>>11705

Last bit of info is I currently have about 100-110 gallery download ques (none running), that would send the pending tags through the roof but since Hydrus only displays 1 at at time then I dont think this would do much for thumbnails displayed. It would certainly push up high use of the CPU, disk writing and some ram but only when Hydrus is busy. Despite all the stuff I listed I'm only around 450 MB of Ram.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.


[Return][Go to top][Catalog][Nerve Center][Random][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / / / / / ] [ dir / random / 93 / biohzrd / hkacade / hkpnd / tct / utd / uy / yebalnia ]