/hydrus/ - Hydrus Network

>>11962

>I wrote this specification of the XCF format in mid-2006. Later it got included in the official Gimp sources, and unless you're explicitly looking for historic data, you should go there and get the specification instead.

Try here instead

https://gitlab.gnome.org/GNOME/gimp/raw/master/devel-docs/xcf.txt

>>11962

https://docs.krita.org/en/general_concepts/file_formats/file_kra.html

Hmm

>>11962

ahhh… https://docs.krita.org/en/general_concepts/file_formats/file_ora.html

>>11969

>>11970

>>11972

Thanks lads. I'll queue these up and I'll see what I can figure out. Adding 'I can recognise it at least' .psd support wasn't a massive hassle this week, so I expect to do more of this sort of work. But I suspect I'll have to refigure some UI like the system:mime panel checkbox hell if we go too nuts here.

>>11947

Yeah, encryption would be pretty important. I personally don't care because lol openvpn server at home, but that's not normal.

>>11955

The numbers there are the 'status' of that list of tags. In hydrus, a file's tags can have four different statuses:

0 - current - the file has the tag for this service

1 - pending - this tag is pending to the service

2 - deleted - this tag was current but has now been deleted

3 - petitioned - this tag is current and is petitioned for deletion

Pending/petitioned relates to the 'pending' menu in the main hydrus client gui. Repository tags are not fixed as current/deleted until pending/petitioned are commited through that menu.

In writing this out, I realise I fucked up here because a 'local tags' service won't ever have 'pending' tags. I will rewrite this help section to have it refer to a hypothetical tag repository.

>>11964

To follow up here, if you wish to edit a file's tags with /add_tags/add_tags, you will need to use this more detailed tag status information to figure out if you are 'petitioning' or 'rescind pending' or whatever.

For practical purposes, if you want to calculate what tags the file currently 'has' for easy phone display, you can go:

tags = ( current ∪ pending ) - petitioned

Petitioned tags are pretty rare, so you can probably just go ( current ∪ pending ). You'll then be unioning those sets across all given service_names to get a file's 'all known tags' as you would see in the client.

>>11964

>>11965

Ah. I forgot tags can have statuses. Thanks for clarifying

>>11838

A user helpfully pointed me to some here:

http://www.brucelindbloom.com/index.html?RGB16Million.html

They render ok, so I guess a clever sRGB conversion is happening. I also added support for 'MM' tiffs for tomorrow. Should be complete tiff support.

>>11840

>>11841

If you have a lot of processing still to do, I recommend you do not do explicit processing from that review services frame. It can only be cancelled through the UI, and after a long time, there is a decent chance the UI will deadlock until the job is done (this is due to some shit code by me that will take a bit to clear up). If the job still has 15 hours of work left, the whole program can hang that long.

I recommend you only let processing happen during shutdown, where it has a timer, and idle time, where moving the mouse will tell it to safely cancel.

That review services button is hidden behind advanced mode and is only ever really a pleasant experience when I am doing some testing or when on a fast SSD without much total processing left to do.

>>11842

Yeah, for some complicated websites, an import 'item' produces more than one file or a new page to pursue. It is a technical pain in the ass, but for now, the x/y progress on an importer refers to 'import items processed' not 'potential files imported'.

If you would like to check those 'already in db' files, you should be able to right-click on them and say something like 'open these in a new page'. Since in that note it says '10 seconds ago', they are almost certainly duplicates from above. (I don't know anything about nijie.info, but yeah, it looks like the _0 url without diff/main is a 'cover image' for the subsequent mini-gallery?)

Again, some galleries give the same file twice, even on their nice APIs. I don't know why this parser pulls that 'cover', but I did not write it, so I can't confidently say this strategy isn't needed to catch edge cases. The nuts and bolts of this stuff are frequently fuckery duckery doo, particularly on Japanese sites.

>>11844

Thanks, that is interesting. I assume the lossless compression mode is functionally 4:4:4, right? But any lossy encoding is coerced to 4:2:0?

My hesitance for webp in the past is that it still is sRGB, so I don't know how it is useful as we move to HDR in the coming years. Maybe they will do like webm and add new accepted encoders, but I dunno. It doesn't seem to be taking the world of internet browsers and phones by storm as it is. HEIF and FLIF seem to have their good points, but they are still similarly meme status for now.

I'll play with animated webps a bit when I find time, as I'd really prefer not to use shitty gifs for animated thumbs, and I don't want to go completely nuts with apngs either.

>>11846

Thanks, I am glad you like it. When you get an app ready and want to share it about, please send me a link to it and I'll put it up in my release posts and help!

Current plan for Client API is to get a simple 1.0 out the door. This should be done tomorrow for 243, where basic file search and file/thumbnail retrieval are finished. You'll be able to find all the files in a client with the tags 'blue eyes' and 'blonde hair' and then shows thumbs and files and tags. It should be possible to replicate very basic booru functionality. After the release tomorrow, please check the updated help here:

https://hydrusnetwork.github.io/hydrus/help/client_api.html

Which will have the last '/get_files/…' queries all filled out.

With the 1.0 done, Client API work will then be fit into the regular weekly 'small jobs' schedule. If someone wants the ability to archive files or search by duration, I'll see if I can squeeze it into a small job. If you have ideas for your app, please submit them any way that is convenient–these release threads are fine, and you can also email me or even DM me on discord on Saturday afternoons US time if you want to talk live.

>>11783

Thanks. I see one of those definitions is unhappy with UPnP. Feel free to delete the three upnpc executables from install_dir/bin–they aren't needed to run the program, only if you want to explictly tell some hydrus service to keep a NAT port open for itself. They are just copies of the exes available here:

http://miniupnp.free.fr/

The others, I suspect, are as I said before. I haven't had a true positive here yet, but I'd appreciate any future information you have here.

>>11785

Thank you, I really appreciate this recommendation and your thoughts about it. I will definitely revisit this when work on hydrus multi-page support starts.

>>11789

Thank you for this report. My dark mode is a bit of a fudge–I only have control over the colour of my custom objects, whereas everything else (like window borders and backgrounds and button colours) is supposed to be inherited from your OS's current system colours. If you have a window manager that changes your OS colours, hydrus's dark mode switching can be helpful to make sure that my custom objects also change with it.

There's a very speculative long-term plan to move from wxPython to Qt. If this ever happens, theming support will get significantly better, but until then it is all duct-tape, I am afraid.

>>11793

Thank you for this report. Are the updates processing during idle time or shutdown? If you have problems during idle time, can you change your settings under options->maintenance and processing to be shutdown-only, and, say, only 5 minutes? If that relatively small amount of work can go through, can a bit more, like 15 mins? Is there a significantly 'cleanup' lag, like 5 mins of work takes 3 mins to commit?

>>11810

Thank you. That is very strange. Here's SQLite's and Python's info on this:

https://sqlite.org/tempfiles.html#temporary_file_storage_locations

https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir

So, for some reason, a different folder was not permitting a program launched from it to access any system temp location. Windows has simpler temp structure than Linux, so perhaps this really just meant stopping you from accessing the one under AppData/Local/Temp.

Could somehow that directory (or say the client.exe) have got a compatibility mode or some other protected state applied to it? I am afraid this is way beyond my expertise.

Sauce, devbro?

>>11801

Dunno, sorry! It was just in my inbox this week.

But perhaps source could be your life, one day.

>(including I think the Linux >0 distance similar files search crash, finally!)

Was that at all related to any of my posts:

>>11736

>>11758

>>11797

Because I was having an issue on linux crashing. In the process for searching for duplicates an SQL statement similar to

SELECT phash_id, phash, radius, inner_id, outer_id FROM shape_perceptual_hashes NATURAL JOIN shape_vptree WHERE phash_id IN (1, 2, 3);

I imagine based on the code I looked at having a search distance greater than 0 would make it far less likely to happen if not impossible.

>>11804

There are quite a few places where the client fetches things with lists up to 256 items long, so if you found that just loading some files and tags was ok, I _think_ you weren't getting hit by that exact issue.

But yeah, your posts are the bug I think I fixed. Only affected some Linux users, and I could not reproduce it. I worked one on one in the week with a user who had the crash and we ultimately figured it was a problem with SQLite hanging on to the phash while I did a Hamming Distance search on it (which requires some low level C++ struct calls and then some mickey-mouse CPU-level binary counting). Something to do with memory access violation is my best guess. Python usually can't crash by itself. Previously, I was doing hamming searches interleaved with the sqlite call iterating each row, but when I separated it into completed batches, so sqlite was 'done' with the rows, no crash happened.

The 0 search distance could do a direct lookup without the hamming check, and that was running fine. It just seemed to be the combination of iterator on sql results and then low-level access on 'bytes' type response from that that caused the crash. Core dumps suggested the crash was in sqlite code, so I guess it was still hanging on to a bytes buffer or something that wasn't cleaned up. Could indeed still be a (different) bug in sqlite, or python's wrapper for it, but I dunno for real.

Let me know how 342 works for you!

>>11801

Owarimonogatari

>>11803

wholesome

https://kanban-1.herokuapp.com/boards/9

>>11721

Trying to find a kanban for people to put their ideas on the board.

>>11718

the program tends to eat memory when you heavily use it, let me give you a picture of what I mean

I save every thread on 4chan that catches my interest, at some point the reposts start to overtake the new images to a point that this is a viable option, thy im getting close to the edge of I need some delete functionality for telling me what the file was deleted for. that aside, right now, over the coarse of 2 months I have created around 6400 watchers, adding somewhere around 150-200k images, almost none of them are displayed, with around 100k displayed.

the 100k displayed images takes about 1.5-2gb, I know this from the time I culled the watchers, my current session is 4.3gb and bloats out to 6.5gb on saves.

I have absolutely no idea what takes ram, if they were all current watchers, ok, I would believe that, but from experience current watchers for all the threads adds up to around 500mb or less, so for some reason, the program is using an ungodly amount of ram for reasons that are unclear because just storing all of the the data in text form could not possibly add up to over 20mb much less what I estimate at being over 2gb with a doubling in save.

I should also note that I come from a session that was hitting 14gb on startup and when a save happened would bloat to 18+gb

depending on where hdev is focusing, memory reduction has many different ways to happen.

>>11724

>the program tends to eat memory when you heavily use it

I think that goes for most programs.

When I had 8GB of RAM my average overall usage was around 3GB, but now that I have 16GB it averages 6 or 7 even though I'm doing the same things.

>>11718

Bare client is about 115MB ram on Windows, so the absolute minimum is pretty low compared to what many users see. I am about 490MB on my IRL laptop that has about 17k items in its main session (mostly thread watchers).

The python 3 change lead to some pseudo-memory-leaks, so that was a recent chunk I was able to recover, and I am making an important db-memory change in 341 today.

I would appreciate continued feedback here. There isn't a huge amount I can do atm about clients that have hundreds of thousands of pending URLs in their session, but PTR update processing and some other db stuff should be much less traumatic now. There's still more to do, and I still plan to make a memory profiler this year to draw some pie charts or something and squeeze this further.

>>11699

>>11700

Thank you. Your situation does not sound crazy–2k thumbs is fine, for example. I have now done more for this for 341. In particular the db will now spool temp data to disk if it gets too big. I expect many users to see significantly less memory use during PTR work.

Please keep me updated here.

>>11663

>>11636

I did some work on this today and it went well! I have a 'thumbnail experiment mode' in for v341 under help->debug->gui so you can try it out yourself. It loads the fullsize and resizes in memory. There is not a human-noticeable difference in speed. On my dev machine here, which is a few years old, I added some timers and got (once the file data was in memory) approx 500µs to load a resized thumb and 1.8ms to load a fullsize thumb and scale it, which I was impressed by. I have a Nvidia 1030 here to drive my 4k monitor, so perhaps that is accelerating this.

I am willing to experiment more here, so I will mention it in my release post and see what you and other users find.

After looking at the code, I think that in exchange for simplifying the file system by only having one set of thumbs, I could justify making that single thumb more intelligent in how it swap outs bad-size thumbs when needed (i.e. basically removing the master rather than the resized). So, I think we can get the best of both worlds here, saving space and keeping things fast. I have a job set out for it now, so I'll try to chip away at this in the coming weeks. Thank you for bringing this up.

>>11705

Last bit of info is I currently have about 100-110 gallery download ques (none running), that would send the pending tags through the roof but since Hydrus only displays 1 at at time then I dont think this would do much for thumbnails displayed. It would certainly push up high use of the CPU, disk writing and some ram but only when Hydrus is busy. Despite all the stuff I listed I'm only around 450 MB of Ram.

>>11676

Take care of yourself and get well soon, hydrus dev.

>>11683

Is this when she dies? I've never seen this in game

>>11455

Ask for a parser

I was able to script about 70% of it but ran into an issue. The site has one of those fucking click through screens. other than that it is practically identical to e621. Just with worse tagging.

>>11470

>The site has one of those fucking click through screens.

You mean, the confirm age one?

That may be solved by connecting to the "yes, I am 18" URL directly.

>>11521

it blocks that. you need A verified cookie.

>>11596

After a bit more poking around, I think regular garbage collection wasn't running for some reason. I'll do some more work on this today, it may be related to another issue.

>>11587

Thank you for this report. Just to check, this is non-Windows, right? Did you happen to migrate an install from Windows? I have had the worst luck trying to get the floating version of the autocomplete to work on non-Windows for these sorts of focus issues, so Linux and OS X installs start with it always in embedded mode, like how the 'manage tags dialog' works.

Please hit options->gui->always embed autocomplete dropdown results window and let me know if that works better.

>>11599

Oh, and to add to your question: I indeed have migrated from Windows long time ago. I just tried a fresh install and it defaults to that setting, so the problem was just outdated defaults. If I wasn't pantsu-on-head retarded, I would've checked if it was working on a fresh install first. Sorry for wasting time.

>>11603

I reinstalled 339 and havent noticed the memory spike as bad as before but it is locking up and stops responding pretty often so i still end up having to restart anyways for any downloading. the program still works for main archival and search though.

Read >>10203 then write up a proper proposal.

>>11462

Thank you, this vid is very helpful. It looks like you have a pretty heavy client, with a lot of pages open and a ton of pending tags. Do you have, say, more than 100 pages open? I am wondering if your client is running out of ids for menus quicker because it has all the other stuff going on. Is it possible you can temporarily close some of those pages, restart the client, and see if that eases things up a bit?

Have you tried 'page of pages', btw? This helps nest pages Hit F9 (i.e. choose a new page) and then special->page of pages. You can drag and drop page tabs onto that new page 'folder' and you won't be navigating left and right across a long ribbon of tabs.

>>11510

>>11517

Thank you for this report. I apologise, I fucked up a change here. I will have it fixed for 339.

Hi, having a small problem with the Manage Import Folders option.

TypeError
CallBlockingToWX() missing 1 required positional argument: 'func'
Traceback (most recent call last):
  File "include\HydrusThreading.py", line 342, in run
    callable( *args, **kwargs )
  File "include\ClientGUI.py", line 2608, in THREAD_do_it
    controller.CallBlockingToWX( wx_do_it )
TypeError: CallBlockingToWX() missing 1 required positional argument: 'func'

Not a problem with export folders.

>>11532

Thank you for this report. I made a couple of stupid mistakes last week–this was one. I am sorry for the problem. It will be fixed in tomorrow's release.

This thread is getting laggy, so:

N E W

E

W

T H R E A D

H

R

E

A

D

>>11542

>>11542

>>11542

>>11542

>>11542