/hydrus/ - Hydrus Network

It looks like 'namespace:*' searches, despite now working, may be running super slow or doing too much database work, in IRL PTR-syncing situations. I will look into it next week! If you are hit by this, please turn them off for now under tags->manage tag display and search.

>>15014

Two issues I've noticed network-wise so far.

Issue 1 Disclaimer: no idea if this is a regression - first time I've tried this on any version.

On v421 I'm running into an error when querying for IDs with specific filetypes through the API.

I'm trying to get a list of all files of type webm, wav, etc. but I can't figure out the proper way to do that in the "get_files/search_files" endpoint. Stuff like "meta:webm" works in both the client and API but it's missing a lot of files and seems to be manually assigned; on the other hand, using the query "system: filetype is foo" in the client seems to pull directly from the local DB of files and doesn't miss any but when I hit the API with "?tags=%5B%22system%3A+filetype+is+webm%22%5D" I get no results.

I'm not sure if this is user error, by design/wontfix, or you just haven't gotten around to handling/exposing it yet. Is there currently a way to filter on the filetype over all local files? If not, could you add that to the suggestions you're polling on?

Issue 2 Disclaimer: I have a janky workaround for this

I'm working on a browser-based frontend for Hydrus, and one of the problems I've hit is that safari for iOS (and maybe Mac too, I don't have one to test) won't download media like mp4 files because it tries to sample the file with a range download, and if the remote server doesn't support byte ranges then it bails and the file doesn't load. My current workaround is an nginx reverse proxy with the proxy_force_ranges option set, but it's kind of janky and I can't really ask people to run an nginx server to use the webapp if I ever get it to a releasable state. I'm not sure what library you're using to serve the API but if there's an easy way to enable byte ranges that would be great, and if not no worries, it'll be quite a while before it'll matter.

I had a good week making some small fixes and improvements to finish up the year. Autocomplete works a bit faster, and some quality of life is improved.

The release should be as normal tomorrow. It will have the 'next big job' poll and be the last release for the year.

>>15016

Hey, unfortunately system predicates do not work on the Client API yet. They aren't text-based in the actual client, so I can't just pull text in the query to accept them (yet). They are top of the list to add for Client API features.

Here's the master job for Client API by the way: https://github.com/hydrusnetwork/hydrus/issues/656

Thank you for the note about file ranges. I have not run into this need yet, so I will research how to do it properly. I use twisted as my main server engine, so I know this is possible, I just think I have to link some things together.

>>14950

Thank you, I saw your post on Endchan, I'll paste my response there here:

Thank you, this is very useful information. I am not sure if I have the ability to set these opcodes from my python layer, but I will investigate this. If I can set a permanent WAL of a certain healthy size, it sounds like that would be a great improvement for HDD users. Can you say more how you achieved this to get 20k rows/s? Did you use a custom sqlite.dll, or were you able to set an environment variable to change my code to do this?

Another potential option here, although it is a bit crazy, and probably stupid if the user is on an older system with an HDD, may be to tell SQLite to use a memory journal, or memory temp file. I wonder if there is a similar problem as the fraggy WAL with fraggy temp files (SQLite creates files to store temp table data when it does larger transactions).

I have discovered that some PTR syncing is creating absolutely gigantic temporary files, sometimes >20GB. I want to investigate shrinking this significantly, although I worry that making more smaller commits will just result in more HDD work overall.

Also ultimately, I will be adding filters to the PTR in the coming months so users can just sync with a subset of PTR data (e.g. just creator/series/character tags), which I hope to produce several speed and storage improvements for users on slower drives.

Did the twitter (nitter) media lookup importer start failing for anyone else? I think it's a problem with the source site changing settings or something, since it happened while using the same version.

I had a great week. I fixed bugs in drag-select, an undisplayable tag, some autocomplete result weirdness, and logical problems that were causing some grandparents and siblings not to appear in the new virtual system; I fleshed out some maintenance modes and added better autocomplete lookups in manage tags; and I sped up several autocomplete, sibling, parent, and general tag routines.

The release should be as normal tomorrow, maybe a little late.

>>14956

Sorry, I don't know, I'll check it out.

>hydrusdev fell into the rabbit hole

Did this update touch wildcards at all? copying a namespace:*anything* query only copies the namespace again when you fixed this previously, and 'namespace:*' doesnt bring up autocompletion results like I was expecting

>>14994

That's the hope, at some point. I want to unify the local and remote file import metadata pipelines more, not only to add features but also to reduce maintenance and expansion work. My dream is that you'll one day be able to apply parsing tech to sidecar files like neighbouring .txt files and just pull whatever the hell data you want out of json or xml files.

I do not know when notes will be able to do this, but first steps are to get .txt working for import/export folders and have the filename->tagging dialog work with the new string converter system instead of my ancient hardcoded regex input fields.

>>14995

I am sorry, 'namespace:*' and friends seem to have been broken significantly as of about v419 when I did some 'optimisation' work to fetch regular results faster. I have fixed it all today and added specific unit tests so these advanced searches can't be lost due to typo or ignorance again. Please let me know how 422 works for you.

Can you say what you want copied when you copy a 'namespace:*anything*'? I am sorry that I have forgotten about this issue.

>>15002

No problem; when copying a tag like "character:*anything*" I was expecting to have a "character:*" result I could copy and paste in a different search

>>15004

Thanks, that makes more sense. I'll change it to that and make a note not to change it.

I had a great week. I mostly fixed bugs, including the recently broken 'namespace:*' autocomplete lookups, extended database recovery code, improved some quality of life, and added some advanced tools for downloader makers.

The release should be as normal tomorrow.

I downloaded the zip file and I scanned it. Scan pulled up PUA:Win32/Presenoker threat.

>>14981

>what is a false positive

shiggy diggy

>>14965

Great, I am glad things are working quicker.

My new db mode would have sped up users with their database files on an HDD. Unfortunately, due to some SQLite peculiarity, it wasn't as successful as I hoped. I did however speed up write speed for all users in tomorrow's release, and HDD db users may still get a boost in long-running clients. Should be less cleanup chug after big jobs finish making updates, in any case. Let me know how it works for you.

>>14973

There's some new cookie you need, I think: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Downloaders/Gelbooru

>>14981

Thank you for this report. These pop up every few months. They all seem to have been false positives so far. I would be interested if you know exactly which files were causing problems. One file I use to make custom UPnP mappings easier for advanced users - upnpc_win32.exe - is so often the problem that I am thinking of not even bundling it with the program.

It is also interesting to know which virus scanner gave the result. On those big sites where you can scan with like 60 different software at once, the false positives only tend to show up on a handful. The problem with the specific anti-virus program usually goes away in a few weeks when updated definitions roll out.

I had a good week. I fixed several important bugs, improved db write speeds for many sorts of clients, and made the 'potential duplicates' search run in the background, in a way that does not interrupt you.

The release should be as normal tomorrow.

>>14983

Hi, dev.

I'm >>14965.

And version 421 is even faster than 420.

In my case, database files and thumbnails are on SSD and images are on HDD. But when I'm browsing images(I'm ESL, so I don't know how I should exactly say this in English. What I want to say is like "switching images in a tab", "show images one after another" something like that.), the stuttering doesn't occur.

This stuttering suffered me a lot. So this improvement gives me a comfortable experience of image viewing. Thanks!

As you add siblings to tags with the right-click menu in the manage tags window, the tag list doesn't update like it used to anymore.

>>14939

Thank you for this report, I will check it out.

I had a great week. I expanded the editable search predicates, making it easier to edit, and expanding what you can edit, including rating and OR predicates, and I was able to add custom defaults for all editable system predicate types. There are also misc fixes and additions, and significant speed/cancelability improvements to tag searches and wildcard-based tag and file lookups.

The release should be as normal tomorrow.

I unfortunately had a bit of a mixed week. I was short on work time, so I could only get basic editable system predicates done. This means you can change a system:filesize (and so on) after you add it to a search.

The release should be as normal tomorrow.

I accidentally put the 415 release links in here when I first uploaded, fixed now!

>>14895

>My test application of 6 tags to 10,000 files went from 52 seconds to 4.8!

VERY big optimization, thank you dev!

I did a lot of my preliminary mass tag-correcting last year and I had to do it in batches to keep everything from crashing and burning at the time, and still it hung a lot. Now that my duplicates are largely done with I've been going about my first serious attempt at comprehensively trimming down junk files. So far I've done a cursory run through tagless files for things to throw away, purged my lowest-quality source of files of all immediately obvious deletes, and now I just finished processing my 1280 smallest image files in batches of 64, deleting most of them. I've gone from around 2 saved per batch to around 12, so I'm getting to the end of that being practical.

My next task after sorting my less-strict duplicates might be finally tackling the hellscape that is my tumblr imports, which in addition to being the worst leg of the pre-full-tagging-and-archival-process sweep of my db, will have a lot of things that need the tags fixed because I'm dumb and switched my methodology halfway through (and so did Tumblr, but at a different time, and so did Hydrus in a way).

So this will definitely still be extremely useful to me because some of those tumblr image collections were giant.

Also my backlog for downloading videos and importing them to Hydrus has moved from deleting the extra junk links in jDown to actually having no more drive space left until I watch/delete/keep/edit video clips.

The ride never ends, but I have finally settled into a sustainable cruising velocity for now.

>>14903

Great, I am glad this will be working for you. I understand the manage tags dialog can still lag a bit on the final 'apply' step for very large changes, so I will look at that too soon, to spread the work out a bit.

I had a great week. I did a variety of small work, fixing bugs, adding new shortcuts, updating some libraries and downloaders, cleaning up old code and UI, and adding a couple of advanced features to the downloader system.

The network version will go up tomorrow, which means clients will have to update if they want to keep syncing with the PTR. There is no need to rush, and no penalty if you delay–syncing will just pause until you update.

The release should be as normal tomorrow.

>>14913

Thank you very much, this actually helps so much in downloading art. God bless you!

>>14851

>I do not plan to bring it back with these systems…

Could we get an option to display siblings in reverse? I mean instead of "censored (1) (will display as meta:censored)" it will show as "meta:censored (1) (tagged as censored)"?

For me, and the main reason I used the hard replace tags with siblings, was that it was easier to see which tags were missing and needed to be added. As it is displayed now a file might have the correct series: tag but it's added through a sibling to a non-namespaced tag long down the list of tags. So I might try to add the series tag when it essentially already exists, wasting my time. If siblings displayed in reverse it would be immediately noticeable.

Thank you as always, based dev.

>>14870

Thanks. I agree about it is now less easy to see which tags a file actually has. I won't replace the current system with what you are asking for here, as the main purpose of that dialog is to allow editing, and if I show the reverse, then if multiple tags went to that ideal sibling, it would not be easy to double-click it to remove.

I have two main things to do next:

-Update the render code to draw multiple colours per line. I can then highlight siblings by drawing the 'will display as' part in the correct namespace colour.

-Update the list code to allow for 'ghost' entries, which will render with softer colour or something to make them look different. I can then populate the list with parents implied by other tags, and then potentially add what you want here, 'x (implied by y)' as a ghost in addition to the original 'y (will display as x)'.

I had a great week. I caught up on smaller work, added some new default watchers, fixed some of the dodgy multi-column list sizing, and greatly sped up the manage tags dialog when dealing with thousands of files at once.

The release should be as normal tomorrow, maybe a little late.

Thanks for your hard work! I just updated from ver.395 (don't worry, i jumped from 395 to 400 to 405 to 414.) Is there a way to remove tags implied by a parent to stop showing up in suggested tags in the tag manager? Hydrus used to remove "male" from suggested tags if I had added something like "1 male" which used to add "male" to the file.

>>14854

Ah, damn, I see. I had to compromise a couple things in manage tags to get this working in time. I will see if I can fix this next week.

>>14859

I am sorry for the trouble. 415 tomorrow should have much less laggy background maintenance. Please give it a go, check your sync progress under tags->sibling/parent sync->review tag sibling/parent maintenance, and let me know how it goes.

Most SSD users with a typical PTR-syncing database should have seen the lag spikes disappear after no more than, say, 40 minutes of use. If you still get it in 415, and the sync in that review window does not seem to be progressing at any reasonable speed, please try disabling the maintenance in active and idle time in that same menu, and let me know.

>>14862

Damn, thank you for this report! This should not be happening, I will check out what is going on.

I had a mixed week. While 414's parents cache seems to have gone well, making the new maintenance code less laggy proved difficult, and it monopolised my time this week. I did a small amount of other work, but not much, so 415 will mostly be a 'polish' release that makes 414 nicer for all users.

The release should be as normal tomorrow.

>>10279

is there any info on what the calls will be? wondering if it's read only, because it would be nice to sort on mobile too

>>10332

Read-only would be easy, for writes, the things are 1. These apps willing to make a write function 2. Hydrus can handle write functions

BooruNav - discord.gg/4Sb5YrV (Android exclusive)

Animeboxes - discord.gg/kYgGJhF (Android or iOS)

Mignori - discord.gg/zf3SANY (iOS exclusive)

Nori - discord.gg/3DTNcHu and https://github.com/tjg1/nori

MBooru is on Telegram only https://github.com/onlymash/moebooru-android

Chinese: https://t.me/joinchat/BYcr90XsQO6188i597Qe7w

English: https://t.me/joinchat/BYcr9xAeSpDoEDf9swQw1g

>>10279

any updates on the progress of this?

Yes im aware I necro'd the hell out of this thread

>>14760

check ver412 dev post.

He said that we can browse via phone by animeboxes

Are there any working open source apps?

I don't want to trust closed code with full access to my loli porn.

>>14242

>lolifucker

god i wish

>>14233

op can u link the instruction plz?

>>14667

seconding this

>>14667

>>14759

>>14806

>>14800

Thanks. Yeah, I agree. When I next do big work on the network engine, I will be making some UI for the 'domain manager', and more per-domain tools, so the system will be better able to detect 'ah, this site is down' and not affect the whole system. When I have that up, I could migrate the proxy options to that new system and UI and change depending on domain.

The new 'no_proxy' setting, btw, seems to be ignored due to a bug with the networking library I use (requests), so with luck that should be fixed soon, which may make things easier in the meantime. If there is any way you can set up a system-wide TOR proxy, that hydrus can inherit with no proxy settings or a careful 'no_proxy', it might be possible to figure out multiple proxies for certain sites more quickly than waiting for me to develop it. This might only be reasonable to do on Linux, though, where there is better support for mapping this sort of thing.

I had another ok week. I am not working as fast as hoped, but the new maintenance system works. The very long wait times to recalculate siblings is gone–it now all happens in the background over time. I am now working on converting parents to the new system.

I am less confident I can get 414 ready and good for October the 14th, but we'll see how it goes.

I had a great week. The siblings and parents database code is mostly in place, and I was successful in unifying their application method to a simpler algebra that should result in faster siblings/parents work across the board.

I now have to tie up many database loose ends, cleanse the old UI-side parents system, and polish. About 150 small things and 3 big things left. I think I can be ready for October 21st!

I had a good four weeks migrating tag parents to the new database cache. Like siblings, parents are now applicable across services and completely undo-able. Furthermore, the large CPU requirements we experienced after the siblings cache are gone–a new maintenance routine cuts all parents and siblings work into pieces and figures it all out quietly in the background.

The release should be as normal tomorrow. It should only take a few seconds to update!

Sorry Chris, I hope you find some 32-bit software that will do what you want!

>>14719

Gaming? It has been something like 13 years since I switched my non-gaming Linux to 64bit.

Very few maintain the 32bit x86 side of things anymore. Not just Windows software, even most Linux distos partly or fully dropped support. It just doesn't make sense to run x86 processors in 32bit with fewer less capable registers and many other downsides.

> Several months of work to tune up Windows again

Maybe you'll switch over too at some point. Copying configs or running ansible scripts or whatever you do is not quite as hard.

Or you pick something else, but I expect almost everything to eventually drop 32bit x86 on Windows, and it mostly already happened, even for things as popular as web browsers. It's definitely not just Hydrus. I believe it won't be fun going forward on that.

>>14719

Hydrus wouldn't be able to shine with the limitations of a 32 bit environment. It's just not meant for the types of use-cases that a 32 bit machine is capable of. Believe it or not, managing hundreds of thousands of files requires a lot of memory.

just install a 64 bits OS lole

It's time to leave 2004, neetboi