/hydrus/ - Hydrus Network

As always, thanks for the update.

It seems to have some problem though. On the Linux version, I'm getting this error for the url downloader, for apparently any website I try:

TypeError

PendURLs() got an unexpected keyword argument 'filterable_tags'

File "hydrus/hydrus/client/gui/ClientGUICommon.py", line 1794, in eventFilter

File "hydrus/hydrus/core/HydrusData.py", line 1542, in call

File "hydrus/hydrus/client/gui/ClientGUIControls.py", line 732, in EnterText

File "hydrus/hydrus/client/gui/ClientGUIManagement.py", line 3780, in _PendURLs

Also I think hydrus companion doesn't work alright, but I haven't found a good error to copy for that yet.

>>14744

Hey, I am sorry for the trouble. I messed something up here at the last minute last week and I did not catch it in testing. Top priority this week is to write a better test suite for the downloader pages and the client api so this does not happen again.

I have made a new hotfix today and replaced the files the download links above point to. Please download again and let me know if you have any more trouble.

I had an ok week. I mostly fixed bugs, aiming for a nice 'clean' release before I go for the parents database cache.

The release should be as normal tomorrow.

I had a great week. I was able to catch up on smaller work, optimised a variety of db functions, including the new sometimes-slow tag and wildcard queries, and added a 'lite' mode for the big 407->408 update so PTR-syncing users on HDDs can update in less time.

The release should be as normal tomorrow.

Just updated from v374, I made a backup just in case but it went smoothly. Thanks dev!

Also finally polished off the last of my duplicates just before the update, my high watermark was in the high 70ks, so I think I sorted out at least 200k duplicates overall since whenever I started using Hydrus, probably late 2015 or sometime 2016?

>>14741

That is pretty killer mate, well done. I am confident a big update to dupes is coming in 2021, including filter workflow and beginning of filter automation, so if you have thougthts on the current system's strengths and weaknesses, I would be really interested to know.

>>14742

I went over 8k characters and notepad++ doesn't count spaces but 8ch does, so rather than try and figure out where to split it I just pastebinned

https://pastebin.com/mBZLa7tk

>>14700

Do you have help->advanced mode turned on? I think I hide some of those clever commands behind that.

>>14702

Ah, do you mean for 'export folders'? I experimented with this with manual export (I think you need help->advanced mode turned on), and I did have a plan to try this out for export folders. I'll bump this up and take a proper look.

>>14703

It is not a pipe dream, but it is such a big job that it'll have to win a 'big job' poll. I expect to get through three or four big jobs a year once I am back to normal schedule, and the next poll should be up by Christmas. Let's see how it does there, and we'll see when we can expect it.

>>14709

>>14712

>>14713

I plan to have nice support for traditional zip/cbz/cbr files in future. It is sufficiently popular that I expect it to win a 'big job' poll pretty soon. I'll have proper thumbs, browsing of internal pages, and support for conversion back and forth between individual files and compiled 'chapters' or 'volumes' or whatever. This will be a part of my attempt to solve the problem of handling awkward 'page' tags.

>>14715

>>14721

Thank you for this information. I will attempt to detect WAL and networking gubbins in future. I generally do not recommend running the db itself across a network share due to latency issues, but I hope I can at least detect the situation and throw up a warning, and ideally recover and set WAL off.

NEW THREAD

>>14731

>>14731

>>14731

>>14731

>>12874

You dont. You just hope that the techno-jew doesnt care about whatever it is you're hoarding.

>>12872

>60 bucks per year

It's not expensive, but it's still 60 dollars more than my current backup expenses.

>>12874

I don't give a fuck that some NSA employee can look at my patreon content.

>>12872

I like knowing that all my kino tiddies are saved somewhere in Buttfuc, Tennessee.

It's 3h of my job for a year of peace of mind.

>>14699

Got an error when trying to search by time imported

DBException
OperationalError: no such column: timestamp
Traceback (most recent call last):
  File "hydrus\core\HydrusThreading.py", line 382, in run
    callable( *args, **kwargs )
  File "hydrus\client\gui\ClientGUIManagement.py", line 4932, in THREADDoQuery
    query_hash_ids = controller.Read( 'file_query_ids', search_context, job_key = query_job_key, limit_sort_by = sort_by )
  File "hydrus\core\HydrusController.py", line 615, in Read
    return self._Read( action, *args, **kwargs )
  File "hydrus\core\HydrusController.py", line 194, in _Read
    result = self.db.Read( action, *args, **kwargs )
  File "hydrus\core\HydrusDB.py", line 1028, in Read
    return job.GetResult()
  File "hydrus\core\HydrusData.py", line 1755, in GetResult
    raise e
hydrus.core.HydrusExceptions.DBException: OperationalError: no such column: timestamp
Database Traceback (most recent call last):
  File "hydrus\core\HydrusDB.py", line 629, in _ProcessJob
    result = self._Read( action, *args, **kwargs )
  File "hydrus\client\ClientDB.py", line 12868, in _Read
    elif action == 'file_query_ids': result = self._GetHashIdsFromQuery( *args, **kwargs )
  File "hydrus\client\ClientDB.py", line 7215, in _GetHashIdsFromQuery
    files_info_hash_ids = self._STI( self._c.execute( select ) )
sqlite3.OperationalError: no such column: timestamp


Database Traceback (most recent call last):
  File "hydrus\core\HydrusDB.py", line 629, in _ProcessJob
    result = self._Read( action, *args, **kwargs )
  File "hydrus\client\ClientDB.py", line 12868, in _Read
    elif action == 'file_query_ids': result = self._GetHashIdsFromQuery( *args, **kwargs )
  File "hydrus\client\ClientDB.py", line 7215, in _GetHashIdsFromQuery
    files_in

Hey dev, I'm having some trouble with the Twitter subscriptions. I just tried to set one up and it says to put in the username but whether I put the handle in with or without an @ it always says this subscription appears to be dead when it tries to update. I've tried this with multiple different accounts. Any idea what could be causing that?

I had a good week. I fixed some recent search bugs and I capped off the new siblings work. Everything siblings is now running off the new cache, so the slow 'loading tag siblings' step of boot no longer occurs!

The release should be as normal tomorrow.

>>14706

Thank you for this report, I am sorry for the trouble. I believe this happens when the 'time imported' search is mixed with certain other search predicates, either tags or some more unusual system preds. It should be fixed tomorrow, but let me know if you have any more issues.

>>14707

I am not sure. This is on the new nitter downloader? Can you give an example username, so I can try my end?

>>14707

are you using "nitter media lookup"? I have my subs for media and retweets split up so the "media and retweets" one might not have worked.

Currently updating many more versions at once than recommended. 384 to 407 went really fast, but not it's taking forever on "generating combined tag display cache 2/2". After about half an hour it now reached 56/41,805, so will probably take about 17.4 days at this pace. DB is on a NAS using HDDs, while Hydrus is set to use a 2 GB DB cache. Can I expect it to speed up, or should I revert to my 384 backup and only update to version 407 for now?

>>14687

I believe you can force the App to look to a new location with this:

open -n -a "Hydrus Network.app" --args -d="/path/to/db"

Basically you are doing the same -d parameter, but you have to do it slightly odd for the App. More general info here:

https://hydrusnetwork.github.io/hydrus/help/database_migration.html

Worth doing a test on an empty location to make sure it works before doing the real thing, and always have a backup.

>>14692

That 408 update >>14654 involved a lot a big db work, I am afraid. It should accelerate, as the early numbers are the heavy ones, but it may just not be feasible on an HDD, especially in a high latency situation. The PTR is mostly SSD-only now.

If it is still chugging along, you should be able to cancel and just install 406. In tomorrow's release, the code that does this works faster. The speed-up will be retroactive to that update step, if you want to try again with v410.

If you have not cancelled it already, and it has accelerated, it is probably worth just letting it finish. That combined part, though heavy, should be the last step.

There will be another heavy update for v412, but not as big.

I had an ok week. I wasn't as productive as I hoped, but I fixed some bugs and made great progress in optimising the new sibling code, and older code too. The various lag spikes in systems like 'related tags' should be smoothed out, many file searches are snappier, and tag processing should be faster than it was in v407.

The release should be as normal tomorrow.

>>14696

Circling back this worked perfectly. Using absolute paths it was easy enough to create a quick batch file and chmod it +x.

Hey, I messed up the siblings dialog in 407, it won't ok correctly. I'll fix it all up in the bigger siblings work in 408.

>I fixed a bunch of weird hover window issues in the media viewer

This caused an annoying bug. When you have the manage tags window open in the media viewer, and you move the mouse pointer outside its window it loses focus and that then makes the left tag list pop up over it when you move the mouse pointer back over the window. Quite annoying.

Previous workflow would be having the tag window in focus all the time and with the mouse over media viewer use scroll wheel to go previous/next image. Can't do it now because the tag window loses focus so I have to click on it again to edit the tags but then the annoying tag list covers it when I try to click it. Please restore previous behavior.

I had a good week getting into the meat of the tag siblings database work. There is a lot to do, but I have a plan I am happy with, it is going well, and I am confident I will have a good first version ready for next week, Wednesday the 19th.

So, no release tomorrow!

>>14641

Thank you for this report. It is driving me nuts as well, I know what is going on and will fix it.

I had a great couple of weeks moving tag siblings forward. The client now handles siblings more quickly and accurately using a new database cache. They are easier to undo going forward, and you can also choose where siblings apply on a per-service basis (e.g. turning off the PTR's default siblings and using your own instead).

The release should be as normal tomorrow.

I had a great week fixing some bugs and making the slowest parts of the new sibling cache code work much faster and more efficiently. The lag that was introduced last week should be reduced, I hope, by about 4-10 times, making it much less noticeable. The UI feedback on longer jobs is also improved.

The release should be as normal tomorrow.

>>14597

Got a small issue. For a few versions now, when the schedule maintenance on client exit is for vacuuming, it gives the "is now a good time for maintenance y/n?" dialog, but when I click "yes" the vacuum never actually runs. Client just exits as normal. All other maintenance seems to work, just the vacuums don't (external mappings, master, etc.)

>>14600

Hmm, thank you for this report. I'll double-check how those tests work. Generally the issue here is that vacuums take a while, so it tries to go 'hey, the user doesn't want shutdown work to be longer than x minutes, can you do a vacuum in <=x?'. If the dialog's test is passing, but the test on the actual vacuum is failing, it'll quit out immediately, thinking there isn't enough time.

>>14604

I've still got it set to the default 30 minutes. My DB is fairly small, so usually the vacuums don't take longer than 10-15 minutes for the whole thing (maintenance -> vacuum)

I had an ok week. Some of the work I did (siblings improvements) is not ready to be turned on yet, so tomorrow's release mostly just has bug fixes, including several related to the media viewer's hover windows positioning and behaviour.

The release should be as normal tomorrow.

>>14613

Hmm, thanks.

>>14577

At the moment, if you add 'series:metroid' to 'character:samus aran', then anything with samus will actually get the tag metroid added. This applies retroactively to existing files.

The problem with this is when some lad adds a bad sibling/parent/whatever that adds 'shadow the hedgehog' to every file with 'shadow'. Then that damn hedgehog appears all over the place, and undoing the operation is tricky. Same deal if they have some private, local tag parents that leak over onto the PTR or similar.

So, with virtual parents, the actual tag tables will not store 'metroid'. There will be a second, mirrored table that applies metroid to all the samus stuff, but it will be a 'virtual' tag, implied by the parent relationship but not actually on the tag service when you open up manage tags. It'll have some entry, a bit like how siblings say 'samus_aran (will display as character:samus aran)', just so you know it will show, but it won't be hard-baked.

Then, if the relationship is bad, or needs to be remapped or whatever, then the virtual tag table can be recalculated and the base tag store is not affected. Parents become nicely undoable.

This 'presentation' tag cache will be useful for pre-computing siblings as well (which are currently mostly virtual, but computed expensively on the fly), and some other things like tag filtering so I can hide all the 'title' tags if you want without spending CPU on it for every request. I can also search that pre-computed table efficiently, and reduce/eliminate the inaccurate (5-9) tag counts you sometimes see. It'll also allow cheap namespace-to-namespace siblings, so you can rename creator to artist or whatever.

>>14581

That's an interesting thought. I enjoyed working on the local booru, but I never had time to make it nice. I don't think I'll work properly on it again, instead focusing on the client api so others can do it better.

I actually think what you would like here is sort of doable with the client api right now. That can query the current session, and all the files in each page. If you wanted to map it to a nice URL structure, you'd have to host your own web server that talked to the client api, and it would obviously be a bunch of work to implement this all, but doable.

The lads who are working on different front-end browsers for the client api are coming along, by the way. I expect to announce at least one in the next 4-8 weeks. Should be an Android APK or something, let's you search by tags and view thumbs/files and everything.

I expect to keep working on the client api every now and then to keep adding tools.

Thank you based dev

I had a good week. Along with some simpler code cleanup, I fixed some bugs, added some quality of life, and made it easy to detect and remove duplicate subscription queries, even across different subscriptions.

The release should be as normal tomorrow.

>>14583

I look forward to seeing what the community can do with the client API! The idea of an Android browser sounds great, and I already get a lot of use out of Hydrus Companion.

I think Flask could get pretty close to what I was thinking, since custom URL's can be added to specific app functions.

I'm a beginner when it comes to this stuff, but I think it could be fun to play with the API, especially if there are a few code samples floating around that I could learn from as well.

>>14540

I have a master job to migrate the old wx-based colours in options->colours to Qt qss stylesheets, so everything is unified there, but I don't think I have power over the colour of the title bar. I think it is entirely OS-based, so I don't actually render any of it, nor have control over it other than basic flags like 'show a minimise button'.

I saw some other users talking about this recently. Under Win10 'personalize' options, where you set the colours, you can force the title bar to shade with your chosen colour with a checkbox, but it might be brash, and of course it is system wide, so it may look bad with your other programs.

>>14543

Is this for launching a known url? I think if there are multiple urls, it opens a list asking you which one to open, right?

What would you like to be the option/discriminator here? I assume URL Class would be correct, so I could add an edit panel for the 'open known url' command that let you select like 'gelbooru file page', and it would target that.

If you mean to open the 'edit known urls' dialog, I think this is available under the 'media' shortcut set, called 'manage file urls'.

Thanks dev

We've made it to the Forbidden version and I've finally made it under 30k dupes

I had a good, full week that was almost entirely focused on migrating all the multi-column lists in the program to a unified options system. All lists in the program now remember their columns' widths, and those that typically sort now also restore their last sort.

This is a neat change, and it means it will be easier to add column selection and rearrangement in future, but that's basically it for the week. If you aren't interested in columns, feel free to skip!

The release should be as normal tomorrow.

>>14545

Oh, I meant in the system tags, where you have to select "known url", then paste what you want into the field. It would be nice to have a shortcut to get to that field, or maybe the option to just search for URLs directly.

Aww no, I made a mistake!

I typed this in notepad and when copying it to here, I accidentally missed the first two paragraphs. I'm very sorry about that.

Here it is:

Hey there, I apologize if making a new thread isn't warranted for this but it's not really a simple Q&A question, so…

I use Hydrus Client for sorting through my massive image collection every day and this program really is amazing. Never before have I been able to find my favorite images so fast.

One more thing actually: if you think these ideas aren't good or you don't have the time to implement them, then is it hard for a complete noob in programming to go into the source code and find the menu code and change it to suit my needs? Making the menu have easier access to alternates and slightly easier access to editing tags would be wonderful for me, but maybe it wouldn't be for other people.

Dev has said he wants right click menus to be entirely customizable, but that is probably not a very high priority feature so don't expect it anytime soon.

Meanwhile I suggest you learn the keyboard shortcuts. F3 is manage tags, F9 is new page etc. Change or set them up yourself in File > Shortcuts menu. I have "set alternates" as shift+a, for example.

Thanks dev!

Not a single 404 not found joke, I'm disappointment.

>>14567

You could not find any joke? :^)

I had a good week. As well as misc cleanup and bug fixes, I added a namespace filter to 'system:number of tags', so you can now search for files with 2 series tags, more than four character tags, or any combination you can think of. Most 'number of tags' searches are faster, as well.

The release should be as normal tomorrow.

I am pretty sick, so I will delay 403 a week to July 8th. Covid positivity is about 9% here, so it probably isn't that, but I am knocked about a bit.

Thanks everyone. I hope to be back on Saturday for messages as usual.

>>14513

Oh I'm sorry that you are sick.

Get well soon,Dev!

Thanks lads, back in business. Just knocked me out a couple days.

I was knocked out for a couple days from illness, but I otherwise had a great couple of weeks doing small work. Shortcuts are nicer to work with, a bunch of UI is cleaned up and fixed, and tag import options now supports an explicit whitelist for requiring tags for file import.

The release should be as normal tomorrow.