/hydrus/ -Version 321

Name
Email
Subject	REC STOP
Comment *
File	Select/drop/paste files here
Password	(Randomized for file and post deletion; you may also set your own.)
Archive	Archive [500 char limit]
* = required field	[▶Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field)
Allowed file types:jpg, jpeg, gif, png, webp,webm, mp4, mov, swf, pdf Max filesize is16 MB. Max image dimensions are15000 x15000. You may upload5 per post.

YouTube embed. Click thumbnail to play.

Version 321 hydrus_dev ## Board Owner 09/05/18 (Wed) 21:31:52 e19786 No.9891

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v321/Hydrus.Network.321.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v321/Hydrus.Network.321.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v321/Hydrus.Network.321.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v321/Hydrus.Network.321.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v321/Hydrus.Network.321.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v321.tar.gz

I had a great week. An important milestone in the downloader overhaul is done. Users who use 'custom' boorus will want to read this post and may like to put off updating a week or two.

downloader overhaul

So, this week I have switched over the old hardcoded method of converting search text into an initialising gallery url to the new user-customisable and -shareable GUG (Gallery URL Generator) objects. This was the last hardcoded component I needed to replace, and so I was finally able to disconnect and delete all the old downloader code. We are now entirely running on the new customisable and shareable system!

Regular users should not see much different today. Your subs and galleries should switch over to the new 'gallery selector' values automatically. Nothing is too functionally different, but a bunch of the behind the scenes is completely new and much neater all around. The manage boorus dialog, which was the old way of cobbling together some user-customisation, is also gone–all sites now run on the same new network submenus.

I've added new downloaders for artstation and derpibooru and brought back the newgrounds artist lookup (which works better than you might think!). Now we are completely on the new system, I expect to roll out more new searchable sites in future updates myself and other users to start creating and sharing complete downloaders themselves. In the coming weeks, I will write an easy one-action way for regular users to import these new downloaders (at the moment, it is still a little ugly and technical to do and set up).

Needless to say, while my tests have gone well, this is a big update, so make sure you refresh any backup before you update, and let me know if you have any problems.

Now the new downloader is 'out' and its functions and terms are pretty much fixed, I will also be finishing the 'how to write a downloader' help, which will be a comprehensive guide on the whole new system.

custom boorus

Any 'custom' boorus (i.e. ones you created yourself or imported via the manage boorus dialog) will unfortunately not be completely automatically updated. There is not quite enough information in the old booru object to generate all the parts I need in the new system (I was able to generate GUGs and gallery/post parsers), so you will have to add/do some things to get them working again.

If you are comfortable in the new system, you will need to:

-Get a gallery and post url class for the booru.

-Double-check (and rename, if you want) the auto-generated GUG and parsers.

-Link the url classes to the parsers.

-Run a test search on the new downloader.

-Update the downloader selection in the downloader or subscription's gallery selector button.

If you are not comfortable creating url classes but are ok with importing and linking them, I recommend you check the github site here:

https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/NEW%20Download%20System

–for url class updates, and if what you want is missing, drop requests here or in the discord, and I or another advanced user will see what we can do.

If you are completely lost but you still want to use your custom boorus, I recommend you put off updating this week. You can either find a convenient time to learn how to do the above steps (I am happy to help if you run into trouble), or you can wait a few weeks for me to bring out this 'downloader easy-import', which will make it as simple as opening one dialog and dropping a png file from the above site on it.

Note that if you used a custom derpibooru booru, since I've added explicit functionality this week, all you have to do is the final step above: go to your derpi gallery downloaders or subscriptions and change the selected gallery from the broken entry, which will be something like 'derpibooru: not found' to the new one 'derpibooru tag search' and you should be going again. You might still like to run a test download if you use more than one tag–the GUG I made uses their search format, which preserves spaces but requires a comma to separate multiple tags, like 'rainbow dash,straight'.

misc

I put some time into the native video renderer this week, cleaning up the pipeline and how the different components talk to each other. It has cleared out a bunch of jitter and scanbar calculation jank. I think I will put some more time into it, but overall I am happy with the changes. Please let me know if you run into any particular trouble with certain files.

Also, before doing client processing, repository services will now check for a little additional free space (now half the size of your client.mappings.db file, rather than a flat 1GB). If they lack the space, they'll throw up a popup and safely pause the repo. Since a PTR client.mappings.db is now about 8.5GB, please make sure your hydrus drive (and system drive, for its temp folder!) have about 5GB free at least!

full list

- downloader overhaul:

- the basic downloader overhaul is complete! at this point, any user can create and share the objects required for a completely new downloader! it is still rough in some places, so a round of EZ-import is coming to make adding new downloaders a single easy drag and drop action

- rounded out the ngug (nested gugs, which contain multiple gugs) code

- updated the edit gug panel to deal with gugs and ngugs on different notebook pages

- added a bunch of logic to this panel and backend data handling to deal with missing gugs in ngugs

- if an ngug cannot find a gug by its internal identifier key, it will now attempt to fallback to its simple name, and will silently fail if no gug can be found. all gug tracking now uses this 'key first, name later' id method, so downloaders and subs should generally survive gug renames and same-name overwrites

- the gallery selector now works in gugs. it has two 'pages', depending on which gugs are set to 'display', and will note if the chosen gug is cannot be found in the current definitions. the gugs have slightly more specific names ('gelbooru tag search', 'hentai foundry artist lookup', etc…) than before

- the gallery selector also puts 'non-functional' gugs (i.e. those with no parsable gallery url class) to a third page

- moved the gallery downloader gallery and file pipeline completely over to the new system

- the gallery downloader will now bundle nested gugs (like hentai foundry artist, which searches both works and scraps) into a single downloader

- moved the subs gallery and file pipeline comppletely over to the new system

- the subs gallery sync now handles nested gugs (like hentai foundry artist, which searches both works and scraps) in an interleaved manner and make behind-the-scenes checking decisions in a clearer and more logical way

- subs should now make correct 'hit limit' stop reason reports and not generate new gallery pages when the current page has exactly enough results to hit the current file limit

- artstation artist lookup is now available as a default downloader

- newgrounds artist lookup makes a triumphant return. it works pretty well, given how flash and NG has changed since

- derpibooru tag lookup is now available as a default downloader. due to unusual search syntax on derpibooru, please enter queries exactly as you would on derpi, using ',' or ' AND ' to separate tags (such as 'rainbow dash,straight')

- pixiv now has multiple artist lookup options–either images, manga, ugoira (doesn't work yet!), or everything

- the old downloader code is deleted!

- the old manage booru dialog is deleted!

- 'custom' boorus (i.e. new ones you created or imported to 'manage boorus'), cannot be completely automatically updated to the new system. I've figured out a way to generate new gugs and gallery&post parsers, but they will miss url classes to get working again. your custom-booru subs will notice this and safely pause until the issue is fixed. if you rely on custom boorus, please check the release post for info on this–you might like to put off updating

- many misc changes and fixes to gugs and overall gallery url handling pipeline

- some misc refactoring and concept-renaming in gallery pipeline r.e. gugs

- when the downloader tries to import what looks like a raw html file, its error notes will specify this and suggest a parser may be needed

- moved the 'media viewer url display' options panel from the manage url match links dialog to the new network->downloaders->manage downloader and url display

- this new dialog also hosts a list for managing which downloaders to show in the first list of the downloader selector

- .

- misc:

- gave the video rendering pipeline communication logic a quick pass, cleaning up a bunch of bad code and other decisions. the video renderer should be quicker to respond to various changes in scanbar position, and incidences of the frame buffer suddenly sperging out (usually inexplicably falling behind the current frame position or deciding to regen for no apparent reason) should be greatly reduced if not completely eliminated

- the test that stops repository processing if there is not enough disk space now uses half the current size of client.mappings.db for its estimate (previously 1GB) and also tests temp folder location free space (just as the vacuum test does) and reports this nature of the error along with pausing the repo, stopping further attempts

- might have fixed another out-of-order dialog close/open event combination during manage tags close->advanced content update open

- fixed gallery queries that include '/' (or some other unusual characters) that end up in the 'path' of the url (as opposed to the query). this fixes 'male/female' on e621, for instance

- 'advanced mode' users now have a 'nudge subs awake' menu entry below 'manage subs'. this simply wakes the subs daemon (which usually only checks once every four hours), in case any subs are due

- 'db report mode' now reports every db job as it comes in (formerly, it only reported some optimisation esoterica). this makes it a more lightweight version of 'db profile mode' for several debugging tasks

- fixed a tiny issue in fetching the 'how boned am I?' stats when the user had zero inbox/everything count

- fixed a typo in the default new url class object that was breaking the edit ui panel

- highlighted the quiet filename tagging options on edit import folder panel

next week

I'd like a simple week. I'll deal with anything that comes up due to this week's big changes, but otherwise I'd like to catch up on help docs and just do something easy and interesting.

____________________________

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/06/18 (Thu) 03:55:21 db8892 No.9892

Firstposting in aesthetic landmark thread.

You said the process to import custom downloaders is a bit obscure still, but is there a simple export to image process already set up?

My database is long overdue for a full backup sadly, otherwise I'd try for the first custom site script too.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/06/18 (Thu) 12:32:17 55f78d No.9893

Nice, so it's finally here, seems like my subs are working just fine as too. Better get to downloading all the files I've been holding back for a while and set up more syncs. Maybe I'll take a look at custom GUGs but I don't think I'm technologically competent enough for that. Thanks dev.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/07/18 (Fri) 06:20:59 84d252 No.9894

File: 1409beeece6ef6e⋯.jpg (665.69 KB,3160x3160,1:1,56911da30471e8bb2349eefd02….jpg)

While I appreciate your work on this downloader stuff, since I personally don't use it much I am glad you're almost done and looking forward to what you're going to work on next.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/07/18 (Fri) 08:16:26 d9aca2 No.9898

I havent updated since v293

In a nutshell, what have I missed?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/07/18 (Fri) 14:32:47 db8892 No.9899

>>9894

So what do you use hydrus for, just sorting/storage?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/07/18 (Fri) 14:37:12 76064c No.9900

I have some booru tag searches that have failed due to temporary server-side errors; is it possible to retry the gallery page that failed or resume searching from that point?

The custom parsers are otherwise working great.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/08/18 (Sat) 14:11:53 84d252 No.9903

>>9899

I mostly use this script https://8ch.net/hydrus/res/1914.html

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 09/08/18 (Sat) 18:55:43 e19786 No.9909

BTW: It looks like artstation and maybe tumblr are only searching the first 'page' of api results in the new pipeline. It is fixed for v322.

>>9892

Yeah, if you are willing to dig into it, the technical dialogs are actually easy to import/export with. The manage gugs/url classes/parsers dialogs under network->downloader definitions all have import export buttons that work either in clipboard or png files, and they'll all accept png file drag and drop imports. The guys running the github prefer the text clipboard format, but I expect/hope the 'all in one' thing to favour pngs.

>>9893

Best way to learn this new system is just to check out what I've got in for the defaults. It isn't that complicated–basically a combination of 'generating an url', 'recognising an url', 'parsing the html/json result of an url' all working together.

>>9894

Me too. Happy it is done, but I'm pretty sick of it now. I'll spend a few weeks cleaning up and finishing last things, and then I am on to the python 3 update. Back to fun stuff in November, is current hope.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 09/08/18 (Sat) 19:01:58 e19786 No.9910

>>9898

Mostly downloader stuff. If you try to update to the latest all in one go, make sure you make a backup beforehand. Depending on what you do with your client, your downloader pages and subs may be converted to funky new objects. The ideal will be that everything just works for you, but simply looks a little different, but it might not be that easy.

If you can, see if you can 'clean up' your current gui session before you shut down last time before updating. If you can try to not have 100k files open and 30 complex nested download tabs, it'll all go easier and quicker.

Have a notepad nearby to write down any subs or anything that break or you need to check back on, and let me know if anything big doesn't work. If going up to 321 in one go doesn't work, try doing v300, then v310, then v321, or even with smaller gaps.

>>9900

Yeah, please make a note of them/keep them in your session if convenient. Now I have the final pipeline more firmly fixed, I'm more comfortable adding right-click->retry search from here kind of menu stuff to the gallery log, like I did with the file import status window after that came in. Should be in in the coming weeks, if not for v322.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/08/18 (Sat) 23:00:37 330634 No.9912

>Current to-do list has: 1,595 items

I'm not sure if this is a meme or not, but if you're willing to throw it up somewhere for me to read I might be willing to throw $$$ at specific things if I want the feature and/or want to see the bug fixed.

I can't think of anything offhand to ask for since you've done two of my largest requests already and I haven't had much time to use Hydrus since V315.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/09/18 (Sun) 08:00:43 bb34c2 No.9914

>>9912

The whole process is democratic, you can't "buy" features.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/09/18 (Sun) 08:43:57 d13b1b No.9916

File: 844b8cfd955677c⋯.mp4 (626.88 KB,1920x1080,16:9,hydrus.mp4)

>>9891

Thanks for all the work you've done, dev-san. I've got a weird issue happening with the file search tag dropdown.

(Worded weirdly, but you'll understand once you see what I'm talking about)

This is my first run of hydrus under Linux, my db had no issues crossing over from the Windows install it was originally created under. I've not had any issues thus-far with anything else specifically, except for this. Being able to tell hydrus to wait a second and not to search immediately is more useful than you think when you can't use it. ;_:

If there's a better avenue for reporting this, let me know. \o/

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/09/18 (Sun) 10:49:20 84d252 No.9918

File: 104943eaa6af34f⋯.jpg (31.49 KB,427x427,1:1,1482605384810.jpg)

>>9916

>PENDING 500,785

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/09/18 (Sun) 15:56:58 d13b1b No.9919

>>9918

They just kinda pile up, and I don't want to flood the ptr with potentially bad tags.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/09/18 (Sun) 17:37:38 b5ab8a No.9920

>>9916

What is this? Cinnamon?

Go to options > Gui > Always embed autocomplete…

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/09/18 (Sun) 20:38:08 d13b1b No.9922

>>9920

Perfect! That does the trick.

I'm running XFCE w/ Equilux.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/09/18 (Sun) 21:31:36 330634 No.9923

>>9914

This is demonstratively false, given that I already have [0]. It's a monetarily backed vote saying "I feel this is important, I'll even spend money to see it prioritized."

Bug/Feature bounties are not unheard of and are quite common in the FOSS world. Some bugs are particularly tricky to solve but only affect a minor amount of people - if someone wants to spend $1,000 on a feature/bug fix you bet your ass someone working on the project is going to add that feature/fix that bug, especially the more minor ones. [1]

[0] https://8ch.net/hydrus/res/9208.html#9220 and https://kimiwo.aishitei.ru/i/2o6sXeqkaDgemsPm.png

[1] https://www.bountysource.com/

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/10/18 (Mon) 03:37:22 5ead3f No.9925

>>9923

It does not have the ID of the Hydrus-dev, so that it could be a third party bounty, and will not be merged to the main repo (as he have not done so for years now)

See: https://www.patreon.com/hydrus_dev

"I have no rewards to offer beyond my appreciation and continued work. If you decide to support me, thank you!"

Also your bountysource does not show WHAT the bounty is and WHO is creating/accepting it.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/10/18 (Mon) 05:08:18 4a3ae8 No.9926

Thank you again based dev.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 09/11/18 (Tue) 17:46:33 e19786 No.9933

If someone wants to offer hard dosh for something that just stretches a bit beyond what I am already working on, as in >9220 , I'll take it. But I don't have the energy and social skills to regularly balance different paid priorities or individual bounties on random items. I mostly work on what interests me and what I've talked to people about recently. The current 'big thing' is typically voted on in a poll, although this giant downloader overhaul was my decision, just because the old system was about to catch fire.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 09/11/18 (Tue) 17:54:25 e19786 No.9934

File: 3816a418e83d2f0⋯.jpg (82.97 KB,1024x576,16:9,3816a418e83d2f076426007821….jpg)

>>9912

The actual to-do is a big mess on my desktop. I use ToDoList 6.8.10 by AbstractSpoon and highly recommend it. It has personal details from users copy-pasted into it and personal notes from me, so I'd need to sanitize it before I ever threw it up online. I most likely will not do this–I'm not a good team player, and my experience with collaborative to-dos and so on is mostly just overwhelming and cyclic talking about more and more issues rather than knuckling down and writing code.

The todo has a bunch of dupe items on it and stuff that is done but I haven't cleared out. It is basically just a big list of everything people have said 'hey, this would be neat' and I said 'sure, I'll add it to the list', and then the various addition sub-jobs and thoughts I've had.

A typical week adds 3-30 items! Sometimes I actually cut into the number–and as I tear down the big downloader overhaul job, I expect dozens if not a couple hundred to go as I delete old things that got done already.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 09/11/18 (Tue) 17:57:39 e19786 No.9935

>>9916

Yeah, I've had a lot of trouble getting different kinds of floating window to work on non-Windows, Linux especially. By default, it is embedded for a Linux/OS X client, but anyone who switches over from a Win install keeps the old setting.

BTW: if your Linux flavour has more problems (particularly instability problems) taking the 'run from source' pill is actually proving a decent solution these days. If my build runs fine, then keep using it, but if you get a lot of crashes, check out this document:

https://hydrusnetwork.github.io/hydrus/help/running_from_source.html

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/12/18 (Wed) 02:54:31 1925e5 No.9938

>>9934

You could make this public so people can write better reports on new features, you know?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 09/12/18 (Wed) 17:40:21 e19786 No.9942

>>9938

I'm afraid that's my point–I know I am bad at dealing with collaborative systems. I find it just becomes talking talking talking and stresses me out to the point of abandoning the whole platform and the users along with it. This was most recently true of Github Issues, which I ended up turning off because I couldn't keep up with it enthusiastically.

Like many other Anons, I'm a complete fucking sperg, so figuring out what workflow lets me put the keyboard work in week to week without imploding in drama has been an important part of the whole deal for me. Although the current to-do is a mess, I can't see a more productive system for me atm. I appreciate your point, but for me the benefits are outweighed by the inevitable personal drawbacks. I'm open to other solutions, if you can think of them.

Some users on the discord put together platform where they could discuss new features and so on–which I had no problem with, although I did have this same convo with them–but I don't know whether it actually succeeded beyond the first two fun weeks. If you are interested in this stuff, you might like to chase down whatever happened to it.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 09/14/18 (Fri) 14:55:49 c24889 No.9969

>>9891

Hey dev,

The Derpibooru integration works pretty well, but seems to be subject to the site's default filter which hides anything not kid-friendly.

You should request no filtering by passing `&filter_id=56027` in with each call to the search endpoints (assuming showing everything is a reasonable default).

The filter_id parameter is mentioned briefly on the API docs page [0], and the ID is from the Everything filter on the filters page [1].

[0]: https://derpibooru.org/pages/api

[1]: https://derpibooru.org/filters

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.