/hydrus/ -Version 314

Name
Email
Subject	REC STOP
Comment *
File	Select/drop/paste files here
Password	(Randomized for file and post deletion; you may also set your own.)
Archive	Archive [500 char limit]
* = required field	[▶Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field)
Allowed file types:jpg, jpeg, gif, png, webp,webm, mp4, mov, swf, pdf Max filesize is16 MB. Max image dimensions are15000 x15000. You may upload5 per post.

HookTube embed. Click on thumbnail to play.

Version 314 hydrus_dev ## Board Owner 07/11/18 (Wed) 21:03:35 8097cc No.9368

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v314/Hydrus.Network.314.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v314/Hydrus.Network.314.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v314/Hydrus.Network.314.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v314/Hydrus.Network.314.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v314/Hydrus.Network.314.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v314.tar.gz

I had a great week. Tag import options are now easy to mass-manage, and I have fleshed out the new downloader system's file page parsers.

default tag import options

Almost done with tag import options, I promise!

This is an important but slightly complicated update. Nothing huge will change, and you'll see what is going on as you use it. tl;dr is: go to network->downloaders->manage default tag import options and set something good for default for file posts if you haven't already.

This week has an important update to how tag import options (TIO) typically work. First, the old TIO options under options->importing are now deprecated–they are no longer referred to by any downloader, and I will be removing the ui entirely in a couple of weeks. All downloaders now inherit from the newer network->downloaders->manage default tag import options, which operates in the new download system. Please make sure your default tag import options are set up how you want, as any new downloader or sub now inherits from this new location! You will get a popup about this on update as a reminder.

Second, the TIOs for new downloaders and subscriptions now start in a 'default' mode, meaning 'I do not have any specific options, just use whatever is set in the defaults at the time of import'. In the TIO edit ui, this is a new checkbox that greys out the rest of the panel. This means that if you change the new defaults under network->downloaders->manage default tag import options, every 'default' TIO across the program will inherit those changes!

This new 'default' mode makes the job of managing TIOs for dozens or hundreds of downloaders and subscriptions much easier, particularly now that TIO have become more powerful and complicated. Rather than, let's say, adding the tag 'vore' to twenty different subscriptions' blacklists, you can now just add it to the defaults and all your 'default'-set subscriptions will work with that updated default when they next run. But if you have a particular subscription or downloader that needs a special blacklist or namespace filtering, you can still set a specific TIO and make it happen.

Your existing downloaders and subscriptions will keep their specific TIOs on update. They will keep parsing tags like they did before.

But when you are comfortable with this new system, you may want to set many if not all of your subscriptions to default TIO, so I have added a button to manage subscriptions that sets TIO to all selected subs! Furthermore, any normal 'tag import options' button now has a right-click menu that lets you copy/paste the TIO to/from clipboard, so if you do have some clever tag filter or blacklist you want to spam to several places, you can now do it with just a few clicks.

Also, the 'url downloader' now has its own TIO. This also starts 'default', which is basically what it was doing before–referring to network->downloaders->manage default tag import options based on import URL right when it imported–but now you can overrule this 'refer to defaults' mode and set a particular set of rules if you are doing some clever one-time job with the URLs you are giving it.

more parsers

In the new page downloader, if a site does not yet have a new parser, it falls back on the old system. This left me the job of filling in the blanks for all the default sites that come with a fresh install of hydrus. I have been adding new ones here and there, but this week I pulled out the stops and finished them all off. There are new parsers for furry.booru.org, hentai foundry, moebooru (konachan, yande.ro, sakugabooru), shimmie (rule34.paheal, rule34hentai, mishimmie), newgrounds, sankaku (thanks to some users for contributing this one), and tumblr. All these sites now support Post URL drag-and-drop import (and sakugabooru doesn't actually have a search by default, so this is likely the only way you can take advantage of it for now) and have improved file parsing support and can sometimes grab a source time and so on.

So, by default, a new hydrus client now deals with file pages entirely on the new download system. This frees me up to delete more and more of the old legacy system in the coming weeks, shifting everything over to the newer and neater code. I expect I'll throw in some other new parsers for imgur and so on as well!

There remains the issue of custom boorus. Any user who added a custom booru in the old system either by downloading a yaml file from somewhere or making it themselves has something that is a bit tricky to convert to the new system automatically. I made this new system in part so I wouldn't be needed to make new downloaders, so I don't want to go down a rabbit hole of writing a hundred new parsers, but I think I'll figure out a simple way to update the old custom booru objects into barebones new parsers in a coming update. I will probably do it all in one go, when the gallery and 'searcher' object work are done, but it may need a bit of extra work on your end to connect things together. If you care about this stuff, please watch this space. I will be working with the users who are interested in making new parsers, and we'll figure out a variety of 'make it work now in a difficult technical way' to 'I can wait a couple months for the single drag-and-drop solution' methods so you can get derpibooru and the other unusual but desired boorus working properly.

Btw, the new tumblr parser does not work with the in-client tumblr downloader yet, but it will in a couple of weeks when the gallery stuff is finishing up. But if you want to try it out, just try drag-and-dropping some tumblr post URLs on the client in v314. As well as doing the difficult 'raw' URL translation correctly (and not doing it for the pre-2013 URLs for which the translation is invalid), it now figures out a creator: tag, including swapping out for the original root creator on reblog posts! It is almost as complicated as the pixiv parser, and I am really pleased that the new download system is flexible enough to have supported what I needed it to do.

misc

There are some other quality-of-life options and improvements this week:

options->speed and memory now has 'cache timeout' options for thumbnails and rendered images.

options->downloading now has 'hide/show the 'D' on short import summaries', the default being hide.

options->gui now has a 'only autosave last session during idle time', which is useful if you run a really heavy client (like 200k+ files/import objects) that you leave on all the time and are getting regular 10s+ lag from the session autosaving. I still recommend you try if you can to keep your client below 50k at once, but if you can't, try this out and let me know how it goes.

The network menu is now categorised into something better than 'a giant single mess'.

Edit subscription panels now has 'copy queries', which will copy all the selected subscription queries' search texts to your clipboard. This is basically the opposite of the 'paste queries' button on the same panel, and makes it easy to copy twenty queries to a new subscription or chat window or whatever.

full list

- tag import options can now be set to 'default', meaning 'use whatever the default is at the time of import', which will be an easier way of managing TIOs for many subs that you'd prefer all share the same TIO settings anyway

- updated tag import options ui to enable this default setting where appropriate

- updated the newer import pipeline to work with 'default'-set tag import options

- new downloaders, subscriptions, watchers, and multi-watchers now start with 'default' tag import options

- deleted the old default tag import options management code and put some text up on options->importing making notice about the impending shift. a popup message will also say this on update

- tag import options buttons now have a right-click menu with copy/paste/default options for quick assignment and duplication!

- added 'overwrite tag import options' to manage subscriptions–once you are comfortable with the new 'default' TIO mode, and after some small tests, you might want to switch all your subs over to 'default'

- the 'urls' downloader now has a tag import options–it initialises as 'default'

- added furry.booru.org to gelbooru 0.2.0 parser

- wrote a hentai foundry file page parser

- wrote a moebooru file page parser (this works for konachan, yande.re, and sakugabooru)

- wrote a shimmie parser (this works for rule34.paheal, rule34hentai, and mishimmie)

- wrote a newgrounds parser

- integrated the user-created sankaku parser

- wrote a tumblr parser that handles photo, photoset, and video posts, auto-converts to 'raw' urls for those post-2012 urls that can handle it, figures out a creator tag (reverting to the reblog root if it is a reblog post!), and cooks you breakfast

- (hence all these above sites now support drag and drop!)

- rolled out some new tumblr url classes to handle all this.

- added sakugabooru url classes

- fixed an issue where url classes were not normalising api urls in all cases, meaning some url classes would not api-link correctly in 'manage url class links' panel

- fixed an issue with deviant art legacy gallery parser pulling some funky 'creator:' tags

- some misc new downloader error handling improvements

- the watcher now uses the new gallery object to parse and generate file import objects

- the downloaders with gallery logs should now report non-success gallery fetches, along with error tracebacks (this will include some no-worry 404s the legacy downloader sometimes uses to terminate searches)

- added image and thumbnail cache timeout time delta buttons to options->speed and memory

- added a 'show the D on short file import summaries' checkbox to options->downloading–it defaults to off

- the 'I' on short file import summaries is now 'Ig' to clear up 1/I confusion

- added 'copy queries' to the edit subscription panel, which lets you copy all the selected queries' search texts to clipboard, newline separated

- added a checbox to options->gui that commands 'last session' only be autosaved during idle time. this is useful if you usually have a huge (200k+ file) session and your client is always on

- fixed file import status button right-click, which I messed up somehow last week with the 'retry ignored' add

- shook up and collapsed the network menu into neater categories

- tightened-up the rarely used pre-parsing conversion panel on the edit page parser panel to just a button with a bit of explaining text

- if database errors include the word 'malformed', the client now throws a little extra error text pointing people to the help.txt in the db dir

- cleared out some legacy download code

- cleared out legacy hard drive import error handling, moving it all to the new file import object

- misc refactoring and cleanup

next week

I moved the new gallery watcher forward as well, this week. The new gallery pipeline is mostly in place, and thread watchers now use it, so now I have to re-jigger the regular 'walk through gallery page' downloader code around a bit and then I can blitz a bunch of new gallery parsers (which are thankfully very simple) like I've just done the file page parsers today. So, I'd like to have gallery url drag-and-drop (i.e. you drag and drop a danbooru gallery search url onto the client, and it starts a new hydrus page for that search, either getting just that page's thumbnails, or starting a new search from page index 0, or starting a new search from the dropped page) in for next week.

____________________________

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/14/18 (Sat) 01:00:44 e0c005 No.9386

Dear dev,

I'm a beginner at Python with a little background in C/C++.

If I read your source code will I understand how to do file metadata and image/video display stuff and file similarity stuff (provided I use the same libraries you do), or will I just get eye cancer and die in agony?

How convoluted are we talking here, how hydrus get formed???

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/14/18 (Sat) 01:01:29 e0c005 No.9387

Also thank you for yet another regular commit with new features, I didn't mean to be rude, I just didn't realize how new thread was so I didn't think I would be first comment.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/14/18 (Sat) 03:37:35 4d3c80 No.9389

Now seems like a good time to add support for grabbing the rating tag to the booru parsers ;)

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/14/18 (Sat) 05:13:22 af8b62 No.9395

are there any plans for a furaffinity gallery downloader?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/14/18 (Sat) 08:04:48 727088 No.9399

File: 09be75060bc524b⋯.png (1.83 MB,3011x2130,3011:2130,Hydrus.png)

Ok, I have 89 pages, 600k some images open, probably around 500~ active thread watchers with so many more that 404'd in multi watcher, and I got something interesting happening that hasn't happened before.

so I find an artist, I like their shit, but there is no 'download off every booru' option yet, so I load up e621, tumbler, r34 1 2 and 3 and furrybooru and put the name in.

then the thumbnail's start glitching where its not displaying the correct ones for that page, along with some other weirdness I go to any other tab but those 6 and they are fine and appears to be working normally, but those 6 feel like they are in the process of crashing and cant make up their minds.

I think they regain normal function once they finish, but I have to wait for more then 1 to finish to say.

On a side note, I have been trying to think of how to cull through images, my main problem is when I see something was deleted I have to see what it was and know why.

Its the thing that stops me from doing almost anything with culling, from deleting images I want gone to actually using the duplicate filter at all, then a few updates back user notes were something you wanted use to try out.

So here is my idea, on delete it asks you for a note, and you can have a number of pre made ones, as I assume some things like 'low quality' 'image macro' and others will come up many times and you would want to use it, clicking the numbers or pressing a number button for the premade reason wont close out, but instead display text so you can confirm.

Duplicate filter would also have the ability to add its own text for various functions, and on duplicate note, a 'prefer alternate' would be appreciated as an option because looking through a cg set where there are 20-30 images that all trip a duplicate flag, I can see instances where I would label something as an 'alternative version', or get rid of the other because I 'prefer alternate' seeing that in the reasons for deletion would be nice rather then just calling it a lower quality duplicate

I think most of what i'm asking for is possible to incorporate into the program now without to much hassle as the functionality for the notes is there, the notes remain after deletion so they could show up on a download attempt, having duplicate filter add text to the note on action I dont think is difficult, the hardest part would likely be adding the text input and prefab slots to a trash/delete action and the notes column to the parts it shows.

any thoughts on this hdev?

and on the downloaders glitching, once they all finished, it was more or less business as usual.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/14/18 (Sat) 08:09:40 5ad569 No.9400

>wrote a tumblr parser that handles photo, photoset, and video posts, auto-converts to 'raw' urls for those post-2012 urls that can handle it, figures out a creator tag (reverting to the reblog root if it is a reblog post!), and cooks you breakfast

Oh golly.I'm gonna try this feature.

Thanks always Dev.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 07/14/18 (Sat) 20:45:13 8097cc No.9405

File: 2c5ee8aaea49d55⋯.webm (3.91 MB,1280x720,16:9,2c5ee8aaea49d551ba62aaf88….webm)

>>9387

>>9386

My code is not great, and I started writing hydrus when I was a python/development beginner as well, so a few areas that I have still to get around to rewriting are just ghastly.

But if you want to see some examples of how to render stuff in python, yeah, please check it out. cv2 and numpy are the shit, and ffmpeg does all my video. Check out The Client/HydrusXHandling.py files in the include folder for some stuff to just copy to your own program. ClientRendering has some really shit code, but it does mostly work for actually getting pixels on screen.

I use wxPython for my ui, but if you have no preference, I encourage you to look into Qt, just in case you like it more. I am fond of wx these days, but it has its share of issues.

Feel free to ask questions about the code and grab me on the Discord (I am on Sat afternoons) if you want to talk live.

>>9389

Yeah, the guys who are making custom ones over at:

https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/NEW%20Download%20System/Parsers

Are into the ratings tag, so you may find a fix there already to what I have rolled out.

I know some users absolutely do not want the ratings tag, so I want to hammer out the last tag import options updates to make it easy to select namespaces in the new system before I start adding it for the defaults as a matter of course.

>>9395

Yeah, absolutely. Many users have asked for this. If another user doesn't do it, I'll tuck it into an update in the nearish future, maybe as a complete downloader when the gallery and searcher stuff is done. Do you have to be logged in to see NSFW stuff on FA? If so, that part will likely have to wait for the login manager, which will come after all that.

>>9399

Thank you for this report. I am not sure why the thumbs would be showing up for the wrong page, except maybe just that the heavy client is causing the thumbnail canvas to show up a bit slowly or something? Given that they run ok for any other page, does visiting another page and returning kind of 'fix' them into displaying the right thumbs again? If they show the wrong thumbs, are those thumbs clickable, or if you click them, do they redraw themselves to the correct thumb again?

I am of two minds about adding notes to file deletion like this. I like the overall idea and would like to add support, but I am not totally keen on doing it messily. Most people would not use this workflow, so I can't put much time into it, but I don't want to serve you a fuckery-duckery solution that I half-assed.

I will make a note in my to-do to look at this this week. If I can see a simple way I can move it forward, I will.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/14/18 (Sat) 20:59:16 391c1c No.9406

I get output like this in the terminal on fresh Manjaro and Hydrus installs when trying to open videos externally with mpv or vlc and nothing happens after:

https://pastebin.com/qjLxHtVF

Opening them outside of hydrus works as usual. I was on a very old version before so this only became an issue for me now, sorry if this is old stuff. I tried it with older versions and it works in 270 but not 271 or the higher ones that I tested.

Thanks for your work.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/14/18 (Sat) 21:31:35 727088 No.9408

>>9405

>I am of two minds about adding notes to file deletion like this. I like the overall idea and would like to add support, but I am not totally keen on doing it messily. Most people would not use this workflow, so I can't put much time into it, but I don't want to serve you a fuckery-duckery solution that I half-assed.

I can see people not using it at all, which is why making it non intrusive would be necessary while also making it easy to use if someone did want to use it, hotkeyed pre filled out reasons an open text box on a delete that is ignorable and with duplicates a rubber stamp note.

now as to how you do it, I think notes is the best way to do it, because if someone makes notes on an image, I think they would like to see it when something comes through the downloader, so for them. At least to me it seems like it's the most in intrusive out of the way method that's using parts of the program already in the program.

Really for my whatever hacked together half assed method you may choose would be great so long as it stemmed around a system that isn't going anywhere, as in you make it, and regardless of how the ui may change for the function, the data is still there.

Thinking of it, for people who want it completely out of their way, there could be few option toggles for 'notation mode'

the file import I assume more people would want then just me, so split that off form delete notation.

at least that's my thoughts on the current idea I have.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/15/18 (Sun) 02:34:12 e0c005 No.9410

>>9405

I'm already familiar with Qt from using it for C++, but as cool as the XAML or whatever visual builder that makes the .ui files is, a lot of times I have to end up writing the UI code myself anyways, so since I need wrappers to use it with python last I checked, I may give wx a look as an alternative.

Thank for the info, I'll check out the handling scripts. I'm usually busy on Saturdays but if I find time I may drop in some week or other with code questions.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/15/18 (Sun) 03:11:39 4d3c80 No.9412

File: a1958d5d8325fcd⋯.png (56.95 KB,985x466,985:466,Untitled.png)

Something wrong with database migration. It's trying to move files to and from the same folder. Pic related.

I'm just trying to separate the actual files into one location while keeping the db + thumbnails in the same location. I can't figure out how to do it without encountering this error.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/16/18 (Mon) 07:10:33 727088 No.9415

>>9408

jesus christ I need to start using dragon again, even when it fucks up it makes what I want more readable than me just typing.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/16/18 (Mon) 14:03:43 af8b62 No.9416

reporting a bug: the tumblr downloader isn't picking up the tags, even with all options properly set to "get all tags".

>>9405

>Do you have to be logged in to see NSFW stuff on FA?

yes, sadly. like every other art site, they protect their asses from legality this way.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 07/18/18 (Wed) 18:18:20 8097cc No.9425

File: ee47c64520d991e⋯.png (3.62 KB,512x86,256:43,tumblr api post page parse….png)

>>9406

Thank you for this report. I've had a devil of a time getting xdg-whatever to work on some different flavours. What happens if you set an explicit launch path under options->files and trash? Does that work, or do you get a different error?

>>9408

Thanks. I have split the job up into like five pieces that I can slip in a bit every week. I'll basically make notes into multi-notes and then auto-attach a 'deleted' note in several new ways. Let me know how it works for you when I get to rolling out the fun bit.

>>9410

wx has a bunch of ways you can do it, I think, including like some popular 'ui builder' app where you are drag-and-dropping elements into a test window and it writes all the code for you, but I prefer to hand-code it all. There's a 'wx demo' that you can get alongside the base code, here:

https://extras.wxpython.org/wxPython4/extras/4.0.3/

https://extras.wxpython.org/wxPython4/extras/4.0.3/wxPython-demo-4.0.3.tar.gz

Which is the absolute shit and lets you see how different widgets go. It works well for me in doing multiplat with native controls, but it has instability issues if you do things in the wrong way and I know a lot of people who don't like it for a variety of reasons.

I've got a ton of experience with it now, so let me know if you end up liking it but want to know the right way to do sizers and layout flags or whatever. Feel free to copy any of my code or classes if they are useful.

>>9412

Thank you for this report. I am sorry you are having trouble. I gave the migrate db dialog a pass this week–can you take a screenshot of the main dialog, where it thinks everything is and is ideally supposed to be, in the new version and either post it here (with blurred out base path, if it is private) or email it to me?

I am non sure how it would come to this logical mistake, so I am keen to help you out here in the code and with a manual SQLite fix if needed.

>>9416

Thank you for this report. Is this the new tumblr downloader parser that I posted? I don't think I added the regular tags to that. If you'd like that, please try drag-and-dropping this attached png to your network->downloader definitions->manage parsers dialog and then update the tumblr link under manage url class links under the same submenu. I'll roll it into the v315 later today as well.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/18/18 (Wed) 18:23:40 9a73bb No.9426

some feedback

i looked at the program for half an hour or so, tried to do some things and failed. if you have 150 IQ and are in software design or programming its probably easy to use but for just avg IQ person like myself i found the program to be not very intuitive :(

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/18/18 (Wed) 19:39:37 9182e6 No.9427

>>9426

Did you use the help pages? Hydrus has a bit of an initial learning curb, but it functions like any booru so you should be able to do some basics.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/18/18 (Wed) 21:02:40 9a73bb No.9428

>>9405

>>9427

ok, ill study it some more

i dot know what a booru is, i just manually downloaded about 500 memes from 4chan-pol over the last year or so and then i got the program ChanThreadWatch to download some image threads. Was hoping that i could just put about 100 images in a folder and then point Hydrus to that folder and let it go online and find some matches. Looks like it will take a few hours of reading though

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 07/19/18 (Thu) 14:45:11 4d3c80 No.9436

>>9425

re: db migration

With the new version I'm not getting the issue anymore, thanks for fixing it.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.