[ / / / / / / / / / / / / / ] [ dir / random / abcu / abdl / pcal / pdfs / random / rule34 / sl / tingles ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.
Name
Email
Subject
Comment *
File
Password (Randomized for file and post deletion; you may also set your own.)
Archive
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Voice recorder Show voice recorder

(the Stop button will be clickable 5 seconds after you press Record)
Options

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 5 per post.

New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Experienced user with a bit of cash who wants to help out? ---> Patreon

Current to-do list has: 2,017 items

Current big job: Catching up on Qt, MPV, tag work, and small jobs. New poll once things have calmed down.

HookTube embed. Click on thumbnail to play.

52e266  No.8651

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v303/Hydrus.Network.303.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v303/Hydrus.Network.303.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v303/Hydrus.Network.303.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v303/Hydrus.Network.303.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v303/Hydrus.Network.303.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v303.tar.gz

I had a great week mostly working on downloader code. There are some new options for gallery downloaders and subscriptions, and some downloader stuff for advanced users to play with as well.

downloader improvements

I have updated a great deal of the old downloader's behind-the-scenes to the new system. The way the client's various downloader objects handle pre-import status testing and tag application and many other things are now simpler, cleaner, and more unified.

As is often the case in this sort of stuff, you will likely not notice much difference beyond some small bug fixes, but I made great progress on the gallery overhaul this week, and I feel good about the coming weeks.

I also made subscription merging and separation much more powerful–you can now just separate some subscriptions from a large merged sub, and you can also extract them as a single new merged sub. Handling names for the results is also much better throughout.

You can now also alter the amount of time the client waits in between individual gallery page fetches. This used to be a static 5s, but you can now edit it under options->downloading. The new defaults are 5s for subscriptions and 15s for regular gallery downloads. The feedback during the gallery parsing stage is also improved–it'll count down this new flexible delay, and there should be some better error reporting and faster responsivity when you click.

The 'raw urls' downloader is now just called the 'urls' downloader. I expect to give this downloader a lot more to do in the coming weeks.

misc

You can now set the rough maximum width of the popup messages, in character widths, under options->gui. The default is 56, but if you have large or small font, please change it and let me know how this works for you, particularly if you have had problems with popup layout in the past.

I gave the 'export files' frame another pass, brushing up its code and how it generates paths. If you are a Linux user who has had crashes trying to open this frame, please give it another go and let me know what happens.

advanced downloader improvements

Advanced users who want to write parsers for the new download system only!

I have written some basic parts of the new gallery downloader. It isn't completely ready yet, but if you want to play with it, please do and let me know how you get on.

You can now link 'post urls' to parsers! If one is linked, the client will accept it as a drag-and-dropped URL and will open a 'urls' downloader and queue it up and parse it (including tags and urls) all in the new system. The urls downloader accepts these parse-linkable URLs in its standard text/paste button input as well. There's a new system for setting 'tag import options' (based on url class) for the new engine under the network menu that I will eventually migrate all the old options to, as well.

I was pretty amazed that this came together this week. We'll need to figure out a bunch of page parsers as new defaults, but I think we should have easy drag-and-drop url support within a week or two.

There's also some new super-barebones 'what's in the network engine right now?' ui under network->review network jobs.

full list

- file post url classes can now be linked to parsers!

- the 'raw url' downloader is now just the 'url' downloader. if a dropped url is matched as a 'file post' url and links to a parser, it will download it with that parser and default tag import options

- url drag and drop now recognises file post urls that have linked parsers and will send them on to a url import page, just like 'raw' urls! they will use the new parsing system to parse tags and known urls and all that

- wrote a way for the new download system to store and edit default tag import options–this can now be found under network->manage default tag import options, although it does not do much yet. eventually the old options will be transferred here, and the different downloaders will consult it more

- refactored and cleaned up how some default tag import option calculation is done

- the fixed 5 second wait time between gallery page fetches is now editable for downloader pages and subscriptions under options->downloading. subscriptions default remains 5s, gallery default is now 15s

- the gallery downloader will present its 'x urls found so far' string in a more uniform and reactive way. it will also count down until the next page fetch

- subscriptions will now similarly react quicker while the gallery parse step is ongoing, and it will count down to the next page fetch

- the manage subscriptions dialog will now note in BIG RED TEXT if subscriptions are currently globally paused

- subscriptions now consume less parallel-timeslot overhead, meaning they sync much faster when they have no work to do (which is most of the time)

- separating subscriptions is now more sophisticated–if you wish, you can now choose just to separate a subset of the large subs's queries, and if you do so, you can decide whether to merge the extracted subs into a new larger sub or just have them as individuals. also, you will be asked for name/base_name for what to name the new subs

- since separating subs is more complicated, the button is only enabled when a single large sub is selected. please do your separating one sub at a time!

- merging subs will now ask if you want to rename the merged subs as it goes

- refactored a giant heap of duplicate import code into single locations in the new file import object–things like applying tags and generating pre-import status are all now done in one place. a number of weird behaviours (like not applying known url associations in certain circumstances) should now be a lot better and unified

- the way the db and other import testers handle whether a pending import is new or unknown is simplified

- optimised the way the importers figure out whether to display a new import

- the 'has the client seen this url before?' test used to figure out whether to download a file now explicitly ignores anything it isn't certain is a single-file File URL or Post URL (according to current url classes). associating 'gallery' known urls is no longer catastrophic to this system

- the simple downloader now associates the entered url with the files subsequently imported

- file import objects working in the new parsing system are aware of the new 'should this url be associated' option and will be given gallery urls to hold on to as appropriate

- url classes now have an explicit option as to whether they should be associated with files imported from them, which is not appropriate for dynamic CDN File URLs but is appropriate for multi-image tweet links (which will likely be Gallery URLs in the new system), for instance

- general importer code cleanup across the board

- threw together some early ui to show current jobs in the network engine under network->review network jobs

- popup messages will now try to set a max width of roughly 56 characters wide rather than 400 pixels. this value is editable under options->gui

- updated the listctrl in the export files frame and the way this frame generates and stores export filenames. let's see if linux users who had crashes with this have better luck

- refactored stuff out of the bloated ClientImporting file into seperate new files

- refactored client network bandwidth and session code out to their own files as well

- refactored client network objects to its own file

- cleaned up some open externally debug code linked to callto report mode

- a common grid layout used across the program will copy control tooltips to the string on the left

- removed the 'downloader' network context type, which was never fully introduced and wasn't turning out to offer much in the new bandwidth tracking system

next week

I want to keep going on this downloader work and get this week's advanced stuff ironed out and working by default.

I was crushed for time this week and couldn't get to some smaller issues, so I will put some time aside and make sure I catch up on those as well.

____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

52e266  No.8652

File: ec27eb489b75fa4⋯.png (2.44 KB, 512x84, 128:21, e621 file page parser.png)

File: 636ef21fc2edecd⋯.png (2.56 KB, 512x85, 512:85, safebooru file page parser.png)

Here's some test parsers I threw together this week, for advanced users who want to play with the new stuff. I can't guarantee they are perfect, but I think they'll do the file url and tags. I expect a whole bunch more of these to be floating about in the coming weeks.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

5229d3  No.8661

Aaaw, didn't have time for >>8600 ? Oh well, next week maybe? Thanks

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

5229d3  No.8667

I was wondering about the security of Hydrus' network traffic. Are the downloaders/subscriptions using https when possible? Is the traffic to/from the PTR encrypted?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

df48d0  No.8670

>>8652

how do you use these?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

52e266  No.8673

File: 0f7c68db84326c8⋯.jpg (996.66 KB, 1600x1073, 1600:1073, 0f7c68db84326c82b747370027….jpg)

>>8661

Yeah, I looked at it but couldn't slip it in quick enough. I'll have to expand the shortcut action object to hold service identifiers, which will take a little more work. Plus a little more ui for it. Hopefully have it going for this week.

>>8667

Generally, everything is https by default. I think a couple of downloaders aren't because the actual boorus don't have https support (EDIT: I just checked–furry.booru.org is still like this, but it looks like xbooru have https now, so I'll update them as I move to the new download system). Hydrus won't force https, so if you create a new booru with http links, it'll just follow whatever the given html points it to, which is typically more http.

There will be better tools for this in this new download system, which has new URL Classes that can 'normalise' dropped URLs into a pretty state that also has a preferred scheme, which I am setting as https wherever possible as I create them.

Hydrus network repositories work in https, but they self-sign their own certificates (it is free to self-sign, and as repos work outside of the DNS system, they can't acquire signed certificates anyway). This means their traffic is completely encrypted to all outside observers, but they are potentially vulnerable to MitM attacks from a specific attacker along the chain, i.e. ISP/government. This self-signing isn't ideal, but it has been an ok compromise that delivers good general privacy with technical simplicity so far (especially as Cloudflare and AWS do similar fuckery with big-boy https in any case). Repositories forget all IPs as soon as the connection is done.

Next time I do some network work, I expect to write some certificate management ui to do certificate review and some kind of 'setting up connection to this repo' workflow, or maybe even do a pseudo-certificate authority inside the network to get validation, but I am still thinking about it. This stuff is technically complicated both for me to perfect and for users to interface with, and most of the attackers it attempts to deter can already hack everything and fuck everyone on a whim, so it could just be wasted work. Still, I expect to move the whole network towards more P2P stuff–mostly 'four friends sharing some files in a private channel one of them runs'–in the coming years, so some ui/workflow for peer trust management is definitely coming at some point.

I would be interested in your thoughts on this, if you have experience with it.

>>8670

There will be some 'just do it' easy ui for this by the time the downloader is done, but for now you can import these pngs under network->manage parsers. You can then link them to the appropriate URL Classes under network->manage url class links, and then drag-and-drop will work in the client.

I expect to add a bunch of this new parsing functionality over the coming weeks. I will auto-add it on update.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

df26ec  No.8675

>>8673

If you control the certificate, could you post your version of the SHA hash of it on 8ch so it can be verified by a means other then hoping the government/ISP doesn't MITM it?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

df26ec  No.8677

>>8673

If you are going to insert P2P capabilities, why not just piggy back off of go-IPFS so that it just werkz (TM) and is a easier maintenance burden on you? IPFS seems to be doing a great job already and it would be perfect for those four guys just sharing files to send a hash of the file and download it over ipfs, and then have hydrus tag everything.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

e1d20a  No.8680

>pause the update processing part way through

>let it commit

>start the processing again afterwards

>it finishes almost immediately even though it usually takes ages to process

This doesn't seem right

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

5229d3  No.8681

>>8673

Under manage boorus, several sites still has http addresses while they appear to support https from checking in my browser. Gelbooru for example

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

e6a1bf  No.8682

>>8680

If you have a hard drive they often have very fast write caches, meaning hydrus and the db can write to you disk very vast for a couple seconds until the often only 256 MB cache fills up and your drive get's limited by it's speed again.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

6203c0  No.8686

Have you considered adding the ability to import, tag and categorize todo list items? :^)

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

6203c0  No.8687

>>8677

On a more serious note, I second this. I haven't even gotten into IPFS yet, but if it were implemented in Hydrus, I would.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

eec81a  No.8688

>>8687

>>8677

Hydrus already supports IPFS, but using it doubles files, because it doesn't use filestore or whatever ipfs uses when you type IPFS add –nocopy /path/to/file in your terminal. You also have to got external IPFS node to which Hydrus can connect via API. But there is nothing you can do with that, because as far as I know no one is working on python implementation of IPFS right now.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

238db7  No.8689

So, I just returned to hydrus after about a year dormant and updated, so far so good.

I just wanted to ask, do we have the ability to put login information into subscriptions yet? Because I have all of these tumblr subscriptions (which I can apparently combine multiple blogs into one subscription now, neat feature), but last I knew they were all failing because I needed to make an account.

Well, I'm back, and I made a Tumblr account, but I don't know where to plug it in. Anyone know?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

52e266  No.8693

File: 365c58a04d6e0c7⋯.jpg (612.85 KB, 1280x997, 1280:997, 365c58a04d6e0c7054d2c913a6….jpg)

>>8675

Sure, it should be:

sha256

D9:19:E8:44:65:C8:E8:B9:DF:19:7C:C3:A7:3A:E9:26:07:DD:47:A7:04:8B:AC:7C:76:D3:03:51:B9:78:0B:DA

sha1

27:6A:38:10:84:76:16:5E:EC:7C:CC:3E:79:72:33:C2:90:47:3B:E3

You can test it in your browser by going to https://hydrus.no-ip.org:45871/ , which should give you a welcome page after you add the security exception.

There's no clientside infrastructure that stores and validates against this yet, but since we are on the new network engine and now continuing with downloader work, maybe it would be a good time to do something simple, just so it can be reviewed from inside the client (and so the client can throw an exception if the cert changes).

Since most users who are talking to a repo are talking to my PTR, I figure it'll be sensible at that point to bundle the client install with the cert to begin with.

>>8677

>>8687

Yeah, I would prefer to use existing technologies over reinventing the p2p wheel again. I don't know enough about IPFS specifically to say confidently if it does privacy in the way we would want for four guys just sharing files, but I have been impressed at how well it does the things it does do. All this p2p stuff would be way off, so I haven't put much time into thinking about it properly.

>>8681

I think this depends on when an individual user started using hydrus and hence which updates they caught and then how neatly those updates applied. A lot of boorus only had http when I first added them, so if you are a long time user, you might not have caught the https update as intended. If you want to double-check, try hitting the 'restore from defaults' button on that dialog, which should force-apply the the newer defaults' https versions where it can.

As I switch over to this new downloader engine with its new URL Classes, everyone will automatically inherit its better solutions, and we'll finally be able to ditch the old hackery dackery doo shit. In fact, now I think of it, the work I did this week should actually normalise all new gelbooru post urls (and anything else with matching URL Classes) on any import context using the new system automatically, which will include the https preference, despite most of the downloader still using the old system.

tl;dr: I hope to have this all working on better systems that'll force update in the coming weeks.

>>8680

>>8682

Another bottleneck I frequently encounter is just sheer access latency during random access (which happens when writing the changes in a commit journal back to the master db file). Whenever you repeat a job like this quickly, the second time around go doesn't have to wait for your OS to fetch the most common db pages to figure out what to write back, so you save a ton of time and can just spam synced pages straight from memory right back without waiting.

But there is also some SQLite weirdness I often just don't understand. It seems to do maintenance/cleanup sometimes on its internal stuff during commits sometimes. I generally recommend leaving all big jobs to idle time rather than forcing them manually, just to let your OS, hard drive, and SQLite figure it out on their own terms.

>>8686

😐︎🔫

>>8688

Next time I work on IPFS, I'd like to add the -nocopy thing. I tried with the current system, but unfortunately, when I last checked, I can't do it with the http API since the 'addfile' (or whatever command it is) won't take paths, even when both caller and ipfs daemon are on the same machine–I can only POST the file with a multipart form, wew. So I'll ultimately add the ability to talk to the exe directly, which does support -nocopy and a path.

>>8689

Not yet, but we are on the home stretch on this now. I am on the verge of updating the tumblr downloader and all the others to the new parsing system, and then I'll be focusing on the login manager. I've already got several pieces already in place–I just need to pin down a new login script object that'll do parsing and cookie-juggling.

However, you might like to test one of your tumblr nsfw artists in the client tumblr downloader right now in any case, because afaik the API I use actually doesn't need the login to view nsfw posts. Nor do the 'raw' urls the tumblr downloader now picks up, like so:

https://data.tumblr.com/7c06451405435ce48d2c14cfaa9c6e20/tumblr_p7hjnoznll1wwh8exo1_raw.png

That'll need a security exception (or just change to http) due to AWS gubbins, but you should be able to view it on a non-logged-in browser.

I don't remember the situation a year ago or know the full details here now–if you find a nsfw tumblr blog that doesn't work with the downloader, I'd be interested–but if they do work for you in hydrus atm, you might like to try turning your subs on and seeing if they work.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

81bb0c  No.8696

>>8693

Thanks, I tried it and Tumblr seems to be working fine, but I'll tell you if I find any exceptions that aren't just deleted blogs.

The subscriptions panel does take a bit to pop up, but that's probably just because it has to populate 200ish subs. I think I'm going to leave them separate for now since they all get tagged slightly differently.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

5229d3  No.8699

>>8693

>I think this depends on when an individual user started using hydrus and hence which updates they caught and then how neatly those updates applied

Yeah my db was created 2 years ago, v190-something. Restoring default fixed them. Thanks

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.


[Return][Go to top][Catalog][Nerve Center][Random][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / / / / / ] [ dir / random / abcu / abdl / pcal / pdfs / random / rule34 / sl / tingles ]