/hydrus/ - Hydrus Network

I appreciate your feedback, thank you. I'm sorry that hydrus hasn't worked out great for you.

You can upgrade quickly by extracting a release on top of a previous install. When the new client boots, it will see the old database at install_dir/db, notice it is an old version, and make any updates it has to. You don't have to rebuild from scratch every time. I've written more about this here:

http://hydrusnetwork.github.io/hydrus/help/getting_started_installing.html

As for your thoughts on growth of the project, hydrus is certainly not huge, but I am nonetheless pleased and content with the current direction, and it isn't in my roadmap to go for lots of users or lots of money. I'm amazed that anyone wants to use it, and when people say they love it, it easily gives me enough of a boost to be satisfied with my work. I prefer for the growth to be driven by word of mouth, and I'd rather deliver regular honest code to enthusiastic users than desperately try to keep up with overly ambitious Patreon promises.

Although things are held together by duct tape in many places, and the user interface is often ugly and debug-tier, I think these things are improving over time. I've been pleased by the recent window sizing system, for instance, and the new disk cache has eliminated a great deal of the lag when running off an hdd rather than an ssd. Furthermore, I really prefer the independence and drama-free nature of working on my own. I've never been an open source/github or generally social person, and am not interested in working on this in a team.

So, I thank you for the suggestion, but I'm content to keep my current schedule. I apologise if that is disappointing. I really enjoy working on hydrus, and I don't want that to change.

>>3205

thanks for replying. i really wanted to write a proper reply yesterday but i got busy irl, now i'm too sleepy but i will write back this evening. will try to install the today's latest build on windows before posting.

noticed plenty of talk on /g/ today too: > https://boards.4chan.org/g/thread/55663442

>>3212

thanks as well, will write a proper response tonight 4 u. ;^)

>>3215

>>3205

OP here again. here's an update on my progress using this 'app';

>moved old db folder away from hydrus folder

>installed v215 over old version, on separate hdd from windows, using wizard. ran fine on empty.

>move new, 'empty' db folder up, replaced with old db folder, ran test.exe and client.exe

>program presented error, saying to old db was too old to try v207 first.

>move old db folder up again, remove hydrus folder, replace with v207 from wizard (old thread on 8ch)

>move down db folder and replace 'empty' db, run test.exe and client.exe

>program (v207) runs splash, claims to update datebase for a while, then present python ui error of some kind, try relaunching program, same python error but no status about updating database anymore.

>give up

>move old db folder up, clear everything inside except client_files, defrag drive.

>install v215 fresh, launch it, tell it to import files from client_files (as i've done before as mentioned in thread)

>it's done importing (and deleting old copies, which actually moves to windows trash, very annoying to have to purge it often b/c windows is on another hdd and it's thousands of files), setup repos using help menu.

>download public files very quickly, public tags want to download only on close (maintenance script), try this a few times during the day with little success.

>next day, notice the 'force idle mode' in debug, try that, tags seem to DL fine, but 'review services' shows incomplete download of repo ofc.

>happen to need to place an order on amazon for some junk for someone, order an ssd for cheap for myself (i need to reinstall ubuntu since v16.10 broke AMD drivers, might as well put v14.04 on an ssd for boot or w/e, wew lad).

>ssd arrives today, it's only 60gb (says 64), partition drive on ubuntu for half ext4, half ntfs, boot into windows 8.1, mounts fine.

>move hydrus folder to ssd (froget to defrag hdd first), takes a half an hour to move folder (~10gb of data, database is huge, client files is 6gb but that has thumbs in it, so probly on 3gb of actual content). edit windows shortcuts to route to new drive.

>defrag only ssd (lolwut), done in less than a min, fastest defrag ever, try 'optimize ssd' option in asslogics defragger, wew lad, takes 30 minutes.

>finally launch hydrus app, balance files (instantly), close for maintenance, runs public repo downloads and such doing 15-25000 writes/second, launch app a few more times try idle mode, etc, no changes, check services, repo is done, pick related.

>>3282

>tag stats are twice better than before upgrading app and full repo download.

>system:everything [13,430 files]

>system:number of tags > 0 [1,344 files]

>system:number of tags = 0 [12,086 files]

while much of the untagged content is silly reaction faces which aren't really art, there are still hundreds/thousands of high-pics and anime stuff. i don't expect such a small project to have original content from my machine tagged, but the output of tags and untagged are very similar looking mix of content that was originally downloaded from community sources (imageboards, ftp, dumps, etc).

the program is 'usable' at this point but the GUI still takes many many seconds to load content when a tag is requested, even with the ssd backend, it's noticeable faster than when i was using it on the hdd, that would take minutes. this said even though searching content can be sluggish, once a selection is ready the gallery mode and scrolling the thumbnails work fine. the app is still prone to become unresponsive when doing some functions, even ones without warning about gui stalling.

with the build coming out tomorrow i will have a chance to mess up this version or w/e using the wizard or 7zip. so that's good i guess. ideally i would like to increase the tag repos as much as possible or practice a way of doing a blitz tag with autocomplete typing manual, much of the content is similar to images that have been tagged. otherwise i would like to export/move those untagged images out of the program into root folder on the SSD so i can try to manage them by hand/deprecated picasa app and slowly add/tag them in hydrus as needed.

this whole exercise has reminded me how much junk chantard images i have on the machine but tbh i downloaded large amounts from private ftp servers and such so i could test the tagging system in hydrus, likely only had a couple thousand images in my own collection, but that was years ago. the same applies to much data on hdds and such but i don't make efforts to organize it more than removing large files and creating backups, etc. it's all very tiresome to deal with 'big data'.

could the UI be improved if done as local booru-style web UI and a local webserver backend and javascript frontend? surely the browser would save work rending images and caching? are there any other projects you know of that are like that? i know it's easy to speculate but surely there web browser can function to assist, it has full screen functions and multi-instances, thinking like IPFS how does it's local server UI, regardless of the data gateway ofc.

tl;dr how to increase tag repos, slim down database/export images not tagged in bulk, and generally improve performance.

>>3212

oh also wanted to reply to your points to clarify.

>active users

simply taking note of the 8ch traffic at the time, only used the meter to speak to the mount of activity on the board, did not mean to implying i thought only that many people used the app. today it has 17 for example. 170% growth!

>public tag repo

i am aware there's no AI magic to it, it's based on the images that people have tagged and shared, therefore like any MLM scheme it follows that the more people use the program and contribute tags the better the tagging system will get because of a larger sample. ofc there may be bias in who uses it and therefore what type of content they tagged but again, the more diverse the users the less chance of that kind of bias manifesting in content tagging. you put it well in your last line, personally i haven't tagged anything since until today the program was harder to operate.

>centralization

if there is any other repos being hosted online, how can they be found/shared? perhaps a wiki could be setup on the github for these kind of notes,otherwise general threads would do on this board. as for ipfs, i've used the software before and was very impressed but it is also cryptic and hard at times, even after watching the devs talks on YT, mostly the troubles manifest when trying to use it backwards for http(s) hosting since there is no browser extensions (though talk of them), so it's not as easy for nerds like mega or w/e. you can see posts about it here: >>>/tech/499795

>'guerilla marketing'

it's not shady in the least if done tasteful and interactively. there is mention of the project in the ipfs thread linked above ( >>>/tech/500911 ) , and also you can see in my OP the search engines show discussions.

>If people want to use software like this they should be able to find it, there's a lot of presence online for search engines to pick up on.

that's ridiculous and you should know better, the only people finding this project are chan users via word of mouth (aka shill/shitposting). as you said it is mention when questions are asked in certain communities but this is not a given it only works when a hydrus user is present to make such a suggestion and choose to. does the word 'viral' mean anything m8?

if the dev posts in a 'programming general' once a week when there is a new build is subtle enough to keep a presence in that arena but not enough to be considered spam. if anything it could have it's own general thread if the posting in the other general thread became popular. thinking long term ofc.

you do make the point that this board itself is already very interactive but one day/session a week for community outreach is so demanding compared to monitoring this board. a weekly hydrus general on 4ch's /g/ is not much to ask in the long run, even if only for a few hours before it 404s. much of the obscurity comes from the 4ch/8ch divide, which is silly imo.

>developers / patches

>More importantly more users does not inherently mean more contributions.

this is true but the chances of a user being interested in the source code is tangible. it's not linear but it's probabilistic, esp if people are invited to help. but as you can see in >>3205 the dev is not currently require assistance. i only think in this direction because of the nature of the content in the database and positioning of the project/meme inside the imageboard community as a whole.

i don't have magical faith in FOSS, i've only seen projects grow and sustain in the past when there is efficient communication and motivation from small groups of users/developers working together, it usually follows the '1% rule' (pic related, see wikipedia). this ratio shows that of any group of people who are potential able to assist in a project only 10% will be motivated to do so.

therefore increasing the general userbase would increase the chances of contributions, at least in principle. ofc that only works when barriers to entry are low, but in the FOSS world that is a given, those who can code will code, when motivated/inspired. as you say in your line.

>>3205

@ dev, perhaps telemetry/metrics on how many requests are made to public repo tag/file server in X time period, how many instances in the wild, version counts, etc (within reason ofc, opt-in on next update, etc), the patron/IPO comment was joke, meant to complement the fact you get any donations at all, it's a good sign.

tl;dr organic growth is easier when the seeds are watered and spread in the wind.

>>3282

>>3283

>>3284

Thanks for this follow-up. I'm sorry I could only get around to it now.

I'm also sorry you had a bad experience messing around with db versions and slow hdd load times.

I think you make good points. I'm not 100% confident on what the project actually is, nor where it is going, so I will keep your thoughts in mind. Like you said about realising how many junk files you have, I think we don't yet have a good understanding of what 4.4 million file hashes means. I'm increasingly of the thought that direct file/tag mappings are going to be infeasible surprisingly quickly. I never thought we'd be able to have comprehensive tag coverage of all Anonymous images, but I am also surprised at how many files there truly are out there, and how many resizes and so on. I expect to slowly move the program towards a more decentralised network and more automatic tagging systems, with clients sharing tag descriptions (this shape might be "ribbon"-type metadata) rather than direct tag mappings.

I've also wondered, lately, if we'll soonish see something of a collapse of some sectors of new content generation. There's so much stuff already digitised and easily accessible, and people's backlogs/to-watch-lists/hydrus-inboxes are all growing faster than they can be cleared out, I figure non-topical media deflation can only be around the corner. If the rate of new manga being made reduces one day because there is already so much good old manga to read, does that change hydrus stuff? I've been going over some of my older hydrus client's files recently, and I can't remember most of them. A lot of them have the same value to me as new files, except they were already on my drive. I wonder if good search and management of existing collections will be more important than good gallery downloaders in future.

I don't know what to do about the bits of ugly ui and delays on high CPU time, at least beyond slowly improving things. The project was originally going to be browser-based, funnily enough, but javascript couldn't do all that I wanted. I think I'd like to continue having hydrus as a wx application, if for no other reason than it would be a tremendous effort to rewrite for a new graphics library. That said, writing an http api for the client remains a long-term goal so someone can write an external searcher/viewer for web or android or whatever.

I'm not sure about gathering any sorts of metrics–it just sets off privacy alarm bells in my head. Better I think to have completely private clients and let users tell me their info through other channels if I need it. Maybe I could throw up some polls just for fun, although the main bottleneck in my dev process is finding time and energy, not that I don't know what I want to do.

Anyway, I can't promise anything will be very polished any time soon, but I hope every new version will generally be a little better than the one before. I'm still trying to figure things out on my end.