/hydrus/ - Hydrus Network

>>10555

What the actual fug…. I went from version 311 -→ 328 and there were NO ISSUES.

Am I in a dream? Have I killed myself without noticing and am about too get it snatched away and find myself in hell?

Was I hit by a bus and am in heaven?

Bless you guys, I am a happy camper right now.

>>10520

>>10512

Hey, I checked this out today but could not reproduce it. The max whitespace on the bottom of a page should not be larger than a thumbnail's height, so I think the problem here is actually that your scrollbars are too tall and so when you scroll down, you are seeing 'invalid' space that isn't being pasted over with white because the gui doesn't think it exists.

I messed around with sizes on my end but couldn't get it–could you please take a screenshot of your whole media window here so I can replicate your pixel sizes exactly? I presume that when you are at 0 margin, your thumbnail height is accidentally like exactly 3 times your media height and there's a +1 I am missing somewhere in a 'should there be a new row of thumbs' check.

Also, if you change the size of your window by a few pixels, does it draw the scrollbars correct, or do you always get too much whitespace as the end?

>>10554

>>10553

>>10555

>>10557

Great, I am glad this worked. In truth, I haven't made many significant db changes in a while due to all this downloader work–it has mostly been smaller object updates, which for me are less subject to bit rot.

Let me know if you run into any problems. Things like manage subscriptions are probably in a new location for you and a bit more powerful and complicated. Try dragging and dropping an image tweet URL like this–

https://twitter.com/suzybaeily/status/1059168488244342785

–on your client!

is there a way to view your library on an android device via streaming?

>>10734

Not yet. Vote for API here >>10654 .

What version are you using?…

version 310 to 330, but I haven't actually kept up with new features for far longer than that.

Here's the masterlist:

https://hydrusnetwork.github.io/hydrus/help/changelog.html

You can search by 'version x'. Each changelog is generally formatted in the order:

Stuff most users care about (new features)

Stuff individual users care about (bug fixes for some dude)

Stuff only I care about (code cleanup)

So scan the top third or so of the lists and you should get a decent idea of what's happened! v310-> has mostly been downloader and login stuff.

>>6333

Cool trips.

Traceback (most recent call last):
  File "/usr/local/bin/i2vec-cli", line 11, in <module>
    load_entry_point('i2vec-cli==0.2.0', 'console_scripts', 'i2vec-cli')()
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 561, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 2649, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 2303, in load
    return self.resolve()
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 2309, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.5/dist-packages/i2vec_cli/__main__.py", line 20, in <module>
    import peewee
ImportError: No module named 'peewee'

>>6428

running pip install peewee seems to have fixed the "tags not being read from the page" problem. Consider all of these bug reports fixed.

>>6332

>>6295

>>6180

>>6175

>>6165

>>6157

>>6149

>>6137

>>6130

I'm gonna start using this a lot now, thanks.

Necrobumping for great justice

>3 years ago

Oh fuck.

I used to want to be able to add any generic file the most, but out of using hydrus daily what I want the most now is to be able to force-check a picture that failed to import properly with all the tags it should have. Automatic tagging is promising, but ultimately unnecessary for me since I just scrape files most of the time. If anything id use autotagging as a backup system to a failed tag import

The ability to tag images with consecutive numbers outside of the import files dialog. It would make tagging comics/doujinshi downloaded using the downloaders/watchers much, much, easier.

I am likely to make the poll today, with the release. I may unsticky and lock this thread to move convo over there, but I am not sure.

>>10622

That mkv jpg solution looks great! Thank you for figuring out the variable frame rate stuff and putting it a script together. I have copied this to my ugoira notes for when I get to this.

>>10623

Let me know if I misunderstand here, but you can probably do this now by running the problem file through a program like HxD to figure out its sha256 hash and then searching in hydrus in 'all known files'/'public tag repo' search domain using system:hash=abcd… . That said, if a file cannot import to hydrus, it likely doesn't have any tags in hydrus–or do you mean like 'what tags it has on the site I meant to get it from'?

In either case, I'd be interested in examples of files that look fine but won't import. Please feel free to submit the files themselves or URLs to them!

>>10639

Thanks–I put this on my 'see if you can sneak this in' list a little while ago, and it just didn't happen. I am adding it to the list here as "Add an incremental number tagging dialog for thumbnails".

>>10648

I greatly prefer doing transformations like this with our own CPU/GPU cycles, so I would likely start such a system by talking to local executables and then extend it to work with http POST queries depending on demand.

>>10650

The poll is up! Please go >>10654 to vote!

>>10640

>>10642

Ok going to make a sub that is specifically this

first_seen_at.gt:3 days ago,upvotes.gte:150,upvotes.lte:153

first_seen_at.gt:3 days ago,upvotes.gte:154,upvotes.lte:156

first_seen_at.gt:3 days ago,upvotes.gte:157,upvotes.lte:159

first_seen_at.gt:3 days ago,upvotes.gte:160,upvotes.lte:162

first_seen_at.gt:3 days ago,upvotes.gte:163,upvotes.lte:165

first_seen_at.gt:3 days ago,upvotes.gte:166,upvotes.lte:169

first_seen_at.gt:3 days ago,upvotes.gte:170,upvotes.lte:172

first_seen_at.gt:3 days ago,upvotes.gte:173,upvotes.lte:175

first_seen_at.gt:3 days ago,upvotes.gte:176,upvotes.lte:180

first_seen_at.gt:3 days ago,upvotes.gte:181,upvotes.lte:185

first_seen_at.gt:3 days ago,upvotes.gte:186,upvotes.lte:190

first_seen_at.gt:3 days ago,upvotes.gte:191,upvotes.lte:195

first_seen_at.gt:3 days ago,upvotes.gte:196,upvotes.lte:200

first_seen_at.gt:3 days ago,upvotes.gte:201,upvotes.lte:205

first_seen_at.gt:3 days ago,upvotes.gte:206,upvotes.lte:215

first_seen_at.gt:3 days ago,upvotes.gte:216,upvotes.lte:225

first_seen_at.gt:3 days ago,upvotes.gte:226,upvotes.lte:235

first_seen_at.gt:3 days ago,upvotes.gte:236,upvotes.lte:245

first_seen_at.gt:3 days ago,upvotes.gte:246,upvotes.lte:265

first_seen_at.gt:3 days ago,upvotes.gte:266,upvotes.lte:285

first_seen_at.gt:3 days ago,upvotes.gte:286,upvotes.lte:305

first_seen_at.gt:3 days ago,upvotes.gte:306,upvotes.lte:345

first_seen_at.gt:3 days ago,upvotes.gte:346,upvotes.lte:385

first_seen_at.gt:3 days ago,uPost too long. Click here to view the full text.

I just noticed this in the right click menu,

"show all watchers' files in new page"

so, two things

1) watchers' should probably be watcher's

2) can you add a 'show all watchers new files in new page' to the right click menu? this would solve one of my bigger minor issues

>>10644

> watchers' should probably be watcher's

watchers' here stand the for files of the watchers. In English this is correct.

> new files

What is a new file?

>>10642

Your solution in >>10643 looks maybe a bit cumbersome and autistic to me, but if you find it works and doesn't annoy the server you are talking to (I imagine generating those queries is a little CPU expensive on their end), then that could be a solution. I might tune those ranges to be a bit broader, with maybe only five 'bands' of upvotes.

The DEAD result in >>10646 is an artifact of that particular band not happening to have files on its first check, which is related to how thin it is. I recommend you wait a couple days and tell that single query to reset/check_now and see if you can initialise it with some files. As long as your check options for this sub are sufficiently broad, like 'only die if less than one new file in six months', I shouldn't think this kind of ratings based feed work ever die (after the first good initialisation) as long as the site is active.

>>10644

Do you mean like 'the files that appeared since I looked last'? Unfortunately, watchers don't keep those sorts of records, so they only ever reproduce what their file import options have set in their 'presentation/publish' settings.

For the apostrophe, I meant that as the plural of watchers, as >>10645 says, since it works for multi-selection. It doesn't fix itself if you only have one watcher selected. There are a bunch of places around the program where I lazily use the plural label like this even when currently only one thing is selected, like 'do action x to this 1 files', because the finicky text alteration every time for a singular selection is a pain in the ass. I eagerly await the AI takeover that will force all humans to standardise to One Perfect Language that has a single letter for pluPost too long. Click here to view the full text.

Cool!

I thought it was always Tuesdays, but it seems like it's Thursdays.

Can't wait for version 333, it's going to be ebin. :DDD

Maybe a nice place to start is for Hydrus dev to enable the public wiki on the hydrus github page such that people can freely read and contribute without having to visit strange unrelated sites and documents, though it might create unwanted work for hydrus dev as he'll need to check yet another official channel and make sure that there won't be pure bullshit written there.

>>10451

Currently requesting wiki from wikidot, but also having an eye for these three

http://www.editthis.info/wiki/

https://miraheze.org/

http://www.referata.com/wiki/Main_Page

>>10595

http://hydrus.referata.com WEW

>>10417

While that works and does produce acceptable results there is an issue, its a fairly significant downgrade in quality. personally I would rather have extremely large resolution 1 bits that scale down in programs, this way with hither resolutions your images get better and you lose no detail, Ideally it would be a scan once and it's good forever kind of thing. granted if a vector scan happened that could see how big dots are and where they need to be placed, its possible that could outdo quality for relatively low file sizes, but thats a pipe dream till someone actually wants to archive actual print media and not just the print medias words.

>>10397

With what I know for coloring, it use to be done all by hand, but then you haver tools that follows objects on screen for special effects, you could apply that to this, where its tracking objects, then adding color on top and it does it automatically. with enough ai in the background you could easily make have 'looks like this, its brown, looks like this its green' for grass,

Look into deep fakes, as this is likely the closest we will get to doing that to artists, people who would never make more, super imposed on porn scenes, and its community made so you see real results that don't have big businesses backing it with unlimited funding.

as far as tagging goes, I still stand by everything should be human approved.

>>10403

definitely looking forward to it

>>10416

>>10414

That's frustrating–it usually relieves it significantly, if not fixes it completely. When I moved over to the new wx last year, some Linux environments got a lot stricter about when I could access ui objects. Mostly it is accessing variables of ui objects from the non ui thread. I cleaned up a whole ton, as many bad accesses as I could find, but there must still be a couple out there. Unfortunately, because I am in python, it is tricky to get good tracebacks or dumps for these errors, and my test environments here don't have the problem any more.

I am sorry that I can't offer a good fix here. If you notice that doing one thing tends to cause the error (or, in many cases, sets a state that will cause a crash up to 30 mins later), please let me know, and I'll have an area to focus on to see where I am screwing up. A trick I ran for a while when fixing this the first time was to have people just open the client and leave it for a bit, doing nothing. Then trying again, doing one thing, and then trying again doing another, to logically figure out what dialogs or actions were causing a crash state. If you have the patience and enthusiasm, this would be helpful to me.

Can you say which flavour and version of linux you are on, and which window manager?

>>10419

Yeah, for tagging, my idea for a first version here is to add a new suggestions column to manage tags that'll provide rows like '95% confident of "equine_penis"'. So it'll just be a neat tool to make tagging quicker. Because of high error rates with this stuff, I always want human eyes confirming any automatic decision. By the time we can have good systems that don't have any human in the loop, I expect we'll be more concerned with stopping the cloud of flesh-eating nanobots from consuming everything we love.

>>10422

>>>/tech/983134

>>10420

Which websites the login manager will be covering?

>>10424

Anything with a simple login form. It is completely user-configurable and shareable, like the new download system. It can do multi-page scripts, where it will parse some info from the first page and offer that as a temporary variable to another (like a POST form key or CSRF token). Anything needing a captcha is probably too complicated.

It determines login status with cookies atm. Cross-domain woo woo like exhentai is probably too complicated, but hydrus also supports importing cookies.txt from your browser so you can just copy across otherwise.

Today's release will be pretty empty, but I'll roll out some scripts for the big sites in coming weeks.

>>10326

I may be misunderstanding your problem here–do you mean to start a new gallery query paused? You can't do this right now, but I've thought of it, maybe with a cog button on the downloader panel.

If you just want to pause your galleries from downloading files for a bit (but still let them do the gallery search, to build up their queue), select them in the downloader list right after you add them and click pause/play files. You can select and affect multiple queues at once by shift- and ctrl-clicking.

>>10334

Thanks. Yeah, I used to wangle a low-resolution URL for the NSFW DA images by picking some tumblr share link apart, but that got shut down and now you have to be logged in (or age-gate click-through, I think?) to load any NSFW image from DA. I think I maybe skip those urls automatically in the parser? I expect to roll out some login scripts for DA and the other sites like this in the near future to let you access this content (and fix the parser, if need be). Please give them a go and let me know how it works for you.

>>10341

Don't worry about it–that number is inflated because I am currently tracking groups with pairs. You get the number of combinations (as in combinations and permutations), which will reduce in size similarly faster than the number you process. There's a bunch of n(n-1) in there, essentially, when the true number is n.

At some point I expect to move the system to true group tracking, at which point many of the inflated numbers and unoptimised actions here will work a hell of a lot better.

>>10296

You were right about it defaulting to python 3. What actually happened was, I checked the python version and it displayed the 2.7 version. So my system defaults to 2.7. However as soon as I start venv, it defaults to 3.5.

I now installed the packages again using pip2, and tried running it with python2.

Now it shows the splash screen and starts initializing stuff, but then fails with this;

"ImportError: No module named ordered_dict"

I randomly tried installing packages that sounded like they might be related to ordered dicts using pip2:

ordereddict

persistent-ordered-dict

odict

But none of them solved the error. Do you have any idea what I'm missing?

>>10377

The new version of requests can't unpickle a session object from the old version of requests. Try v327, it should be fixed (by clearing your old sessions, argh).

>>10345

Also https://vision.fe.uni-lj.si/cvww2016/proceedings/papers/04.pdf (Quantitative Comparison of Feature Matchers Implemented in OpenCV3) and

https://sci-hub.tw/10.1109/m2vip.2016.7827292 (Comparison of OpenCV’s Feature Detectors and Feature Matchers)

All the algorithms listed are in OpenCV, so there is nothing to lose by implementing some of these.

>>10345

>>10346

Alright got it. It seemed at first just like a huge link dump and "hey I did some research". The hint about all this already being available in OpenCV is great. Sorry for reacting like I did then.

This information is useful and interesting, but I unfortunately do not have time to go through it in any clever way. I can't work on big things like new duplicate finding algorithms in normal weekly work–it'll have to wait for the next iteration of the duplicate system. I'd also like to prioritise the duplicate processing workflow, which is the larger problem with the system right now.

I am most interested in face and other feature detection in cv. I expect to use some of this stuff when we push towards machine learning auto-tagging.

>>10348

For the dev: https://medium.com/machine-learning-world/feature-extraction-and-similar-image-search-with-opencv-for-newbies-3c59796bf774

https://github.com/creotiv/computer_vision

>>10338

In case of ghostbin closing down https://pastebin.com/KPaYiXNM

>>9864

Yes if the md5 of your files is in the PTR they will get tags.

Go to the help menu > I don't know what I'm doing > just set up some repositories for me, and hydrus will set up the PTR for you.

It will take a long time to sync, so instead of waiting I'd recommend you download the update files from here: https://cuddlebear92.github.io/Quicksync/

Then you can import them with services > import repository update files.

It is secure because you download all the tags, even for files you don't have, so no one knows what files you have. Unfortunately because of that your database will increase in size a lot, to about 10GB.

The PTR mainly uses a booru tagging style, but it should be noted it is currently kind of messy with some useless/bad tags. But you can set up tag censorship/sibling rules to get rid of those if it bothers you.

>>9868

Thank you

I know I've been gone a while, but…

Don't forget your backup, anon

Thank you based dev.

Thank you based dev.

>>10225

Right, I just looked into this, and ffmpeg was giving some funny framerates:

mp4: 0.77 fps 20tbr

webm: 2.5fps 50 tbr

tbr is some kind of fps estimate ffmpeg is pulling from the codec or something. iirc, it stands for:

T: Standards

B: Engineers

R: Are fucked in the head m8, big time

I am not totally sure, but it looks like there is some variable frame rate going on when I look at them in MPC, so I guess it is even more complicated. In any case, I have rejiggered my parsing code to be less trustworthy of these values when they differ like this, and now the videos' frames are counted manually–I now get 7 and 13 respectively, with correct durations.

So these should work better for you in tomorrow's release! As hopefully should the other various low-framerate low-frame-count vids that seem to render all in a rush and then hang on the last frame for ten secs. In v326, turn on help->advanced mode and then right-click these videos and hit reparse files and regen thumbs. They should fix up, at least a bit.

Let me know if you run into any more trouble like this.

This week I started getting this error with pixiv subscriptions in version 325:

Exception

The subscription Pixiv encountered several errors when downloading files, so it abandoned its sync.

Traceback (most recent call last):

File "include/ClientImportSubscriptions.py", line 1125, in Sync

File "include/ClientImportSubscriptions.py", line 479, in _WorkOnFiles

Exception: The subscription Pixiv encountered several errors when downloading files, so it abandoned its sync.

It downloads around 10 to 30 pics, then throws that exception. Is there any way to display the post IDs of failed downloads? I retried a few times, with one artist, and it didn't download any new pics on retry, then threw the exception again. So I assume it fails on the same posts again.

>>10275

Pixiv are in the midst of changing some of their layout again. Please try again in v326, which will automatically update you to a hopefully more stable solution.