/hydrus/ - Hydrus Network

>>13387

Got some weird errors after updating

ConnectionException
Could not connect!
Traceback (most recent call last):
  File "site-packages\urllib3\connection.py", line 159, in _new_conn
  File "site-packages\urllib3\util\connection.py", line 80, in create_connection
  File "site-packages\urllib3\util\connection.py", line 70, in create_connection
OSError: [WinError 10049] The requested address is not valid in its context

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "site-packages\urllib3\connectionpool.py", line 600, in urlopen
  File "site-packages\urllib3\connectionpool.py", line 354, in _make_request
  File "http\client.py", line 1239, in request
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "http\client.py", line 1026, in _send_output
  File "http\client.py", line 964, in send
  File "site-packages\urllib3\connection.py", line 181, in connect
  File "site-packages\urllib3\connection.py", line 168, in _new_conn
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x0000000014EA1160>: Failed to establish a new connection: [WinError 10049] The requested address is not valid in its context

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "site-packages\requests\adapters.py", line 449, in send
  File "site-packages\urllib3\connectionpool.py", line 638, in urlopen
  File "site-packages\urllib3\util\retry.py", line 398, in increment
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='0.0.0.0', port=4443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000000014EA1160>: Failed to establish a new connection: [WinError 10049] The requested address is not valid in its context',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "include\ClientNetworkingJobs.py", line 947, in Start
    response = self._SendRequestAndGetResponse()
  File "include\ClientNetworkingJobs.py", line 255, in _SendRequestAndGetResponse
    response = session.request( method, url, data = data, files = files, headers = headers, stream = True, timeout = ( connect_timeout, read_timeout ) )
  File "site-packages\requests\sessions.py", line 533, in request
  File "site-packages\requests\sessions.py", line 646, in send
  File "site-packages\requests\adapters.py", line 516, in send
requests.exceptions.ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=4443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000000014EA1160>: Failed to establish a new connection: [WinError 10049] The requested address is not valid in its context',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "include\ClientParsing.py", line 2673, in FetchParsingText
    network_job.WaitUntilDone()
  File "include\ClientNetworkingJobs.py", line 1151, in WaitUntilDone
    raise self._error_exception
  File "include\ClientNetworkingJobs.py", line 1050, in Start
    raise HydrusExceptions.ConnectionException( 'Could not connect!' )
include.HydrusExceptions.ConnectionException: Could not connect!

Nothing seemed wrong, but I got multiples of this, though I think they were all the same.

>>13391

>>13393

>>13392

My best guess here is that png handles single colors better then it handles the middle, but for the jpeg to get as clear text as it has it has to boost things up quite a bit. otherwise I cant think of a single reason why the png would be smaller than the jpeg regardless.

If you add in a pixel for pixel auto remove, I think it should favor file size rather than favor filetype, as I can't imagine the jpeg getting smaller if it was even higher quality. there will be cases where people screenshot something (god knows im guilty of it) and you later find the original jpeg in a smaller file size, but there will also be cases like this where the smaller file will end up being the png.

>>13397

just to be clear, i'm perfectly ok with an auto decision on pixel for pixel duplicates, but the only way I see it as 100% automatable is if it favors smaller size over extension.

>>13387

I just read the updated duplicate help and have two questions that are somewhat related.

Suppose "A" becames King of B, C and D because it is better and the other three are deleted.

If you import any of B, C or D from a booru, does "A" get the tags?

Does "A" gets the tags of any of the three from the PTR based on its status?

Thank you for the awesome program!

>>13387

does this work on FREEBSD? I got a 20 tb server with tons of stuff that I can help keep alive and shit.. just let me know if there's a pkg i can install and use behind a vpn?

>>13387

Is there a way to add a new local tag service? I've been trying out DeepDanbooru but it would be nice to be able to separate its tags from mine, at least while I test to see how well it works.

>>13394

Thank you for this report. Can you say anything more about the context that this error appeared? Was it part of a downloader, a subscription, or did it just come in a popup out of nowhere? It looks like that FetchParsingText call that it was trying to do some download stuff, but then it was pointed somewhere strange and could not connect. The error looks like a standard 'could not connect' error that you get if you hit a site that is down, but that 0.0.0.0:4443 address seems pretty weird. Maybe it is some initialisation state, but I am not sure. If it was a downloader, what was the URL it was trying to fetch that failed?

>>13397

>>13398

Thanks. I think I'd like more information here before I make a decision. I always like to make these sorts of auto-rule systems as customisable as can be, because Anons have all sorts of preferences, but my strong suspicion is that other than unusual edge cases like pure white squares or other geometric shapes, a png copy of a jpeg is 99.9% going to be the larger of the two files. It is fascinating to me that this jpeg copy of a png could never be, even at the highest quality settings, an exact pixel-for-pixel dupe. If it is the case that jpg copies of pngs will practically never be pixel for pixel dupes due to smooth colour gradients in the png not being replicable in the jpeg colourspace, then the auto-resolution of jpg copies of pngs is outside the domain of pixel-for-pixel dupes. This is a strange thing, but still a minor revelation to me.

I'd never have a system like this default on, in any case. Like everything else, I'll try to massively over-explain it and force the user to turn it on if they want it.

>>13399

Not yet. I had a tentative job to explore post-duplicate metadata merge/permanent synchronisation in this iteration, but adding that system was bigger than I had time to do well. It'll be slightly complicated, and I don't want to mess it up with a quick hack.

So just to be clear, for now the metadata merge only works on the pairs you see in the dupe filter. But since both representatives from the dupe filter are the Kings of their respective groups, and therefore 'good' data travels up from groups to kings, you can generally assume that Kings will have fairly 'complete' metadata based on previous decisions for now.

Since duplicate records are permanent, once I do add something like this, we'll be able to retroactively apply whatever merge settings you prefer, so non-Kings of groups will all sync too in an ongoing basis.

I suspect when I do this I'll have different or better default duplicate merge options here. Some tags aren't appropriate to merge backwards, like 'high resolution' or 'webm', and it'd be nice to have examples of those as default, or easily selectable, rather than making users awkwardly figure it out on their own.

>>13408

I am not certain. I think I remember reading about one user getting it to run on FreeBSD some time ago, but I can't be sure. The program is devved on and works best for Windows, and I build the Linux release on Ubuntu, so some more esoteric flavours of Linux and their Window Managers can have trouble with my build. Your best bet is probably running from source, for which I have some help here:

https://hydrusnetwork.github.io/hydrus/help/running_from_source.html

There's an Arch package maintained by a user, but I don't know enough about FreeBSD to say what is compatible:

https://aur.archlinux.org/packages/hydrus/

I assume that wouldn't work for you, right?

If you have a Windows machine, I recommend you try hydrus on that just to see if you like it first, rather than going through the rigmarole of getting it going on your FreeBSD machine and finding it isn't what you like. If you do give it a go, let me know how you get on!

>>13417

Not yet, but as it happens I am on the verge of doing some heavy work on local tags and I hope to add it then. I'll do a couple of weeks on some audio metadata stuff and then start on all that.

>>13425

I had added the lookup script for Deep Danbooru but couldn't get it to work; it would get stuck in the uploading stage and quietly fail out. But the errors didn't pop up till maybe 20 minutes after my various attempts so I wasn't sure if they were related or not.

>>13426

Like I said I think, the reason this one works for the jpeg being larger is due to the text, at least that what I think. to get clean text on a png, it's more or less par for the coarse and doesn't eat much space, but for a jpeg, to keep it without any kind of flaw, at least visually, it would need a stupid amount of resources. edge cases like this, or potentially others like this where a jpeg is made, and my best guess is one of the joi threads on 4chan had someone blanket convert with the highest quality settings on them, hit this image and it tossed out a larger file then put in. i'm guessing that 2/3 or above of the image being something jpeg does not handle well had a part in why this exists the way it does.

as for an auto decision, I don't see how the difference between 'the pixels are the same choose jpeg' or 'the pixels are the same choose lowest file size' either way, 99%+ of the time it will go jpeg anyway, this would just introduce a safety net for the sub 1% of the time jpeg isn't the smallest choice.