/hydrus/ - Hydrus Network

>>11533

windows offline updater, and look for a list of suspect updates and hard block them, that way you have all the updates with none of the bullshit. and this is a good 2400% faster then normal windows update if I remember correctly.

>>11528

Thank you for this report. This is odd and interesting. My first guess is perhaps that file got reparsed at some point with newer code/ffmpeg, so while the db entry considers it more accurately now an mkv, the file structure does not.

Do you ever do right-click->repare and regen thumbs on your movies?

I had not considered this when I wrote this routine (I was thinking of frame counts and bad thumbs at the time). I will update the routine so it updates the file storage as well, and I'll see if I can get it to auto-heal retroactive extension problems like this too. I may not have time to do this today, but please let me know how you get on with this as I roll it out.

If you would like to fix it immediately, you can just manually rename the file's extension in client_files. Afaik, that's the only problem here.

If you believe the file really should be detected as a webm rather than mkv, and it isn't private, please post it here or email it to me and I'll have another look at my parsing.

>>11533

Thanks, I will.

>>11536

>Do you ever do right-click->repare and regen thumbs on your movies?

Yes, in fact I did this on all my videos because I'm the guy who had Hydrus randomly generate broken green thumbnails for videos (which seems to be fixed now by the way, if not by you then perhaps because I got a new computer and switched from Windows 7 to 10).

An auto-fix would be nice because it has happened to a lot more than just that one video.

Is webm and mkv very close in how the files are structured? Because most of these were downloaded from gelbooru which lists them as webm. But I guess it doesn't really matter if the hash stays the same.

>>11540

Yeah, iirc, a webm is just an mkv with VP8/9 video and vorbis audio, I think. All webms are mkvs–they are a subset. I think the idea was to guarantee a limited mkv format that would work on all devices for HTML5 streaming etc…

I could also have some of my ffmpeg parsing wrong here, or had it wrong when you did this reparse. I used to use an external library to figure this out, and now I compare ffmpeg output specifically. If you run the reparse again, maybe it'll flip back to webm!

Anyway, should be fixed for tomorrow. Let me know if you still have any trouble.

>>11508

六四天安門事件 The Tiananmen Square protests of 1989 天安門大屠殺 The Tiananmen Square Massacre

反右派鬥爭 The Anti-Rightist Struggle 大躍進政策 The Great Leap Forward

文化大革命 The Great Proletarian Cultural Revolution

人權 Human Rights 民運 Democratization 自由 Freedom 獨立 Independence 多黨制 Multi-party system

民主 言論 思想 反共 反革命 抗議 運動 騷亂 暴亂 騷擾 擾亂 抗暴 平反 維權 示威游行

法輪功 Falun Dafa 李洪志 法輪大法 大法弟子 強制斷種 強制堕胎 民族淨化 人體實驗

胡耀邦 趙紫陽 魏京生 王丹 還政於民 和平演變 激流中國 北京之春 大紀元時報 九評論共産黨

獨裁 專制 壓制 統一 監視 鎮壓 迫害 侵略 掠奪 破壞 拷問 屠殺 肅清 活摘器官

黑社會 誘拐 買賣人口 遊進 走私 毒品 賣淫 春畫 賭博 六合彩

台灣 臺灣 Taiwan Formosa 中華民國 Republic of China 西藏 土伯特 唐古特 Tibet 達賴喇嘛 Dalai Lama

新疆維吾爾自治區

The Хinjiang Uyghur Αutοnοmοus Region 新疆维吾尔自治区

>>11508

Other than the shit modern culture, being a literal bugman due to pollution and being the worst kind of nationalist?

>>11507

this is a copypasta that gets autobanned over on cuckchan. people use it to stop chinese threads/ chinese shilling in their tracks because what's listed in the copypasta is all censored by the chinese government.

晚上好。我正在学汉语。

Don't you go to jail for anime titty in China

>>11482

The -d path should be the directory, not the file. Try

/Users/user/Documents/hydrus/db

Maybe the OS is having trouble trying to read the client.db file as a dir or something. I don't know anything about doing '\ ' rather than using quotes, and I had to do -n and -a to get it to A) not highlight the existing running app, regardless of args given, and B) to just run the Hydrus App I had installed in applications.

>>11484

>>11486

Thanks. They changed the access permissions on their main cdn but still link to it in the html meta. I will have a fix rolled into today's release.

>>11492

Are you on Windows 7? You may need some OS updates and possibly VCRedist 2015, as here:

>>11260

>>11495

https://www.microsoft.com/en-us/download/details.aspx?id=48145

Eric S Raymond's Commands

0. Be fearful of your god, that worship without self-sacrifice is a sin

1. You shall complete basic programming training, he who does not is a heretic

2. As per daily rituals, you must code everyday, to seek the road to ascension

3. The hackerspace shall be your temple, and the members shall be your brethren

4. At the end of your journey, you will present your Magnum Opus as a final offering

a. The Magnum Opus will act as a reflection of what you have not learned

b. The Magnum Opus will be a project that you have full responsibility over

c. The Magnum Opus will be on public display, and all critics will be your judge

Taken from http://esr.ibiblio.org/?p=4817 and a message to Hydrus Dev

>>11452

Hey, I strongly recommend you let a new client sync at the default rate. It is good for the database to sync in pieces, so it can run maintenance as it goes.

There are many years of tags in the PTR now, hundreds of millions. I recommend new users not try to get them all at once. Give it time, and you'll see tags start to appear after a few processing cycles.

>>11435

Any ideas of designing OR search?

https://danbooru.donmai.us/wiki_pages/43049

https://gelbooru.com/index.php?page=help&topic=cheatsheet

https://yande.re/wiki/show?title=cheat_sheet_extended

https://e621.net/help/show/cheatsheet

https://hypnohub.net/help/cheatsheet (MyImouto)

https://derpibooru.org/search/syntax (Best syntax)

up to v336, still getting this error when importing a folder

UnicodeEncodeError

'charmap' codec can't encode character '\u014d' in position 71: character maps to <undefined>

Traceback (most recent call last):

File "include\HydrusData.py", line 854, in Print

print( str( text ) )

File "include\HydrusLogger.py", line 130, in write

self._log_file.write( message )

File "c:\python36\lib\encodings\cp1252.py", line 19, in encode

UnicodeEncodeError: 'charmap' codec can't encode character '\u014d' in position 66: character maps to <undefined>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "include\HydrusThreading.py", line 307, in run

callable( *args, **kwargs )

File "include\ClientGUIDialogs.py", line 1031, in THREADParseImportablePaths

HydrusData.Print( 'Unparsable file: ' + path )

File "include\HydrusData.py", line 858, in Print

print( repr( text ) )

File "include\HydrusLogger.py", line 130, in write

self._log_file.write( message )

File "c:\python36\lib\encodings\cp1252.py", line 19, in encode

UnicodeEncodeError: 'charmap' codec can't encode character '\u014d' in position 71: character maps to <undefined>

>>11403

Great, I'll do something like this then.

>>11409

Thank you for this report. I missed some of the unicode stuff last week. I have it fixed for 337.

Note that this is a logging problem, so that particular file it wants to complain about (which probably has some unicode in its filename) may still be broken in 337–it'll just be able to correctly report that it is 'unparseable'. If you get a better error here next week and the file nonetheless seems good in an external media viewer, please submit it and I'll have a look on my end!

>>11390

when it comes to anti virus, they have 2 methods of working

1) knowing the behavior of the program, if its doing something iffy, they may catch it

2) knowing the exact exe

other then this, anti viruses are kind of worthless. you are honestly better off working in a vm, and becaues these things dont typically catch new things, only old ones, by the time antiviruses catch and patch for things, your os is usually already patched, or will be patched shortly, and for the few things that would still hit you that you don't ask for, it would be better to have a hardware based firewall.

https://i.4cdn.org/gif/1548042014339.webm

float division by zero… (Copy note to see full error)

Traceback (most recent call last):

File "include\ClientImportFileSeeds.py", line 1211, in WorkOnURL

self.DownloadAndImportRawFile( file_url, file_import_options, network_job_factory, network_job_presentation_context_factory )

File "include\ClientImportFileSeeds.py", line 544, in DownloadAndImportRawFile

self.Import( temp_path, file_import_options )

File "include\ClientImportFileSeeds.py", line 763, in Import

( status, hash, note ) = HG.client_controller.client_files_manager.ImportFile( file_import_job )

File "include\ClientCaches.py", line 1088, in ImportFile

file_import_job.GenerateInfo()

File "include\ClientImportFileSeeds.py", line 289, in GenerateInfo

self._thumbnail = HydrusFileHandling.GenerateThumbnail( self._temp_path, mime, percentage_in = percentage_in )

File "include\HydrusFileHandling.py", line 113, in GenerateThumbnail

renderer = HydrusVideoHandling.VideoRendererFFMPEG( path, mime, duration, num_frames, cropped_dimensions )

File "include\HydrusVideoHandling.py", line 598, in init

self.fps = self._num_frames / self._duration

ZeroDivisionError: float division by zero

>>11422

Thank you for this report. Someone actually sent me a similar unusual-frame-count music vid the other day, so this is fixed for tomorrow's release already! Let me know if you still have any trouble in v337.

>>7858

IPFS based image sharing is awesome

>>7858

Still no source code one year on… please show us more

>>11386

with 2 references to a "decentralized booru" https://rbt.asia/g/thread/68261237 https://rbt.asia/g/thread/64493084

>>11387

And maybe this https://www.nayuki.io/page/designing-better-file-organization-around-tags-not-hierarchies

>>11468

You can probably find archives in

https://hollaforums.com/

https://powforums.com/

https://supforums.com/

https://veekyforums.com/

https://zigforums.com/

>>11334

>Great, thanks for letting me know. Maybe you can tell Kaspersky to ignore or treat the hydrus exe/install folder a little differently? Turn off 'active' protection or something so it isn't scanning every time it runs?

Just letting you know I whitelisted Hydrus and it works, thanks.

>>11334

>>11328

Hey, on closer inspection, these were giving me 18KB 'webm' files that were actually some kind of CloudFlare html saying 'we think you are a bot, please solve this captcha'. Once I solved that in my browser, they worked mostly ok in the current dev version (which admittedly now has some unrelated ffmpeg fixes). The /trash/ ones are now 404, but https://i.4cdn.org/gif/1547302893499.webm is giving me a resolution problem that I think I can fix today.

I think part of this problem is the result of the cloudflare-4chan setup giving the wrong http header to say the file was 'webm' when it was really an html interstitial.

The other half is that hydrus has no capability to solve these sorts of captchas. I have never encountered this CF page before on a content fetch, and if it is going to be a new 4chan thing, that's a shame. Let's hope it is a temporary thing that was a misconfiguration or an accidental statistical spike on that set of files due to some mis-aimed DDoS or something. Please let me know if you encounter more of this.

>>11351

Thanks. I'd like to add captcha support, but diving back into the downloader code after all the work I did last year is not my top priority right now. It would be a lot of work, whether I would present login/download captchas to the user in the ui or try to solve them automatically.

Current plan is to do it for the next big iteration on the downloader, whenever that will be. If 4chan or other sites start getting this CF captcha shit a lot, I'll bump it up the schedule, maybe as a hardcoded fix that recognises that CF page specifically.

It is strange that both you and I got the same captcha issue, since I hadn't looked at that content/threads on my own machine/IP previously. That suggests to me the data source was the flagged problem, not the specific client, hence my DDoS-false-positive-on-that-thread suspicion. Again, we'll see if that starts happening more often.

>>11354

Thing is, I saw it fail, checked it, and was able to download it outside of client, then when you told me about the captcha and I checked It I got the captcha too.

No idea what what the hell was going on with that one.

is this the kosher switch?

>>11350

Warning: Slight memetic risk, you may lose a couple of hours falling into this:

https://en.wikipedia.org/wiki/Woodworking_joints

https://www.youtube.com/watch?v=Hk4cwtQJmwU

https://twitter.com/TheJoinery_jp

Rats! I was getting excited reading the commits.

>>11236

Changed my mind. I want to hammer this shit out tonight. Uploading to mediafire now, release in 10-20mins.

HYPE!

Cool! Can't wait to see that commit's diffs

>>11203

Hey, I am sorry, I do not totally understand what you wrote here. Can you rephrase it or add a screenshot?

I do remember a user reporting an issue with 'import folders' having bad tracking for the files it did not import (for whatever reason)–was this you? If you check the button next to 'review currently cached import paths' on the edit import folder panel, what do the paths there suggest about your problem?

>>11227

>>11226

Thanks. I will make a job to check this code and have it occasionally clear out paths that no longer exist.

I'll also be looking at changing some of these popup progress gauges to show progress more for the 'current job', like 1/17, rather than always showing entire-life progress like 4001/4017, which is always just a green bar with one pixel of grey, no matter how big the most recent sub sync or whatever was.

>>11231

lol asked about that somewhere too because that was a bit annoying seeing 'oh there are 1000 images here, the fuck happened' only to see the next stage of the sub have 975/983 and realized it remembers everything. not worth resetting the checkers for as to me it was a minor annoyance.

I should also say, the weirdness where on saving the session where it would bloat from 12gb to 18+gb has been solved with the removing the dead watchers from active view, however with a little over 100k images in active view mode, I am unable to set it to every 5 minutes as it does start to bloat and lag a bit, no where near the extent it did before, but a bit.

>>11171

Reeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

Happy New Years hydrus dev!

>>11061

Yeah, it worked pretty good for me. A little spammy, but fast and easy to use. It mostly:

- Put safety parentheses around all my print statements (even though they all already had them)

- Put a safety list() around every iteration over a dict's keys/values/items call

- Changed up some explicit unicode language to str and cleaned up all the u stuff to just

- Did some really helpful urlparse -> urllib.parse and cStringIO -> io.StringIO stuff

This was all stuff I had feared I would have to do myself, so I was happy it was able to catch it all. The code was valid py3 and did boot. Unfortunately, it couldn't handle the encode/decode calls I had all over the place to handle some str/bytes conversion, so I knuckled down and learned how it works in py3 and rewrote the outstanding shit until it booted right.

I am overall very happy with the unicode changes in py3. They realised what was bad in py2 and fixed it all.

I still have a load of bytes/unicode stuff to clean up and test, stuff like object serialisation and network read/write, but that seems to be mostly it. Once I was booting into something stable, video rendering somehow worked right out of the gate.

I wrestled a bit more with pyinstaller today to get a frozen build going, and it was a pretty huge pain in the balls but I think I've figured something out. There's still a CWD issue stopping it getting too far, but all the dlls are loading and the splash screen comes up.

It may require a clean install, and <Win10 users may need Visual Studio Redistributable 2015. I will do testing with several users before I do the proper release so I know this better.

>>11056

There are some unit tests I threw together, and I run them every week, but I have had a lot of trouble fitting the 'make a comprehensive test suite and keep it updated' into my schedule. I would really like to have something neater here, but can't figure out how to get it done.

>>11062

So are there any code that you feel is slow in Python 3? And if so would you consider Nim-lang (using NimPy) to speed up parts of the code? Or maybe golang (with GoPy)?

Nim is basically a Python-like language that compiles to C, and Golang is made by Google to be a faster, better Java.

NimPy: https://narimiran.github.io/2018/05/10/python-numpy-nim.html

https://github.com/jboy/nim-pymod https://github.com/yglukhov/nimpy

GoPy: https://hackernoon.com/extending-python-3-in-go-78f3a69552ac

https://github.com/go-python/gopy https://github.com/sbinet/go-python

>>11083

I am happy with it so far. Some python-heavy computation like the lag on tag siblings/parents cache construction on client boot appears to be running much slower. Overall it feels it is running faster than py2. There's still a bunch of jank thread code on my end jittering things up, which is a long-term project to clean up.

There's a lot of duct tape that makes hydrus run from the frozen executables, and I have no experience with non-vanilla versions of python. I expect it is possible to get them going, but I bet trying to install wxPython and OpenCV is an additional headache. For now, I'll stick to the simpler environment. If you have experience with them and try it, let me know how it goes!

>>11155

Whoops, I meant the siblings/parents is running much faster. No significant problems anywhere, is the experience.