/hydrus/ - Hydrus Network

Ok guys, I'm going to build this from source, don't try to stop me.

It deleted my database.

>>11238

So I just saw this sticky appear and went to the github page for the release as usual, but I saw this:

>for users who are updating, the client has special install instructions for just this week:

>if you are a windows or linux user who extracts to install, you will have to delete your old install's files (but keep your db folder!!!) before installing/extracting the new version so there are no 2/3 dll/so conflicts (don't delete your db folder!)

>if you use the windows installer to install, this v335 installer will do the clean install for you! there is absolutely no way this could go wrong, so no need to make a backup beforehand :^)

And I really didn't trust the smiley with the carat nose. In my experience that literally means you're bullshitting.

Anyway, I know hydrus dev wouldn't do us like that… I would think… but I'm gonna wait for others to make more noise if they will, cause I am just a meek laptop Windows user with a 1TB hard drive with 30 gigs of space left. I can't afford to backup, even if it's easy to just say "I told you to backup bro."

Everything seems to have updated fine here, I'll let you know if I run into any issues.

Proud extract user.

Will wait a good 10 day before doing anything, due to heavy hot repaging action.

Will track attentively nonetheless.

>>11241

Just take a backup before updating and you should be good to go.

>>11245

Nevermind, I'm sleepy and missed last part. .zip your archive to several parts and upload to Mega or something, maybe?

>fresh restart

>almost 3gb ram usage

Just how many files do you have?

>Have an old Vista partition on-hand, for the sake of game compatibility

>It somehow runs most programs designed for Win7 and up, sometimes without issue, sometimes with a little tinkering

>Up to v334, as long as ffmpeg was manually replaced (using a v3.0 EXE), Hydrus was completely useable

>As of v335, the client process immediately self-terminates, without error logs or even attempting to boot

New releases of Python always ditch compatibility with an old Windows OS once the official "extended support" ends, so this was inevitable.

Those of you still in the XP or Vista seats should change cars if you want to continue The Ride, and Win7 folks should be wary of Python updates when their "extended support" ends next January.

>>11240

If you aren't having a giggle to give me a heart attack, can you describe more what your update process was here? Which OS and which build did you use? Did you see any errors?

>>11241

There's no funny business in this release, and I have tried to make everything work properly, but this is a big step and something can always go wrong. Read the update process for your build in this release post carefully and follow the steps. If you are skeptical about the exe installer going right, you can always follow the steps for the extract release, which will also work. Don't delete your install_dir/db folder.

Every week is a great time to start making a backup. I strongly recommend you buy a cheap 1-2TB external usb drive like a WD Passport (usually about $60, or cheaper second-hand from a friend) and set up a regular backup routine, as described here:

https://hydrusnetwork.github.io/hydrus/help/getting_started_installing.html

Losing a lot of shit due to hard drive fault or accident fucking sucks big time, and once you sort out a backup routine, you no longer worry about it.

>>11247

Yeah, my client, which has ~1.3M files, now sits at a happy 500MB memory usage and 0.3% CPU idle, where before it was 850-1,500MB and ~2.5%. I am really pleased with the changes. If I understand right, py3 does some better stuff with the python/C++ interface, so they reduced a bunch of memory (and CPU scheduling) bloat.

>>11254

Yeah, I am sorry about this. The new python relies on Visual Studio 2015 dlls, which I'll bet are just incompatible with <Win7. You'll have to keep on 334 unless/until you can a newer machine. There are no db changes this week, so you can roll back using the same extract only clean install instructions (or just to a backup).

I had thought I had put this in the release post notes, but I think I only put it here:

https://hydrusnetwork.github.io/hydrus/help/getting_started_installing.html

That users with <Win10 may need Visual C++ Redistributable for Visual Studio 2015, as here:

https://www.microsoft.com/en-us/download/details.aspx?id=48145

I didn't get a conclusive answer from my <Win10 test users here, but I didn't hear any problems either, so it may not be completely needed. I expect most people have it for vidya anyway.

If this is a problem for 7/8/8.1 users, I may be able to fold the redist installer exe into my InnoSetup script, just like vidya would.

Here's some more info on this, including the table of working compilers (WARNING: it is Microsoft versioning hell), in case you are a madman who wants to try to get this running from source on 3.4 or something:

https://wiki.python.org/moin/WindowsCompilers

Does't start on Windows 7 with 2015 libraries, no logs or console messages, just silently quits. Tried with clean install and install over the previous version. Starts on Windows 10 though. Will investigate further.

>>11255

Windows 10, not sure what my old build was (it wasnt the most recent), and I used the normal installer. On boot I got the "looks like you're new" message and all the db/client_files folders are empty. Didn't get any error messages.

>>11238

Thanks for the update dev! Everything seems to be working fine on my machine (win10), except that I can't change the ratings of my files anymore.

>>11256

I gained the balls to do it after your reply, and it worked (I just ran the installer on Windows 7). It even preserved my tabs and black theme, which I did not expect. I thought everything would be lost except the files and tags. I didn't read anything beforehand though, like a retard.

Anyway yeah, not having a safety net sucks. It's pretty indefensible. Tbh I was kinda projecting in that post cause I've seen so many people just say "dude, backup", but for the many dozens of times I've seen it said in the wild, not one person ever provided any resources on how to do it. Yet everyone gives the people who get burnt shit for not having backed up. It's just really free to say out of context but that doesn't in itself save anyone.

So thanks for the effort, actually. I appreciate it. And definitely, moving forward you need to have a backup… It should be part of the first step for everyone.

>>11267

Is this real? I'm glad I'm making a backup of my entire directory beforehand.

If you didn't, then I'm terribly sorry.

>>11268

I forgot I deleted my backup last week cuz I needed room. Didn't realize it til after the wipe…

>>11267

Could you try previous versions/shadow copy?

>>11263

You don't need to keep them anymore if I recall. I deleted all mine and the tags were still intact, so I'm guessing it permanently adds the tags to your main db once you sync it once.

>>11274

Is this real life?

>>11238

Nice work! Thanks to the Python3 migration, I'm finally able to run client.py directly.

If there are any Gentoo users here who want to do the same: You probably want "pip install –user wxpython" (takes a while to complete). The rest can be installed from portage.

>>11238

Importing a folder of files where some of the files are likely to be corrupt.

UnicodeEncodeError

'charmap' codec can't encode character '\u014d' in position 95: character maps to <undefined>

Traceback (most recent call last):

File "include\HydrusFileHandling.py", line 333, in GetMime

mime = HydrusVideoHandling.GetMime( path )

File "include\HydrusVideoHandling.py", line 227, in GetMime

lines = GetFFMPEGInfoLines( path )

File "include\HydrusVideoHandling.py", line 117, in GetFFMPEGInfoLines

info = proc.stderr.read()

File "c:\python36\lib\encodings\cp1252.py", line 23, in decode

UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 1646: character maps to <undefined>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "include\HydrusData.py", line 835, in Print

print( str( text ) )

File "include\HydrusLogger.py", line 130, in write

self._log_file.write( message )

File "c:\python36\lib\encodings\cp1252.py", line 19, in encode

UnicodeEncodeError: 'charmap' codec can't encode character '\u014d' in position 90: character maps to <undefined>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "include\HydrusThreading.py", line 307, in run

callable( *args, **kwargs )

File "include\ClientGUIDialogs.py", line 1083, in THREADParseImportablePaths

mime = HydrusFileHandling.GetMime( path )

File "include\HydrusFileHandling.py", line 346, in GetMime

HydrusData.Print( 'FFMPEG couldn\'t figure out the mime for: ' + path )

File "include\HydrusData.py", line 839, in Print

print( repr( text ) )

File "include\HydrusLogger.py", line 130, in write

self._log_file.write( message )

File "c:\python36\lib\encodings\cp1252.py", line 19, in encode

UnicodeEncodeError: 'charmap' codec can't encode character '\u014d' in position 95: character maps to <undefined>

>>11279

Yeah, I think I just got this, but I posted it in the bugs thread on the catalog.

>>11271

>>11265

I have the same rating problem as these anons.

All rating values can only be set to the minimum value, or the maximum value, but nothing in between. Existing ratings are unaffected.

>>11257

3.3 million files here, and currently little over 100k open due to massive file imports to clear up a hdd, going to probably get that down to around 20k open over the weekened if I have time/willpower to do it.

Congratulations on the migration, hydrus.

This error when clicking the "add tags based on filename" button in the import dialog, and then nothing happens

TypeError
'>' not supported between instances of 'str' and 'bytes'
  File "include\ClientGUIDialogs.py", line 842, in EventTags
    panel = ClientGUIImport.EditLocalImportFilenameTaggingPanel( dlg, self._current_paths )
  File "include\ClientGUIImport.py", line 1509, in __init__
    self._tag_repositories.AddPage( name, name, page )
  File "include\ClientGUICommon.py", line 1818, in AddPage
    if current_display_name > display_name:

>>11274

Those kind of programs can only salvage deleted files whose physical location on the disk hasn't been overwritten by something else. Most likely the disk where Hydrus is installed saw a lot of writing after the fact, possibly by the installer itself.

Everything works fine

OS is arch linux and I'm running from source.

I'm the linux user with Filter Tag Dialog issues. Still getting them in 335. Additionally, just now hydrus crashed on me when I attempted to open the subscription manager, after having had the filter tag dialog open some.

Otherwise the update works nicely. Had expected more issues after such a large refactor.

>>11290

Nope, I take it back. I'm getting a lot of:

client: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.0.

I also got following error when just clicking around in a gallery:

The program 'client' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadPixmap (invalid Pixmap parameter)'.
  (Details: serial 14451 error_code 4 request_code 139 minor_code 5)

>>11291

Are you running tarball version or built it from source?

Gentoo Linux, updated via git & built from source, everything works so far

>>11267

>>11261

>>11269

>>11274

I am very sorry you have had this problem. I cannot explain it, and I hope you can recover as much as possible from your backup.

It is stupid to ask, but just to be certain, is the install location correct and as you expected? It didn't change or typo for some reason? If you do a drive-level search for 'client.db', does that turn up a new unexpected location? And you never made a shortcut to launch the client with the -d switch?

Also, is it possible that your install location became/stopped-being read-only? The newer version of hydrus will test if its install location is writeable, and if not, it will instead use ~/Hydrus (i.e. C:\Users\[You]\Hydrus). Is the screen at >>11267 at install_dir/db, and is there anything at ~/Hydrus?

>>11260

Thank you for this report. Is there a crash.log or any .db files in install_dir/db? Is your Win7 an updated version, or does it not pull updates?

Unfortunately, I don't have debug builds available, but how about server.exe in the install dir–does this run, or give you a better error? It should be runnable from the console, and print back to it.

>>11263

Yeah, shouldn't be. As you probably know, the options for syncing those are unavailable and have been for a long time. The data record may still exist though. I would say try moving that folder somewhere and then boot the client and see if you can tag on the service(s) they were ever attached to. If it gives you an error, let me know and move the folder back, and I'll improve the error handling for next week.

>>11265

>>11271

>>11282

Thank you for this report. I will have it fixed for next week.

>>11266

I am glad it worked out. Let me know if you run into any trouble setting things up in future.

Damn, 335 is smooth.

Updated from 332 and everything is smoother now.

>>11274

>>11287

If it helps, a filename to look for, either in a different location or for undelete purposes, that would definitely be associated with the old install would be 'client - 2018-12.log', and maybe a bunch of earlier ones like 'client - 2018-6.log' and so on, depending on how long you have been using the software.

>>11277

This is great, thanks for letting me know. Did you have to do anything special to get wxpython to compile right? I've had the biggest trouble getting the new versions going on Ubuntu. I get 40mins of compile and then a -1 error. Seemingly no matter how many gstreamer1.0-dev-bullshit packages I install. I end up just installing the wheel from their site.

>>11281

>>11279

Thanks, yeah, I am getting a bit of this. I have to double-check how I am logging some shit. I thought py3 would be utf-8'ing the whole time, but I think the OS default CP1252 is overriding. I think I just need to tweak the various file open calls. Should be fixed in 336.

>>11305

>Did you have to do anything special to get wxpython to compile right?

The pip way of installing it takes about 5 minutes on my Ryzen 2400G and works fine.

I'm using pip install –user rather than Portage since WxPython 4.x.x isn't in Portage yet (there's an issue on Gentoo's bugzilla tracking a proposed version bump ebuild, but I didn't use that).

> Seemingly no matter how many gstreamer1.0-dev-bullshit packages I install. I end up just installing the wheel from their site.

If it's any consolation, Ubuntu / Ubuntu Server never worked out well for me either. It broke all the damn time, and the way of creating and/or patching packages there was rage-inducing when I tried it.

I guess I'm going to try to compile the ebuild anyhow to see if I can provide any information.

>>11284

Thanks m8. I am feeling better now we sorted >>11240 's problem. Really happy overall with the reports of better speed and so on. These various encoding bug reports are as I expected and should be clearable-up by 336.

Keep on pushing through 2019!

>>11285

Thank you for this report. Is that on Win7? When you start the program, can you watch client.exe's CPU and HDD during that preboot phase before the splash screen appears? Is it doing a lot of work, or idling a lot? Is there any chance an anti-virus program is checking the files being loaded in memory or something? The boot loader is a little different, so maybe it is triggering something the py2 one didn't.

Also, are you running off an HDD or an SSD? If a HDD, is it defragged? If you hit file->restart, is it just as slow again (maybe with high CPU), or is it much faster (suggesting the program was then cached, and this is a hard disk issue)?

That screenshot suggests your public tag repo doesn't have any siblings–can you go services->review services->remote->ptr? Do the numbers look good there, or does it think, say, that it hasn't processed any content yet? Does it have any tags at all? You can take another screenshot of it if you like.

>>11286

Thank you for this report. I will have it fixed for 336.

>>11290

>>11291

Thank you for this update. I am sorry the py3 didn't do it. I have some ideas on the tag dialog and archive/delete filter that I'll be working on in this and coming weeks.

As >>11292 says, are you on source? I am sorry that I forgot if you were. Some Linux users get much better stability when they aren't trying to cross their OS flavour's with my Ubuntu's .so files.

>>11308

Thanks, that's interesting. Don't worry about it too much though–the wheel here:

https://extras.wxpython.org/wxPython4/extras/linux/gtk2/ubuntu-16.04/

Takes about 20s to install and works great for me. I'm just not experienced enough on the Linux side to figure out what wx wants.

>>11313

>Don't worry about it too much though

No worries, doing just this is not much effort on Gentoo.

Well, I compiled the wxpython 4.0.3 ebuild (with pypubsub rdepend commented out, don't need it) and of course uninstalled the version installed by pip.

Took around 14 from extraction to install minutes with my current low-ish system load, works fine with Hydrus.

The ebuild [it's simple, apart from the exact workings of the Gentoo helper functions] plus patches it applies (files/ subdirectory) is here:

https://github.com/TheChymera/overlay/tree/master/dev-python/wxpython

> I'm just not experienced enough on the Linux side to figure out what wx wants

Maybe someone knows if you post the error. I don't think I'll be of much help without the ability to hands-on replicate the error [plus poor skills with Debian's / Ubuntu's package creation method].

>>11310

>Is that on Win7?

Yeah

>When you start the program, can you watch client.exe's CPU and HDD during that preboot phase before the splash screen appears?

Yeah I can see the client.exe.

>Is it doing a lot of work, or idling a lot?

It seems like its running fine I guess, I mean I can see the RAM usage in the processes tab in the task manager slowly going up from a little under 10k and climbing. nothing big or anything. It gets to around 70-80k before I see the Hrdrus logo and everything is running fine The CPU usage goes up and down from 01 to 00. It just seems like its running like a normal program to me before it starts. Its just that it takes a really long time before I see anything outside the task manager to show up.

>Also, are you running off an HDD or an SSD? If a HDD, is it defragged?

I'm using a 3TB external HDD(Don't know why I still use the install version on Hydrus on it). I just got it a few months ago and haven't bothered to defragged it yet.

>If you hit file->restart, is it just as slow again (maybe with high CPU), or is it much faster (suggesting the program was then cached, and this is a hard disk issue)?

Tried it and its doing the same thing.

>Is there any chance an anti-virus program is checking the files being loaded in memory or something?

I wasn't really sure since I use a free version of Kaspersky and never had a problem with it slowing down any of my programs. Just tried pausing my protection and suddenly it worked. So yeah its definitely my anti-virus program slowing down hydrus, fuck. This is like the 3rd time an anti-virus program has messed with any of my programs and having to hunt for a new one.

>That screenshot suggests your public tag repo doesn't have any siblings–can you go services->review services->remote->ptr?

>public

Wait a minute, I think I got something mixed up here. My last pic wasn't meant to be my public tags, it was suppose to be my local tags. I think I got confused, usually when I right click on a tag to take me to my siblings/parents, it'll take me to my local tags. I think I got mixed up since it was the public tab that was shown first(This would explain a lot of things for me actually). So everything is fine really, I just needed to pay attention more.

>>11298

Resolved this issue by installing the latest Windows updates, it seems that one of them is required by some binary file.

>>11322

Great, thanks for letting me know. Maybe you can tell Kaspersky to ignore or treat the hydrus exe/install folder a little differently? Turn off 'active' protection or something so it isn't scanning every time it runs?

>>11325

Great, thanks to you as well for letting me know!

>>11327

UUUU

Also, in this update, getting OS X PyInstaller to make a Framework-compatible Python 3 exe and app was, at times,extremely painful.

>>11328

Thank you, these are excellent. I also get problems with them. I will check them this week and see if I can improve the file parsing.

>>11334

>Great, thanks for letting me know. Maybe you can tell Kaspersky to ignore or treat the hydrus exe/install folder a little differently? Turn off 'active' protection or something so it isn't scanning every time it runs?

Just letting you know I whitelisted Hydrus and it works, thanks.

>>11334

>>11328

Hey, on closer inspection, these were giving me 18KB 'webm' files that were actually some kind of CloudFlare html saying 'we think you are a bot, please solve this captcha'. Once I solved that in my browser, they worked mostly ok in the current dev version (which admittedly now has some unrelated ffmpeg fixes). The /trash/ ones are now 404, but https://i.4cdn.org/gif/1547302893499.webm is giving me a resolution problem that I think I can fix today.

I think part of this problem is the result of the cloudflare-4chan setup giving the wrong http header to say the file was 'webm' when it was really an html interstitial.

The other half is that hydrus has no capability to solve these sorts of captchas. I have never encountered this CF page before on a content fetch, and if it is going to be a new 4chan thing, that's a shame. Let's hope it is a temporary thing that was a misconfiguration or an accidental statistical spike on that set of files due to some mis-aimed DDoS or something. Please let me know if you encounter more of this.

>>11351

Thanks. I'd like to add captcha support, but diving back into the downloader code after all the work I did last year is not my top priority right now. It would be a lot of work, whether I would present login/download captchas to the user in the ui or try to solve them automatically.

Current plan is to do it for the next big iteration on the downloader, whenever that will be. If 4chan or other sites start getting this CF captcha shit a lot, I'll bump it up the schedule, maybe as a hardcoded fix that recognises that CF page specifically.

It is strange that both you and I got the same captcha issue, since I hadn't looked at that content/threads on my own machine/IP previously. That suggests to me the data source was the flagged problem, not the specific client, hence my DDoS-false-positive-on-that-thread suspicion. Again, we'll see if that starts happening more often.

>>11354

Thing is, I saw it fail, checked it, and was able to download it outside of client, then when you told me about the captcha and I checked It I got the captcha too.

No idea what what the hell was going on with that one.