Migrating database headaches and questions

Name
Email
Subject
Comment *
File	Select/drop/paste files here
* = required field	[▶ Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field) Spoiler images (this replaces the thumbnails of your images with question marks)
Password	(For file and post deletion.)
Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf Max filesize is 16 MB. Max image dimensions are 15000 x 15000. You may upload 5 per post.

File: 1321546c97bc430⋯.jpg (64.66 KB, 620x398, 310:199, 12868-620x-endless-eight.jpg)

Migrating database headaches and questions Anonymous 10/21/17 (Sat) 19:43:23 39ccc5 No.7038

I tried migrating my database today, but some weird stuff happened:

Initial condition was: 100% disk A (750GB)

I had it set to go: 90% disk A, 10% disk B.

then I pressed "move files now" and waited for a loooong time seeing as the speed between the drives is about 10MB/s because of reasons.

After 7 hours I noticed 2 things:

1. that it was still transferring, but the total size of files on disk B already far surpassed 10%

2. about 30 files had a WindowsError which said they couldn't be transferred, no biggy, I thought, as that's not even 0.01% of all that was transferred.

I decided to push the stop button seeing as I also needed sleep today and more than enough should've been transferred. But when the migration screen updated it said only 3.0GB was on disk B and more files had to be transferred. I checked back with the folder on disk B but there we're certainly enough files etc on there.

I did some checking and now all files have been mirrored but the index of the files is still pointed to the files on disk A. This persisted after a hydrus restart.

What I think happened is that after a WindowsError hydrus gave up on re-indexing the complete folder and deleting the files on disk A (for safety's sake I guess) but still moved the rest of the files in that folder and then moved on to try the next folder. +The average of every file folder is 3GB so it would imply that 1 folder actually did get completely mapped, haven't checked this though.

Now I want to know if there's an easy way to manually move the missing files and re-index the folder instead of trying over and over until no WindowsErrors occur anymore which would be living hell on a 10MB/s transfer speed and 750GB of data with a 5% success rate.(using maths that would take 18 days)

Anonymous 10/21/17 (Sat) 19:47:17 39ccc5 No.7040

I'm on version 178 btw

hydrus_dev ## Board Owner 10/21/17 (Sat) 20:58:05 1f99a3 No.7042

File: ea09f40bb35e401⋯.jpg (3.07 MB, 3781x2608, 3781:2608, ea09f40bb35e401f28fc14f2a0….jpg)

I am sorry you have had problems.

Does your client boot ok atm? Do you have missing files? If you have a backup, please consider restoring back to the situation you know works.

If the hydrus transfer is giving your WindowsErrors, that doesn't sound good. Did it give you those errors in a popup? If so, they should have been written to your log. Please check install_dir/db/client (date).log and scroll to the bottom. It would be useful to see the tracebacks, if you could post them here. Or email them, if they private data or whatever.

Now the transfer has stopped, is the 3GB or so the client thinks it has moved to B correct? Is the total size of the written data about that big? I am not sure where the extra data you saw during the transfer would have come from.

If you have a backup, so this is not risky, you may like to just do the move manually, while the client is closed. When it next boots, it should complain that it cannot find some folders and give you a dialog where you can fix it by telling it the new paths. This isn't as pretty as the db migration dialog, but it seems your disk maybe has some other issues that need the manual touch.

You've seen a bit inside client_files. The folders beginning with 'f' are your files, and 't' and 'r' are your full-size and resized thumbnails respectively. There are 256 of each, from 00 to ff. If you want to move 10%, try moving 25 of each and then booting the client, fixing the locations, and then fixing the ideal weights retroactively in the database migration dialog.

Please let me know how it goes.

Anonymous 10/21/17 (Sat) 21:50:51 39ccc5 No.7044

File: f26ab1f56b0eaa9⋯.png (91.62 KB, 2085x884, 2085:884, whatitlookslike.png)

>>7042

The client boots just fine for me, the errors were in a popup yes, I copy/pasted one below.

My f89 folder was the only one that actually got indexed and everything seems to be working fine with it. The rest of the folders on disk B you can see are also populated but not indexed by hydrus and are exact replicas of the folders on disk A minus 1 or 2 files.

I'll try doing what you described by renaming the folders so no data is lost when testing, will report back with what happens.

Here's one of the errors:

2017/10/21 20:45:39: Moving 'f16' from [personal data]\Hydrus Network\db\client_files to [personal data]\the_wired\HYDRUS

2017/10/21 20:56:12: Exception:

2017/10/21 20:56:12: WindowsError

[Error 5] Access is denied: u'[personal data]\\Hydrus Network\\db\\client_files\\f16\\16106782e6a6d2a1b0ad7f81f111183bd13a806af998a04e3914f30277327189.jpg'

Traceback (most recent call last):

File "include\HydrusDB.py", line 526, in _ProcessJob

result = self._Write( action, *args, **kwargs )

File "include\ClientDB.py", line 10449, in _Write

elif action == 'relocate_client_files': result = self._RelocateClientFiles( *args, **kwargs )

File "include\ClientDB.py", line 7990, in _RelocateClientFiles

HydrusPaths.MergeTree( full_source, full_dest )

File "include\HydrusPaths.py", line 484, in MergeTree

shutil.move( source, dest )

File "shutil.py", line 300, in move

File "shutil.py", line 252, in rmtree

File "shutil.py", line 250, in rmtree

WindowsError: [Error 5] Access is denied: u'[personal data]\\Hydrus Network\\db\\client_files\\f16\\16106782e6a6d2a1b0ad7f81f111183bd13a806af998a04e3914f30277327189.jpg'

Anonymous 10/22/17 (Sun) 00:03:18 39ccc5 No.7045

Just finished making sure everything was copied correctly and restarted the client, it asked for the file locations like you described and overall it worked amazing!

Would be nice if the option could be described in the faq for future reference, although manually doing stuff is never praised upon, sometimes it's the only way!

Thanks for the help dev, if you need more logs I don't mind sending em.

hydrus_dev ## Board Owner 10/23/17 (Mon) 22:06:08 1f99a3 No.7067

>>7045

>>7044

I am glad this ultimately got sorted. I will make a note of this error and see if I can handle it any better.

Do you have any idea what might be special about the 1610678….jpg image in that error, or any of the others? Might they have been originally saved/created by an admin user–someone that your user does not have access to overwrite? Or are they simply read-only, if you right-click->properties them? I am wondering if my code just isn't catching and writeable-ing them correctly.