Author Topic: Unable to Restore, help!  (Read 373 times)

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,504
  • Karma: +201/-16
    • View Profile
Re: Unable to Restore, help!
« Reply #15 on: December 03, 2017, 02:07:50 pm »
So, DRU5 failed but you added it back as DRU8. Correct?

Enable TRACE logging (http://wiki.flexraid.com/2011/03/26/how-to-enable-more-verbose-logging/) and redo the restore operation so that there is more details to analyze. Then post the full logs here.

Offline imadork8317

  • Jr. Member
  • **
  • Posts: 98
  • Karma: +0/-0
    • View Profile
Re: Unable to Restore, help!
« Reply #16 on: December 03, 2017, 06:33:44 pm »
This is the error i got when trying to do a swap, for some reason it is trying to restore to the C Drive
C:\FlexRAID-Managed-Pool\class1_0\{338312b0-3cdf-4d6a-b566-958a61e2cbda}

[2017-12-03 16:39:55,459] WARN : Issues detected on DRU1
[2017-12-03 16:39:55,459] WARN :                Failed = false
[2017-12-03 16:39:55,459] WARN :                Changed = 0
[2017-12-03 16:39:55,459] WARN :                Corrupted = 0
[2017-12-03 16:39:55,459] WARN :                Missing = 36
[2017-12-03 16:39:55,459] WARN :                Failed Size = 30985826684
[2017-12-03 16:39:55,459] WARN : Issues detected on DRU2
[2017-12-03 16:39:55,459] WARN :                Failed = false
[2017-12-03 16:39:55,459] WARN :                Changed = 0
[2017-12-03 16:39:55,459] WARN :                Corrupted = 0
[2017-12-03 16:39:55,459] WARN :                Missing = 2
[2017-12-03 16:39:55,459] WARN :                Failed Size = 339
[2017-12-03 16:39:55,459] WARN : Issues detected on DRU3
[2017-12-03 16:39:55,459] WARN :                Failed = false
[2017-12-03 16:39:55,459] WARN :                Changed = 0
[2017-12-03 16:39:55,459] WARN :                Corrupted = 0
[2017-12-03 16:39:55,459] WARN :                Missing = 20
[2017-12-03 16:39:55,459] WARN :                Failed Size = 16659970774
[2017-12-03 16:39:55,459] WARN : Issues detected on DRU4
[2017-12-03 16:39:55,459] WARN :                Failed = false
[2017-12-03 16:39:55,459] WARN :                Changed = 0
[2017-12-03 16:39:55,459] WARN :                Corrupted = 0
[2017-12-03 16:39:55,459] WARN :                Missing = 20
[2017-12-03 16:39:55,459] WARN :                Failed Size = 27102111832
[2017-12-03 16:39:55,459] WARN : Issues detected on DRU6
[2017-12-03 16:39:55,459] WARN :                Failed = false
[2017-12-03 16:39:55,459] WARN :                Changed = 0
[2017-12-03 16:39:55,459] WARN :                Corrupted = 0
[2017-12-03 16:39:55,459] WARN :                Missing = 51
[2017-12-03 16:39:55,459] WARN :                Failed Size = 46904493872
[2017-12-03 16:39:55,459] WARN : Issues detected on DRU7
[2017-12-03 16:39:55,459] WARN :                Failed = false
[2017-12-03 16:39:55,459] WARN :                Changed = 0
[2017-12-03 16:39:55,459] WARN :                Corrupted = 0
[2017-12-03 16:39:55,459] WARN :                Missing = 1
[2017-12-03 16:39:55,459] WARN :                Failed Size = 1762131968
[2017-12-03 16:39:55,459] WARN : Issues detected on PPU1
[2017-12-03 16:39:55,459] WARN :                Failed = false
[2017-12-03 16:39:55,459] WARN :                Changed = 2385
[2017-12-03 16:39:55,459] WARN :                Corrupted = 0
[2017-12-03 16:39:55,459] WARN :                Missing = 0
[2017-12-03 16:39:55,459] WARN :                Failed Size = 2560874250240
[2017-12-03 16:40:23,961] WARN : WARN: The restore process has failed! Nothing has been restored...
[2017-12-03 16:40:23,961] ERROR: [restore] error: IOException[There is not enough space on the disk]
java.io.IOException: There is not enough space on the disk
        at java.io.RandomAccessFile.<unknown>(Unknown Source)
        at com.tchegbe.lib.io.a.g.f.a(Unknown Source)
        at com.tchegbe.lib.io.a.g.f.a(Unknown Source)
        at com.tchegbe.lib.io.a.e.t.a(Unknown Source)
        at com.tchegbe.lib.io.a.e.c.a.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
[2017-12-03 16:40:23,961] INFO : [restore] completed at: Sun Dec 03 16:40:23 PST 2017
[2017-12-03 16:40:26,182] INFO : Sending email alert - message=[Plex][RESTORE] FAILURE
[2017-12-03 16:40:28,762] INFO : [Plex][RESTORE] FAILURE - emailed...

I'll run the restore again with TRACE and post back, each restore attempt takes several hours.
« Last Edit: December 03, 2017, 06:35:19 pm by imadork8317 »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,504
  • Karma: +201/-16
    • View Profile
Re: Unable to Restore, help!
« Reply #17 on: December 03, 2017, 07:14:25 pm »
1. Even if successful, you are going to have a number of corrupted files given the many missing files across your surviving DRUs.

2. A disk should be mounted on C:\FlexRAID-Managed-Pool\class1_0\{338312b0-3cdf-4d6a-b566-958a61e2cbda}. You should verify that by going to C:\FlexRAID-Managed-Pool\class1_0 and ensuring that {338312b0-3cdf-4d6a-b566-958a61e2cbda} is a disk mounted folder. You will need to edit the permission on C:\FlexRAID-Managed-Pool to grant yourself access to it. You will also need to enable the viewing of hidden and system files.

3. As recommended before, next time you restore, restore to a disk outside of the array so that you have a bit more control over things.

Offline imadork8317

  • Jr. Member
  • **
  • Posts: 98
  • Karma: +0/-0
    • View Profile
Re: Unable to Restore, help!
« Reply #18 on: December 04, 2017, 12:31:02 am »
I think I am seeing better results this time, now sure why.  I hijacked DRU8 and mounted it to a drive letter and TRACE logs show attempts to restore files, however as you stated there does seem to be some corruption, so I will have to see how much damage has been done.

Offline imadork8317

  • Jr. Member
  • **
  • Posts: 98
  • Karma: +0/-0
    • View Profile
Re: Unable to Restore, help!
« Reply #19 on: December 04, 2017, 09:57:36 am »
TRACE Logs are too big to upload, Not enough space error again, but I am not sure why.  What is the "Failed size" a reference to? Is that how much space I need on the respective DRU?

[2017-12-04 03:28:49,008] WARN : Issues detected on DRU1
[2017-12-04 03:28:49,008] WARN :                Failed = false
[2017-12-04 03:28:49,008] WARN :                Changed = 0
[2017-12-04 03:28:49,008] WARN :                Corrupted = 0
[2017-12-04 03:28:49,008] WARN :                Missing = 36
[2017-12-04 03:28:49,008] WARN :                Failed Size = 30985826684
[2017-12-04 03:28:49,008] WARN : Issues detected on DRU2
[2017-12-04 03:28:49,008] WARN :                Failed = false
[2017-12-04 03:28:49,009] WARN :                Changed = 0
[2017-12-04 03:28:49,009] WARN :                Corrupted = 0
[2017-12-04 03:28:49,009] WARN :                Missing = 2
[2017-12-04 03:28:49,009] WARN :                Failed Size = 339
[2017-12-04 03:28:49,009] WARN : Issues detected on DRU3
[2017-12-04 03:28:49,009] WARN :                Failed = false
[2017-12-04 03:28:49,009] WARN :                Changed = 0
[2017-12-04 03:28:49,009] WARN :                Corrupted = 0
[2017-12-04 03:28:49,009] WARN :                Missing = 20
[2017-12-04 03:28:49,009] WARN :                Failed Size = 16659970774
[2017-12-04 03:28:49,009] WARN : Issues detected on DRU4
[2017-12-04 03:28:49,009] WARN :                Failed = false
[2017-12-04 03:28:49,009] WARN :                Changed = 0
[2017-12-04 03:28:49,009] WARN :                Corrupted = 0
[2017-12-04 03:28:49,009] WARN :                Missing = 20
[2017-12-04 03:28:49,009] WARN :                Failed Size = 27102111832
[2017-12-04 03:28:49,009] WARN : Issues detected on DRU6
[2017-12-04 03:28:49,009] WARN :                Failed = false
[2017-12-04 03:28:49,009] WARN :                Changed = 0
[2017-12-04 03:28:49,009] WARN :                Corrupted = 0
[2017-12-04 03:28:49,009] WARN :                Missing = 51
[2017-12-04 03:28:49,009] WARN :                Failed Size = 46904493872
[2017-12-04 03:28:49,009] WARN : Issues detected on DRU7
[2017-12-04 03:28:49,009] WARN :                Failed = false
[2017-12-04 03:28:49,009] WARN :                Changed = 0
[2017-12-04 03:28:49,009] WARN :                Corrupted = 0
[2017-12-04 03:28:49,009] WARN :                Missing = 1
[2017-12-04 03:28:49,009] WARN :                Failed Size = 1762131968
[2017-12-04 03:28:49,009] WARN : Issues detected on PPU1
[2017-12-04 03:28:49,009] WARN :                Failed = false
[2017-12-04 03:28:49,009] WARN :                Changed = 2385
[2017-12-04 03:28:49,009] WARN :                Corrupted = 0
[2017-12-04 03:28:49,009] WARN :                Missing = 0
[2017-12-04 03:28:49,009] WARN :                Failed Size = 2560874250240
[2017-12-04 03:28:49,420] ERROR: [restore] error: IOException[Not enough space to restore D:4:0\ISO Files\Blueray ISOs\R
esident Evil.iso]
java.io.IOException: Not enough space to restore D:4:0\ISO Files\Blueray ISOs\Resident Evil.iso
        at com.tchegbe.lib.io.a.e.t.a(Unknown Source)
        at com.tchegbe.lib.io.a.e.c.a.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
[2017-12-04 03:28:49,422] INFO : [restore] completed at: Mon Dec 04 03:28:49 PST 2017
[2017-12-04 03:28:50,065] INFO : Sending email alert - message=[Plex][RESTORE] FAILURE
[2017-12-04 03:28:52,829] INFO : [Plex][RESTORE] FAILURE - emailed...

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,504
  • Karma: +201/-16
    • View Profile
Re: Unable to Restore, help!
« Reply #20 on: December 04, 2017, 11:50:43 am »
TRACE Logs are too big to upload, Not enough space error again, but I am not sure why.  What is the "Failed size" a reference to? Is that how much space I need on the respective DRU?
...
I am pretty sure you did not quick format the "restore to" drive as I recommended. A drive can lose usable space over time due to bad sectors or whatnot.

Offline imadork8317

  • Jr. Member
  • **
  • Posts: 98
  • Karma: +0/-0
    • View Profile
Re: Unable to Restore, help!
« Reply #21 on: December 04, 2017, 09:52:05 pm »
I did a quick format for this next time Restore Pre-Validation Process finished this time, which i dont think it ever did before!

Name: Restore Pre-Validation Process
Start Date: Mon Dec 04 08:17:33 GMT-800 2017
End Date: Mon Dec 04 12:51:36 GMT-800 2017
Duration: 04:34:02
Throughput: 3016.18 MB/s

its currently 78% through parity computation process, probably has another 2-3 hours to go.

logs look to be in processing phase

[2017-12-04 20:10:48,026] TRACE: Processing P:0:1\004842.flxr

This looks to be a good sign!

Offline imadork8317

  • Jr. Member
  • **
  • Posts: 98
  • Karma: +0/-0
    • View Profile
Re: Unable to Restore, help!
« Reply #22 on: December 05, 2017, 01:44:04 am »
So it finished but it said everything is corrupt?? How is that possible?

The validation process is reporting 0 new, 0 changed, 0 deleted, and 132454 corrupted files!
WARN: 132454 corrupted restored files...

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,504
  • Karma: +201/-16
    • View Profile
Re: Unable to Restore, help!
« Reply #23 on: December 05, 2017, 09:06:13 am »
So it finished but it said everything is corrupt?? How is that possible?

The validation process is reporting 0 new, 0 changed, 0 deleted, and 132454 corrupted files!
WARN: 132454 corrupted restored files...

Quote
[2017-12-04 03:28:49,009] WARN : Issues detected on PPU1
[2017-12-04 03:28:49,009] WARN :                Failed = false
[2017-12-04 03:28:49,009] WARN :                Changed = 2385
[2017-12-04 03:28:49,009] WARN :                Corrupted = 0
[2017-12-04 03:28:49,009] WARN :                Missing = 0
[2017-12-04 03:28:49,009] WARN :                Failed Size = 2560874250240

On top of having missing files, your PPU data was somehow modified. Those were deliberate changes and not silent corruptions.
More likely than not, you never had an actual failed disk. You just screwed up your configuration and tried to restore data you had deleted.
Basically, DRU8 was DRU5 all along and you are not telling what really happened to it to cause you to want to recover it.

Offline imadork8317

  • Jr. Member
  • **
  • Posts: 98
  • Karma: +0/-0
    • View Profile
Re: Unable to Restore, help!
« Reply #24 on: December 05, 2017, 09:48:53 am »
I absolutely have a failed disk.  It will not spin up.

Is there a way to remove DRU8?

Here is the timeline
11/2 | I bought a new 8tb drive
11/4 | I followed this guide as i usually do http://wiki.flexraid.com/2013/02/13/replacing-swapping-out-a-non-failed-drive-for-another-drive/ swapping out the 4TB W300HE9R with the new 8TB drive
11/5 -11/29 | Scheduled update job ran as it does every night
11/30 -12/2 | Drive S3014V5Y Failed
12/2 | Update Failed



Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,504
  • Karma: +201/-16
    • View Profile
Re: Unable to Restore, help!
« Reply #25 on: December 05, 2017, 10:04:44 am »
The issue now is that DRU5 and DRU8 have the same volume signature. For this to happen, it not only has to be the same disk but also the same volume on that same disk.

Did you at some point tried to use DRU8 as the replacement disk for DRU5 because it was large enough to have the free space even though it was already in the pool?

FYI, removing DRU8 is not necessary as it is part of the parity. Removing DRU8 would be RAID contraction, and RAID contraction when you have a failed disk is a silly thing to attempt. Luckily, the UI won't let you do it.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,504
  • Karma: +201/-16
    • View Profile
Re: Unable to Restore, help!
« Reply #26 on: December 05, 2017, 10:24:04 am »
Also, per your screenshots, DRU8 is only 4TB. So, that 8TB new disk never made it to the array. What happened to it and where is it?

Offline imadork8317

  • Jr. Member
  • **
  • Posts: 98
  • Karma: +0/-0
    • View Profile
Re: Unable to Restore, help!
« Reply #27 on: December 05, 2017, 10:38:59 am »
there were 2 disk in DRU5 one was replaced by the new 8TB drive, the other is the one that failed.  DRU8 was part of DRU5 as it was the drive replaced by the 8TB drive.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,504
  • Karma: +201/-16
    • View Profile
Re: Unable to Restore, help!
« Reply #28 on: December 05, 2017, 11:09:23 am »
there were 2 disk in DRU5 one was replaced by the new 8TB drive, the other is the one that failed.  DRU8 was part of DRU5 as it was the drive replaced by the 8TB drive.

That 8TB is still nowhere to be found in your configuration. So, the question of its whereabout lingers.

Back to my previous assumption, that 4TB never failed as evidenced by the fact that DRU5 is still made up of two 4TB drives. I suspect that you were reconfiguring your array by contracting and expanding and things did not go as planned. It looks like you expanded the array with a disk already in the pool (one of the DRU5 spans). Next time you wish to reconfigure the array, simply delete the current configuration, regain control of your disks, create a new configuration with the disks laid out the way you want, and compute new parity.

So, delete the configuration, create a new configuration, compute new parity, and protect your current disks.

Offline imadork8317

  • Jr. Member
  • **
  • Posts: 98
  • Karma: +0/-0
    • View Profile
Re: Unable to Restore, help!
« Reply #29 on: December 05, 2017, 11:34:43 am »
My apologies, I was mistaken by which drive was replaced, it was not a drive from DRU5 it was from DRU6.

Drive "P:\ST4000-W300HE9R" (DRU6) used to be a 4TB drive (W300HE9R) and was swapped out with an 8TB drive and is in good working order.

DRU5 has 2 4TB drives, S3014V51 is good, S3014V5Y failed and was pulled out and is sitting here on my desk as it will not boot.

I attempted to swap S3014V5Y (failed drive) with W300HE9R (swapped drive), this restore process failed.
So then I added W300HE9R as DRU8.

Understandably this is confusing.

To add to the confusion I believe the scheduled update job ran after the disk failed or maybe it failed during an update process which may be why some PPU data is invalid but I do not understand why ALL of the data looks to be corrupted. 

Is there any way at this point to recover the 4TB of data?
I am not sure how to get back to a good configuration.