Author Topic: Trying to restore a DRU and getting the "too many failed devices" error.  (Read 184 times)

Offline tonyr80

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Hi there,

Any assistance with the following would be greatly appreciated.

I have a 8 DRU 1 PPU setup, all are same size drives (drive config screenshot attached), running on Win server 2012 standard, I set it up about 15 months ago (did a few simulated failures/swap outs etc which all worked fine) and had largely forgotten about the system

The system is used as a media server with pretty static files (no edits or moves etc), I noticed earlier this week that a load of my movies had disappeared in Plex, which I went about investigating. 

I realised the files were no longer in the storage pool, so I went through the various drives and realised one (DRU5) was only showing a handful of files on it, despite reporting reporting it being 98% full.  The same drive was also getting Reallocated Sectors Count errors on the SMART tests, so I assumed it was dud and swapped it out for a fresh drive.

Going through the swap process/restore process, the drive was swapped into the pool fine but when attempting to restore the error "ERROR: [restore] error: IOException[Too many failed devices! Failed=3 - Number of devices supported to be restored at a time: 1] java.io.IOException: Too many failed devices! Failed=3" occurs.

I've looked a few other threads in regards to this but the solutions/steps taken didn't work or help for me unfortunately.

I've attached the log (I believe in TRACE mode, zipped it as the log is huge) and screenshot of the drive configuration, any help in restoring the files or understanding what is wrong with my config would be appreciated, any extra information required please let me know.

Tony.


Offline adridolf

  • Jr. Member
  • **
  • Posts: 87
  • Karma: +0/-0
    • View Profile
Re: Trying to restore a DRU and getting the "too many failed devices" error.
« Reply #1 on: September 02, 2017, 10:12:07 am »
I had a somewhat similar phenomenon with ReFS. However, I assume you use NTFS?

Offline tonyr80

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Trying to restore a DRU and getting the "too many failed devices" error.
« Reply #2 on: September 02, 2017, 10:22:51 am »
I do indeed yes.

Online Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,367
  • Karma: +200/-15
    • View Profile
Re: Trying to restore a DRU and getting the "too many failed devices" error.
« Reply #3 on: September 05, 2017, 08:57:24 am »
@tonyr80
Please check your files systems aren't corrupted.
Navigate to C:\FlexRAID-Managed-Pool\class1_0\ and ensure that every folder there is disk mounted and that you can properly navigate the file system.

Offline tonyr80

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Trying to restore a DRU and getting the "too many failed devices" error.
« Reply #4 on: September 05, 2017, 04:41:04 pm »
Hi Brahim,

Thanks for your reply!

Unfortunately when I try navigate to that folder I get a "Windows can't find C:\FlexRAID-Managed-Pool\class1_0\ check the spelling and try again.

When I try navigate to "C:\FlexRAID-Managed-Pool\" instead, I get an Access Denied message, despite being logged on as administrator and that being the only account on the system...

Any ideas?

Tony.



Online Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,367
  • Karma: +200/-15
    • View Profile
Re: Trying to restore a DRU and getting the "too many failed devices" error.
« Reply #5 on: September 07, 2017, 10:57:28 am »
Edit the security settings for the C:\FlexRAID-Managed-Pool folder and grant your user account permission to access it.

Offline tonyr80

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Trying to restore a DRU and getting the "too many failed devices" error.
« Reply #6 on: September 07, 2017, 11:12:11 am »
My apologies, I should have been able to figure that one out myself, I had to uncheck "Hide system files" etc. before I could see the folder and modify permissions accordingly.

Within the folder there are a load of shortcuts presumably corresponding to the various DRUs, they all open fine and have all the files/folders matching what I can see when navigating via the FlexRAID web interface using the "view drive contents" feature in the pool browser.  I can open them all fine and access the files correctly.  The new drive is in there also with a couple movies in which I've recently added to the pool (is it ok to keep adding stuff to the pool when in this state or might I be causing extra complications?).

Tony.


Online Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,367
  • Karma: +200/-15
    • View Profile
Re: Trying to restore a DRU and getting the "too many failed devices" error.
« Reply #7 on: September 07, 2017, 01:44:38 pm »
Make sure those shortcut are disk mounted.

The logs clearly spell out what data it sees as missing. Just look at the files flagged and make sense of them.
It could be that you moved or deleted too many files from those DRUs causing the parity to be invalid for restore purposes.

Offline tonyr80

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Trying to restore a DRU and getting the "too many failed devices" error.
« Reply #8 on: September 08, 2017, 10:59:07 am »
All the files found in each of the shortcuts are mounted and accessible in the pool.

Having gone through the log as you suggested I gradually realised that all the files listed as

"WARN : Missing: D:2:0\xxx\xxx"  (or D:1:0) are files that were deleted, mostly last year when I did a big clean up.

Whereas all the :

"[2017-09-02 11:05:42,005] WARN : Missing: D:3:0\xxx\xxx" are the files that have been lost with the hard drive (I actually have the dud drive in my personal computer now, which can read it, but borks whenever I try copy or access a file).

So all the files I deleted last year are being detected as a failed drive, am I understanding that correctly?  I sense if that's the case then the actual failed drive files aren't going to be recoverable, what I have been doing wrong(or not doing at all)?

Online Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,367
  • Karma: +200/-15
    • View Profile
Re: Trying to restore a DRU and getting the "too many failed devices" error.
« Reply #9 on: September 08, 2017, 11:04:22 am »
Snapshot RAID requires that you Update the parity after big data changes. This is unless you have the recycle bin feature enabled.

The fact that the RAID was not updated since "last year"  means the parity has not bee valid for just as long.

Please read: http://wiki.flexraid.com/2011/10/18/understanding-the-limitations-of-snapshot-raid/

Offline tonyr80

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Trying to restore a DRU and getting the "too many failed devices" error.
« Reply #10 on: September 08, 2017, 11:15:35 am »
Thanks, I had read that but the implications of usage hadn't really dawned on me fully, when I setup the system I was in the middle of a big work project and was probably a bit lackadaisical in my approach.

Is there anyway of doing even a partial restore of some of the files?   If no, no worries, and thanks for your help once again, I'll make sure to re-sync the parity after any file deletes etc moving forward.

Tony.

Online Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,367
  • Karma: +200/-15
    • View Profile
Re: Trying to restore a DRU and getting the "too many failed devices" error.
« Reply #11 on: September 08, 2017, 12:43:05 pm »
If you can find the deleted files (backups, re-download, etc.), that will do the trick. The more of the missing files you can find, the more you can restore of the data.