Author Topic: How to get Notification alert "Missing Drive" in pool and prevent update tasks?  (Read 3362 times)

Offline TheJSN

  • Newbie
  • *
  • Posts: 14
  • Karma: +0/-0
    • View Profile
Hello, I've been using Flexraid for years, however i've never had it save my data yet.  The most common occurance is that a drive fails in the pool and goes "missing"  then my schedule update runs, and I'm no longer able to restore any data from the "missing" drive.

Is there a way to get a notification alert, for when a drive goes missing in the pool?   Most often its the update process that hits a bad sector or something that sets the drive offline, update should abort at this point.  So the rest of the data can be saved? 

am I doing something wrong?  The most common failure is that a drive goes, my pool shows it as missing and a recovery operation gives me a blank disc.

I do a fully verify monthly, and a validate and update once a week.  I have never received a note that a drive has gone missing in the pool, I just usually notice because a number of media files are just gone from my Plex or Kodi.  At this point I check to see flexraid is missing a disk.
« Last Edit: July 10, 2016, 11:45:00 am by TheJSN »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
The Update task will abort if a drive fails.
In all cases, drive health monitoring is provided through the SMART monitoring feature. You can configure it to send you SMS and/or email notifications.

Offline TheJSN

  • Newbie
  • *
  • Posts: 14
  • Karma: +0/-0
    • View Profile
But when a drive is missing (ie. unplugged, or off because its failed).  if I try to view files on the drive it shows as blank, and I can't recover from it. The recovery shows successful, with no files recovered.  Is there no way to have the pool alert if a drive goes missing?  SMART monitoring doesn't seem to indicate a drive is gone/missing/offline.  I think if a drive is offline, the smart check just doesn't happen and no alerts are sent.


[2016-07-07 19:12:36,050] INFO : 1262 empty files were recovered! Exiting...
[2016-07-07 19:12:36,050] INFO : [restore] completed at: Thu Jul 07 19:12:36 EDT 2016
[2016-07-07 19:12:39,716] INFO : Sending email alert - message=[SNAP-RAID1][RESTORE] SUCCESS
[2016-07-07 19:12:45,624] INFO : [SNAP-RAID1][RESTORE] SUCCESS - emailed...
[2016-07-08 11:10:36,847] INFO : Unmounting View...

Maybe I'm just unlucky, but in every failure i've had.   I see files missing in my kodi/plex.  Then I check the server, go to the pool, see a drive is missing.  Try to recover from it. and it recovers nothing.  When I do view files on the disk from within the web interface it always shows as blank.  I suspect updates/validates happened with the drive offline and everything seems normal.  The files are all just missing as if they weren't there during the update.  which is likely true cuz the drive was offline and I didn't know about it.
« Last Edit: July 11, 2016, 03:44:35 pm by TheJSN »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
If you wish to have access to all files when a drives goes missing, you need to switch to tRAID.
With RAID-F, the files are to be brought back online or recovered.

Your logs snippet shows that recovery was successful. Any new file after your last Snapshot update will not be recovered since it is not part of the parity.

Offline TheJSN

  • Newbie
  • *
  • Posts: 14
  • Karma: +0/-0
    • View Profile
Yes I realize that the files will not be available with Snapshot, however my point is that Once a drive goes missing in my pool. I can not recover anything from it.  Its empty.  and the files show as empty.  I suspect its because an update or validate has happened AFTER the drive went offline.  So even Snapshot will not allow me to recover the missing drive, since it believes the drive was blank based on the information it has.

1) I believe that if the drive goes offline (prior to a validation or update) I can recover the drive - but since i can't receive a notification when this happens, I am always too late.
2) I believe that if the drive goes offline prior to or during a validation/update procedure, it treats the missing drive as if it was all empty. the files are just gone, and the parity information is updated accordingly.   When I finally notice that the drive is offline.  I am not able to recover anything from it, because it thinks the drive was blank.

am I missing something in my setup here?


Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
No. Again, the Update task will abort when a drive goes missing. This is very easy to test. If you see a different behavior, simply setup a test array to prove it. Take a screencast and share it.
This is a rather fundamental feature.

If you need help with what might have gone wrong with your setup, start by detailing it and post full logs.

Offline horus

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
    • View Profile
I think i might have the same problem.

http://forum.flexraid.com/index.php/topic,48918.0.html

I'm also kinda annoyed i don't get a message when a drive is missing, it only seems to work when an operation fails or smart fails. But a missing disk doesn't seem to be included.....
« Last Edit: July 21, 2016, 02:16:42 pm by horus »

Offline racquemis

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
its seems this functionally is broken.

Today i noticed the update task was busy, 45 minutes had passed when i didnt change anything on my array.
Looking at my array DRU2 was missing. The automatic update didn't abort rendering my parity useless. I aborted it manually.

I recovered the missing DRU, problem was a detached cable, and ran the update again.This time it took 6 minutes to complete the update at the same speed. This doesn't make sense. It means there is still parity data frōm the situation wnere the drive was missing.

also my array now shows less data on the PPU than the fullest DRU.
Something is fundamentally broken
See the log attached.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
@racquemis
Check your PM.

Offline TdiDave

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
    • View Profile
This seems to what i had happen. I had no idea a drive had dropped out until i my media server gave a error that it could not find a file. When i investigated i found the drive was dead, simple fix i thought. Wrong as i have replaced the bad drive and get "no files to restore".  The parity updated with a drive from my array dropped, so it looks like i lost 3 TB of data.

Did you come up with any solution?

Offline TdiDave

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
    • View Profile
its seems this functionally is broken.

Today i noticed the update task was busy, 45 minutes had passed when i didnt change anything on my array.
Looking at my array DRU2 was missing. The automatic update didn't abort rendering my parity useless. I aborted it manually.

I recovered the missing DRU, problem was a detached cable, and ran the update again.This time it took 6 minutes to complete the update at the same speed. This doesn't make sense. It means there is still parity data frōm the situation wnere the drive was missing.

also my array now shows less data on the PPU than the fullest DRU.
Something is fundamentally broken



This seems to what i had happen. I had no idea a drive had dropped out until i my media server gave a error that it could not find a file. When i investigated i found the drive was dead, simple fix i thought. Wrong as i have replaced the bad drive and get "no files to restore".  The parity updated with a drive from my array dropped, so it looks like i lost 3 TB of data.

Did you come up with any solution?
See the log attached.

Offline racquemis

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
It happened again!

DRU2 went missing, found flexraid happily updating parity as scheduled(essentially rebuilding everything without the dru2 data)

Please fix this bug.

Offline smarty12345

  • Newbie
  • *
  • Posts: 27
  • Karma: +0/-0
    • View Profile
What version of Flexraid are you using. The early version of 2.1 had this issue. And though it was supposedly resolved, I have never tried later versions as they seem to also have issue. I use version 2.0 and I have tested that by removing drives and it does detect a missing drive and not do an update.

Offline justincase102

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
This post plus a few others made me a bit concerned about a update occurring when a drive goes offline.  I don't baby sit my server and expect software like this to do that for me.

I've not been able to reproduce a similar issue during my testing. I created a test config and simulated a dropped drive and sure enough the update didn't occur.  I did the same with my production rig, dropped different drus and the update didn't occur in any case.

I realize this doesn't help but there are few people that have reported a similar issue where an update occurs even though a dru is no longer online.  I'm wondering if there is a scenario, perhaps software related, perhaps a confluence of events that causes an update even though a dru has gone offline?

I'm running the same version of flexraid 20151101 that racquemis is running.

Offline TheJSN

  • Newbie
  • *
  • Posts: 14
  • Karma: +0/-0
    • View Profile
I have had this happen a few times, A drive goes offline, I go to recover and the missing drive content looks blank, or recovers nothing.   I think the problem may be when the drive drops during the update.  I believe i have had drive failures where the drive is fine until it hits bad sectors or bad areas of the drive then goes offline during a read process.  I think if a drive goes offline, before the update starts maybe it will prevent the update?    I think its the same situation, if I'm running my drives in a pool and my drive goes missing, The only way I can tell is files I know should be there just appear to be gone/deleted.  I think if a drive fails, the update process see's the same thing, those files are deleted, and them omitted from the parity. The update completes/or doesn't complete (not sure how rollback works) without those files.  When you go to restore the disk, its empty.


Is there no way to get a notification if a drive goes offline?  I didn't find a way.  I always notice some files are missing, then i manually look at the array and a drive is missing.  If this functionality isn't rock solid, i do not believe it can reliably protect our data.
« Last Edit: February 10, 2017, 10:30:22 am by TheJSN »