Author Topic: Failure Email after Verify + Sync  (Read 4116 times)

Offline mcao

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
    • View Profile
Failure Email after Verify + Sync
« on: December 02, 2013, 08:59:10 pm »
I just finished creating a new TRAID array by using Verify and Sync and then received an email with the following content:

Code: [Select]
[FAILURE] Verify Sync RAID [External Raid] - Sun Dec 01 13:59:07 PST 2013

But I don't see any mention of errors in the task description below:

Code: [Select]
Verify Sync RAID [External Raid] started at Sun Dec 01 13:59:07 GMT-800 2013

Name: Verify Sync RAID [External Raid]
Start Date: Sun Dec 01 13:59:07 GMT-800 2013
End Date: Mon Dec 02 12:16:42 GMT-800 2013
Duration: 22:17:35
Throughput: 380.331MB/s
Total Size: 29.109TB

976733481 4KB blocks successfully updated

First byte updated at 4096

Last byte updated at 4000778809344

Verify Sync RAID [External Raid] ended at Mon Dec 02 12:16:42 GMT-800 2013

The array is working fine as far as I can tell. So should I be worried about my array or is this a bug?

Offline terrastrife

  • Full Member
  • ***
  • Posts: 235
  • Karma: +7/-0
  • 38TB FlexRAID
    • View Profile
Re: Failure Email after Verify + Sync
« Reply #1 on: December 03, 2013, 02:50:22 am »
"Failure" as in parity blocks had to be updated (which is normal if you're creating an array). The task otherwise did complete properly.

Offline mcao

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
    • View Profile
Re: Failure Email after Verify + Sync
« Reply #2 on: December 03, 2013, 10:44:11 am »
If that's really the case, then the message needs to be more clear, or it shouldn't say failure at all.

Offline mcao

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
    • View Profile
Re: Failure Email after Verify + Sync
« Reply #3 on: December 17, 2013, 05:49:06 pm »
I decided to run another verify just in case and ran into this error:

Code: [Select]
Verify RAID [Internal Raid] started at Tue Dec 17 14:11:01 GMT-800 2013

Name: Verify RAID [Internal Raid]
Start Date: Tue Dec 17 14:11:01 GMT-800 2013
End Date: Tue Dec 17 14:11:06 GMT-800 2013
Duration: 00:00:05
Throughput: 2.910TB/s
Total Size: 14.554TB

Error Code = 8

2 4KB blocks failure

First byte failure at 135409664

Last byte failure at 135446536

Operation aborted!

Failed UoR position = 8

Failed UoR ID = 10025

Verify RAID [Internal Raid] ended at Tue Dec 17 14:11:06 GMT-800 2013

I recall that UoR is 0 indexed, so the 8 means it's my last disk on the array which is the PPU drive. Does this mean the PPU is somehow out of sync? What does error code 8 mean?

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Failure Email after Verify + Sync
« Reply #4 on: December 17, 2013, 08:06:23 pm »
@mcao
I am looking into this issue.
What programs do you have installed on your system that deals with disks?

Offline mcao

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
    • View Profile
Re: Failure Email after Verify + Sync
« Reply #5 on: December 17, 2013, 08:47:27 pm »
@Brahim

I have the Intel Rapid Storage Technology program but it doesn't interface with the storage pool disks, only my SSD system disk. Disk defragmenter is turned off. I don't have any of the individual disks enabled in disk management since I never access the disks directly, only through the storage pool. The only thing I can think of is I have windows indexing turned on for the storage pool drives.

I actually have two storage pools, so I ran the verify on the other pool and got this:

Code: [Select]
Verify RAID [External Raid] started at Tue Dec 17 17:38:07 GMT-800 2013

Name: Verify RAID [External Raid]
Start Date: Tue Dec 17 17:38:07 GMT-800 2013
End Date: Tue Dec 17 17:38:12 GMT-800 2013
Duration: 00:00:05
Throughput: 5.821TB/s
Total Size: 29.109TB

Error Code = 40

1 4KB blocks failure

First byte failure at 135446528

Last byte failure at 135446568

Operation aborted!

Failed UoR position = 8

Failed UoR ID = 10043

Verify RAID [External Raid] ended at Tue Dec 17 17:38:12 GMT-800 2013

Looks like a problem with the same UoR, though a different error code. All the disks are connected to an LSI 9211-8i IT card and act as passthrough.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Failure Email after Verify + Sync
« Reply #6 on: December 18, 2013, 02:34:41 am »
@mcao
Thanks for the info. I have two theories on the issue. So, I am working on something that will provide confirmation of what it truly is.

For those affected, I think the range where the failure occur should be pretty consistent. However, always run the Verify+ as opposed to the Verify so that we have a full report.

Offline mcao

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
    • View Profile
Re: Failure Email after Verify + Sync
« Reply #7 on: December 18, 2013, 11:40:55 pm »
@Brahim

I just ran another Verify + Sync just to see what would happen and got these results:

Code: [Select]
Verify Sync RAID [External Raid] started at Wed Dec 18 08:45:26 GMT-800 2013

Name: Verify Sync RAID [External Raid]
Start Date: Wed Dec 18 08:45:26 GMT-800 2013
End Date: Wed Dec 18 21:29:49 GMT-800 2013
Duration: 12:44:22
Throughput: 665.551MB/s
Total Size: 29.109TB

2 4KB blocks successfully updated

First byte updated at 135446528

Last byte updated at 3356520496

Verify Sync RAID [External Raid] ended at Wed Dec 18 21:29:49 GMT-800 2013

If I'm reading this correctly then only 2 4KB blocks were updated. So I guess it's not as out of sync as I thought. However, I did get another failure email:

Code: [Select]
[FAILURE] Verify Sync RAID [External Raid] - Wed Dec 18 08:45:26 PST 2013

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Failure Email after Verify + Sync
« Reply #8 on: December 19, 2013, 06:05:55 am »
You got the failure email because of the updates that were made.
I have tracked the source is this issue. Working on a fix.

Offline eeze

  • Newbie
  • *
  • Posts: 44
  • Karma: +0/-0
    • View Profile
Re: Failure Email after Verify + Sync
« Reply #9 on: January 25, 2014, 11:22:47 pm »
Mcao,

I also have the same LSI 8i card, running as passthrough to a Windows host on a ESXI 5.5 server.

Each and every time I do a verify+sync, it updates some blocks.

Every time I do a verify, I receive 4KB block errors similar to my note here
http://forum.flexraid.com/index.php/topic,3184.msg22708.html#msg22708

I'm not sure what error code 3832 but this always seems to be my PPU drive (makes sense given i'm verifying parity)

I've been looking around for a log file that'll tell me what files have changed in order to cause the verify to fail. Once I know this, I can figure out if it's a certain application changing some thumbnails (like Plex) or something similar


Offline scottl2k

  • Newbie
  • *
  • Posts: 22
  • Karma: +1/-0
    • View Profile
Re: Failure Email after Verify + Sync
« Reply #10 on: July 30, 2016, 10:34:05 pm »
Is the information in this thread still accurate? 

That is, if I run a Verify/Sync, it will report "failure" even if it's successful, assuming that blocks had to be updated?

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Failure Email after Verify + Sync
« Reply #11 on: August 03, 2016, 09:06:02 am »
The Verify & Sync task will indeed report a failure if blocks had to be updated. That does not mean that the task failed. It is just to bring to the user's attention that blocks had to be updated.

Offline blankstar85

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: Failure Email after Verify + Sync
« Reply #12 on: August 15, 2016, 03:15:26 pm »
I'm just trying to figure out if i'm falling under the same thing here. Here is my Verify Sync:

Code: [Select]
Verify Sync RAID [MediaStorage] started at Sun Aug 14 22:58:14 GMT-400 2016



Name: Verify Sync RAID [MediaStorage]
Start Date: Sun Aug 14 22:58:14 GMT-400 2016
End Date: Mon Aug 15 07:29:07 GMT-400 2016
Duration: 08:30:53
Throughput: 1.215GB/s
Total Size: 36.386TB


Error Code = -1073741668

7168390 stripe blocks failure

First byte failure at 676394583040
Last byte failure at 716766955520

Operation aborted!
Operation aborted due to disk error!

Failed UoR position = 3
Failed UoR ID = 1000000006
Verify Sync RAID [MediaStorage] ended at Mon Aug 15 07:29:07 GMT-400 2016

Any advice? I'm trying to figure out if i have a bad disk or not, I had a disk that was dropped, after removing it i could still read the data in another computer, (even ran a disk check on it and it was fine), but the smart data showed 1,456,000 head parks and 35000 Power on Hours. It was a Western Digital 3tb green drive, I replaced it with a 4tb HGST drive by copying the data from the drive to the new one and then recreating the array. The DRU was 4 that was dropped.

Thanks!


Update: Ok looks like i may have a bad disk, from the event viewer
The device, \Device\Harddisk8\DR8, has a bad block.

Error    8/15/2016 6:13:57 AM   disk   7   None

Just trying to figure out the drive now and run checks. i'll report back..
« Last Edit: August 15, 2016, 03:25:34 pm by blankstar85 »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Failure Email after Verify + Sync
« Reply #13 on: August 15, 2016, 03:25:37 pm »
Your failure is disk error related as per the message.
Check the Windows event viewer for additional clues.

FYI, UoR 3 is the 4th drive in your disk set.

I'm just trying to figure out if i'm falling under the same thing here. Here is my Verify Sync:

Code: [Select]
Verify Sync RAID [MediaStorage] started at Sun Aug 14 22:58:14 GMT-400 2016



Name: Verify Sync RAID [MediaStorage]
Start Date: Sun Aug 14 22:58:14 GMT-400 2016
End Date: Mon Aug 15 07:29:07 GMT-400 2016
Duration: 08:30:53
Throughput: 1.215GB/s
Total Size: 36.386TB


Error Code = -1073741668

7168390 stripe blocks failure

First byte failure at 676394583040
Last byte failure at 716766955520

Operation aborted!
Operation aborted due to disk error!

Failed UoR position = 3
Failed UoR ID = 1000000006
Verify Sync RAID [MediaStorage] ended at Mon Aug 15 07:29:07 GMT-400 2016

Any advice? I'm trying to figure out if i have a bad disk or not, I had a disk that was dropped, after removing it i could still read the data in another computer, (even ran a disk check on it and it was fine), but the smart data showed 1,456,000 head parks and 35000 Power on Hours. It was a Western Digital 3tb green drive, I replaced it with a 4tb HGST drive by copying the data from the drive to the new one and then recreating the array. The DRU was 4 that was dropped.

Thanks!

Offline blankstar85

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: Failure Email after Verify + Sync
« Reply #14 on: August 15, 2016, 07:26:09 pm »
Thanks, saying that it is the 4th disk confirmed exactly what i narrowed it down to. I'm currently coppying what i can from the disk to a newer disk.
This was one of my older 1 tb disks, with 42922 power on hours and had a high pending sector count (192ish).
So far i've lost two files on the copy. I'm hoping to grab them from the virtual disk.

Then i'll delete and recreate the array with the bigger disk and try the verify sync again.

Thanks Brahim!!
« Last Edit: August 15, 2016, 07:28:02 pm by blankstar85 »