Author Topic: Verify Raid Failure - Repeated  (Read 932 times)

Offline sgrantca

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
    • View Profile
Verify Raid Failure - Repeated
« on: January 02, 2018, 08:03:41 am »
Hello,

I have a recurring issue with verification.  My daily check keeps reporting the same byte failure but I'm unable to determine the disk or files with the Forensic plugin.

This has happened twice now, same byte failure position. After the first time I ran a verify sync and it went away for about two weeks and now it's back with the same byte failure range.

I ran the Forensic plugin but it doesn't show anything, just:
RAW Data
[]

I left the forensic plugin up on the screen while executing a full Verify+ task on the entire array and even just over the partial array covering the byte failure range.

My questions are:
1. How is the forensic plugin supposed to actually work? Running it after a full Verify+ or during a verify+ doesn't show any results in the plugin.
2. How can I determine the files in that position on the array to know if they are good/bad?
3. How can I determine the disk that could potentially be causing the verification failure?

Partial Verify+ Task Details:
Name: Verify RAID [tRaid]
Start Date: Tue Jan 02 01:15:00 GMT-500 2018
End Date: Tue Jan 02 02:05:29 GMT-500 2018
Duration: 00:50:29
Throughput: 382.051MB/s
Total Size: 1.103TB

Error Code = -1

2 stripe blocks failure

First byte failure at 4000785960960

Last byte failure at 4000785981440

Offline adridolf

  • Full Member
  • ***
  • Posts: 101
  • Karma: +2/-0
    • View Profile
Re: Verify Raid Failure - Repeated
« Reply #1 on: January 02, 2018, 01:59:45 pm »
Look here:
http://bug.flexraid.com/view.php?id=438

Can be addresses by reinitializing the PPU from GPT to MBR. Read the whole (!) bug report with comments, this should help you with your issue. If not, I would be interested in the follow-up.

Offline sgrantca

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
    • View Profile
Re: Verify Raid Failure - Repeated
« Reply #2 on: January 03, 2018, 08:39:40 pm »
Thanks adridolf,

My PPU is 4TB, as yours was, how did you go about reinitializing the PPU as MBR when it's over 2 TB? Was there a trick to this?

Offline sgrantca

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
    • View Profile
Re: Verify Raid Failure - Repeated
« Reply #3 on: January 04, 2018, 12:10:02 pm »
So here's what I've done with my testing.

- I enabled Configuration Override then manually failed the PPU in flexraid.
- In windows disk management I brought the PPU disk online, and initialized it as MBR. I left the partitions it created alone (since it's 4TB it force created two partitions).
- I took the PPU disk offline.
- I unfailed the PPU in flexraid which prompted me to reboot.
- Performed a full Verify Sync on array which gave me the standard: 2 stripe blocks successfully updated
    First byte updated at 4000785960960
    Last byte updated at 4000785981440

I believe your bug notes stated that rebooting recreate/triggered the bug, so I should reboot and verify+ the entire array to see if it's resolved?

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Verify Raid Failure - Repeated
« Reply #4 on: January 05, 2018, 08:59:33 pm »
@sgrantca
What release are you on?

This should not be occurring with the latest release unless something is silently updating the backup GPT info of the PPU.

Offline sgrantca

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
    • View Profile
Re: Verify Raid Failure - Repeated
« Reply #5 on: January 05, 2018, 09:49:02 pm »
OS: Microsoft Windows 7 Professional 0245EA88 (build 7601), 64-bit
Broker: 1.1.0 2017.11.22
Kernel: 1.1.0 2017.11.22

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Verify Raid Failure - Repeated
« Reply #6 on: January 05, 2018, 10:54:44 pm »
Use diskpart and report on your partition details: https://www.veritas.com/support/en_US/article.000004907
Basically:
- DISKPART
- LIST DISK
- Then for each disk [for each of your DRUs and PPUs] (you will SELECT DISK {number}), DETAIL DISK
- then, LIST PARTITION
- for each partition (you will SELECT PARTITION {number}), DETAIL PARTITION

Offline adridolf

  • Full Member
  • ***
  • Posts: 101
  • Karma: +2/-0
    • View Profile
Re: Verify Raid Failure - Repeated
« Reply #7 on: January 06, 2018, 05:53:37 am »
Regarding the bigger than 2 TB topic:
This limit refers to the size which can be PARTITIONED, so where you can place partitions. The PPU, however, is just bits build by some kind of checksum algorithm. It does not require partitioning.

The problem with GPT is that the section of the HDD supposed to contain the partition table is too small for the whole GPT information (but MBR fits). Therefore, a dedicated GPT partition is created on the disk. This is what I suspect to interfere with the data written by tRaid. At the section where the GPT partition is located, tRaid wants to place the checksum of the GPT partitions of the DRUs (which for itself might not be a valid GPT partition and thus is corrected on boot).

How to deal with that:
Stop the array, go to Disk Management (Windows) and reinitialize the PPU to MBR (rightclick the disk information there). Be sure to choose the correct disk, otherwise your data will be lost. Then restart the array. Start Specific range operation->Verify/Sync for the first and last 10 GB of the array. (Keep in mind that the "end" of the disk/array is not 4 TiB (1024^4 B), but 4 TB (1000^4 B), but tRaid thinks in TiB. So you have to calculate 4/(1.024^4) TB or 4000/(1.024^3) GB; or you can use the reported error in bytes (and start a little bit before that point) and devide the value by 1024 three times to get GiB.
After the Verify/Sync you should be fine, as only the beginning and end is affected and you do not need to resync everything.
« Last Edit: January 06, 2018, 05:56:02 am by adridolf »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Verify Raid Failure - Repeated
« Reply #8 on: January 06, 2018, 01:04:09 pm »
@adridolf
Great tips. :)

You are correct in that the PPU does not have to be GPT and can be MBR regardless of the disk size.

A GPT PPU will work without issues if the DRUs and PPUs are properly partitioned.
I am realizing now that I only properly documented this on the Linux guide (and not Windows): http://wiki.flexraid.com/2014/06/22/ultimate-linux-guide-to-transparent-raid/
Quote
Formatting or verifying the formatting of an existing disk
Transparent RAID supports disks with existing data on them. There is no data migration required. However, the proper partition alignment for the existing disks must be verified to conform.
In all cases:

  • The first partition on the disk must be at the 2048 sectors (1 MB) or greater offset
  • There must be at least 1 MB left free at the end of the disk (8 MB recommended)
  • Using the GPT patitioning format for all disks larger than 2TB (GPT and MBR are both fine for disks 2TB or under)
If any of your data disks does not conform to the above requirements, you will need to backup the data off that disk, erase and properly partition the disk as required above, and finally copy back the data on the disk.

I think I did not document this for Windows, since by default, it does the right thing when the GPT initialization is done with Windows 2008 or later.

Offline adridolf

  • Full Member
  • ***
  • Posts: 101
  • Karma: +2/-0
    • View Profile
Re: Verify Raid Failure - Repeated
« Reply #9 on: January 06, 2018, 05:27:27 pm »
Note that I did the initialization with Server 2012 (or 2012 R2) and I had the same issues with GPT PPU. However, I do not think that I have more than 1 MB at the end.

Offline hhb97b

  • Newbie
  • *
  • Posts: 26
  • Karma: +1/-0
    • View Profile
Re: Verify Raid Failure - Repeated
« Reply #10 on: January 09, 2018, 11:46:51 am »
Hi

I have also had this bug but never reported it, the new version hasn't helped me. I will try to change the ppu to mbr and see if that helps me.

Offline sgrantca

  • Newbie
  • *
  • Posts: 15
  • Karma: +0/-0
    • View Profile
Re: Verify Raid Failure - Repeated
« Reply #11 on: January 13, 2018, 08:14:45 pm »
So after changing to MBR I haven't had the issue anymore. I've rebooted multiple times and ran full sync verify without any parity errors.

@adridolf thank you so much!