Author Topic: Verify & Sync task vs Create Parity task  (Read 21250 times)

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Verify & Sync task vs Create Parity task
« on: July 08, 2013, 12:09:13 pm »
Verify & Sync task vs Create Parity task

An interesting discussion took place in the Transparent RAID RC1 thread.

A number of users have been reporting that the Verify & Sync task was running much faster than the Create Parity task when used to initialize an array.
Well, it turns out that this is to be expected.

I indeed designed the Verify & Sync task to be faster than the Create Parity task from a theoretical perspective.
I really wasn't sure how it would translate in practice, but mathematically, it is supposed to be faster.

The theory is as follow:
1. That users will typically have significant unused disk space that has never been written to and is effectively all zeros
2. That even for space where data has been written to, there is great probability that the disks will have blocks of matching data

What I was not sure of was the overhead impact of first checking the data for match prior to writing where there is mismatch.
In the Verify & Sync task, the parity disks are subjected to both read/write operations decreasing their overall write performance.
In contrast, in a Create Parity task, the parity disks only execute write operations maximizing the write throughput.

I was not certain of the net effect and what task that net effect would favor.
Well, per current user reports, the net effect is favoring the Verify & Sync task. That makes sense for cases where the Verify & Sync task does significantly less writes than the Create Parity task.

When initializing an array made up of fairly new disks, the Verify & Sync task is guaranteed to be faster.

So, when initializing or wanting to re-create the parity data, do consider using the Verify & Sync task instead as it might be significantly faster.

Offline SirMaster

  • Jr. Member
  • **
  • Posts: 78
  • Karma: +4/-0
    • View Profile
Re: Verify & Sync task vs Create Parity task
« Reply #1 on: July 08, 2013, 12:46:12 pm »
So you are saying it turns out to be faster to check and write only if mismatch than to always write?

I would have though blindly write only would be faster than read and write.

Because for blind write, you have to write 100% of the disk.  Read/Write you have to read 100% of the disk and write maybe 50% on average depending on the data.  I though read and write speeds for mechanical disks were fairly close in speed.

I guess the way that you have to write is not the same as a sequential read and write speed test.
« Last Edit: July 08, 2013, 01:07:08 pm by SirMaster »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Verify & Sync task vs Create Parity task
« Reply #2 on: July 08, 2013, 01:31:06 pm »
Yep. You would think blind writing would be faster.

Nevertheless, something clicked in me a while back even though I was not 100% sure of my hunch.
I just implemented it, made it an option for RAID initialization, but never really benchmark'ed it (got busy with other things).

This is one of those things where you can theorize all day, but testing will hold the final say. :)
Plus, I do have some optimization in my implementation that I think are making a difference here.

Another motivation for implementing it is that it creates less wear and tear on SSDs.

Offline NLS

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1,018
  • Karma: +29/-4
  • Look ma, no hands!
    • View Profile
    • iLogic
Re: Verify & Sync task vs Create Parity task
« Reply #3 on: July 08, 2013, 01:48:46 pm »
You realise that if a method is both better and faster, the other should just go (or remain as admin option).


---
NLS
Production system: SBS2011 fully patched, intel Core2 Quad, 8GB, 12 disks (1 system IDE, 1 backup IDE, 10 for array and parity most SATA3), parity is 3TB, largest data disk is 3TB, millions of smaller files, common browser Chrome latest.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Verify & Sync task vs Create Parity task
« Reply #4 on: July 08, 2013, 02:27:12 pm »
You realise that if a method is both better and faster, the other should just go (or remain as admin option).
No, "I'll rather be a slow turtle so I don't get run over".  ::)

Focus. We need to collect more data so that better recommendations can be made.

For one user here things were much faster only because he had already created parity on those drives.
What we need to do is collect stats for users coming in with fresh new drives and for users coming in with greatly used up drives.
There cases where the Verify & Sync task will simply be slower. The scenarios users are going to encounter in practice are what matter.

No jumping to conclusion yet. This thread is more of "food for thought" king of thing and to recommend giving the "Verify & Sync" task a shot.
The Create Parity task is predictable and will perform the same regardless of the drives contents, which is why it is the one recommended in the tutorials.


Offline monkeysez

  • Full Member
  • ***
  • Posts: 101
  • Karma: +0/-0
    • View Profile
Re: Verify & Sync task vs Create Parity task
« Reply #5 on: July 08, 2013, 03:33:30 pm »
When running the verify and sync command, at the end of it I received the following message. Does that mean that parity was not created properly?
« Last Edit: July 08, 2013, 03:49:28 pm by monkeysez »

Offline NLS

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1,018
  • Karma: +29/-4
  • Look ma, no hands!
    • View Profile
    • iLogic
Re: Verify & Sync task vs Create Parity task
« Reply #6 on: July 08, 2013, 03:40:21 pm »
No it's what it synced.

---
NLS
Production system: SBS2011 fully patched, intel Core2 Quad, 8GB, 12 disks (1 system IDE, 1 backup IDE, 10 for array and parity most SATA3), parity is 3TB, largest data disk is 3TB, millions of smaller files, common browser Chrome latest.

Offline monkeysez

  • Full Member
  • ***
  • Posts: 101
  • Karma: +0/-0
    • View Profile
Re: Verify & Sync task vs Create Parity task
« Reply #7 on: July 08, 2013, 03:48:44 pm »
No it's what it synced.

Gotcha. As a suggestion, I would add some language to clarify that verify/sync was successfully completed.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Verify & Sync task vs Create Parity task
« Reply #8 on: July 08, 2013, 04:21:29 pm »
Gotcha. As a suggestion, I would add some language to clarify that verify/sync was successfully completed.
Done. :)

Offline NLS

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1,018
  • Karma: +29/-4
  • Look ma, no hands!
    • View Profile
    • iLogic
Re: Verify & Sync task vs Create Parity task
« Reply #9 on: July 08, 2013, 04:23:40 pm »
I think I remember someone saying about this before. I might be wrong.
---
NLS
Production system: SBS2011 fully patched, intel Core2 Quad, 8GB, 12 disks (1 system IDE, 1 backup IDE, 10 for array and parity most SATA3), parity is 3TB, largest data disk is 3TB, millions of smaller files, common browser Chrome latest.

Offline facke02

  • Full Member
  • ***
  • Posts: 106
  • Karma: +0/-1
    • View Profile
Re: Verify & Sync task vs Create Parity task
« Reply #10 on: July 08, 2013, 07:37:21 pm »
During my testing, Verify Sync was 3 times faster then Create Parity.  I've run each about half and half when I recreate the Raid.
Ken

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Verify & Sync task vs Create Parity task
« Reply #11 on: July 08, 2013, 08:41:24 pm »
What would be more meaningful is for everyone to post their Verify Sync task report.
What I want to see is the number of reported 4K blocks processed.

Offline Gravis856

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: Verify & Sync task vs Create Parity task
« Reply #12 on: July 09, 2013, 11:27:33 am »
I'll post mine when it finally finishes. Started/stopped Create parity getting 95 MB/s and now am getting ~328 MB/s with Verify & Sync.

Edit: And 5 hours later its slowly dropped to 35 MB/s. I might abort it.
« Last Edit: July 09, 2013, 02:21:49 pm by Gravis856 »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Verify & Sync task vs Create Parity task
« Reply #13 on: July 10, 2013, 04:11:36 am »
I'll post mine when it finally finishes. Started/stopped Create parity getting 95 MB/s and now am getting ~328 MB/s with Verify & Sync.

Edit: And 5 hours later its slowly dropped to 35 MB/s. I might abort it.
Yikes! 328MB to 35MB?
Something is majorly wrong here.

The absolute worst case scenario for Verify & Sync is that it might get 1/3 slower than Create Parity.
However, that's for the most extreme case (very unlikely scenario).

How many PPUs do you have?

Offline Gravis856

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: Verify & Sync task vs Create Parity task
« Reply #14 on: July 10, 2013, 04:17:52 pm »
Yikes! 328MB to 35MB?
Something is majorly wrong here.

The absolute worst case scenario for Verify & Sync is that it might get 1/3 slower than Create Parity.
However, that's for the most extreme case (very unlikely scenario).

How many PPUs do you have?

One 2TB PPU for four 2TB DRUs. Four of them being WD Greens and one Seagate. I just went back to RAID-f for the time being and got about ~69MB/s steady til then RAID was initialized.