Author Topic: Setting up very large system (24 x 1TB and 24 x 2TB)  (Read 25854 times)

Offline pclausen

  • Jr. Member
  • **
  • Posts: 96
  • Karma: +0/-0
    • View Profile
Setting up very large system (24 x 1TB and 24 x 2TB)
« on: November 20, 2013, 04:55:34 pm »
I currently have 24 1TB drives in a RAID6 and another 24 2TB drives in a different RAID6 array.  Each RAID is run by a Areca controller (both in the same physical server but the 48TB RAID is in a 2nd chassis connected via a 4 lane SAS cable to the SAS expander).

I've been real close to loosing the 48TB a couple of time now when a rebuilt took place and I lost a 2nd drive during the rebuilt and a 3rd stated getting timeouts.  So I'm looking to move to a solution that will prevent a total loss of data in case more than 2 drives completely fail.

Does the latest version still only support 2 PPUs?  If so, does that mean I should be setting be setting up the following: (assuming using just my existing drives for now, see paragraph below)

RAID1 10 x 1TB DRU + 2 1TB PPU
RAID2 10 x 1TB DRU + 2 1TB PPU
RAID3 10 x 2TB DRU + 2 2TB PPU
RAID4 10 x 2TB DRU + 2 2TB PPU

Going forward, I plan to begin purchasing 4TB drives, and I'll need a handful to get this thing kick started so that I can take the 24TB off line and redeploy as tRAID.  I currently has a little over 19TB of data on the small array and I have 6 spare 2TB drives and 1 1TB drive laying around.  So that workout out to 13TB.  So if I pick up 3 4TB drives, I will should be able to setup a "kickstart" tRAID to offload the 19TB of data onto, right?

Any best practice advice on how to get started with setting up a large system would be appreciated.

Thanks
« Last Edit: November 20, 2013, 05:06:50 pm by pclausen »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Setting up very large system (24 x 1TB and 24 x 2TB)
« Reply #1 on: November 20, 2013, 05:47:37 pm »
Yes, tRAID is currently limited to 2 PPUs. The plan is to enable more PPUs later.

Your real challenges will be:
- finding the spare 19TB of free space to offload the current RAID content into
- making sure that your controllers play nice in JBOD mode

After your free up the old disks, then your proposed setup sounds sensible:
RAID1 10 x 1TB DRU + 2 1TB PPU
RAID2 10 x 1TB DRU + 2 1TB PPU
RAID3 10 x 2TB DRU + 2 2TB PPU
RAID4 10 x 2TB DRU + 2 2TB PPU

If you trust your disks, then you could push it to (still I would go with your proposed setup):
RAID1 20 x 1TB DRU + 2 1TB PPU
RAID2 20 x 2TB DRU + 2 2TB PPU

Once you get ready to move to 4TB disks, the PPUs will be the first to get upgraded.

If it was me, I would buy 4x 4TB and then setup the initial migration array:
- 6x 2TB (spares) DRUs
- 2x 4TB DRUs
- 2x 4TB PPUs

Then copy the 19TB of data into this array.
After that, dismantle the old disks and setup new arrays.

Offline pclausen

  • Jr. Member
  • **
  • Posts: 96
  • Karma: +0/-0
    • View Profile
Re: Setting up very large system (24 x 1TB and 24 x 2TB)
« Reply #2 on: November 20, 2013, 06:33:45 pm »
Thanks.  I'll go ahead and pick up 4 4TB drives to get started.

I read through the wiki and have a few questions;

Am I correct in assuming that I can add the disks from multiple arrays into a single pool?

Since I'll end up with well more than 26 drives, as long as I do in in stages and delete the drive letters as I go, I'm not going to have issues once I get to the 27th disk am I (past the letter Z)?

My understanding is that my controllers play nice in JBOD mode, but initially, I'll have to use pass through mode for my "kickstart" tRAID since setting the controller to JBOD mode is a global setting.

p.s. How many posts do I need to have before I don't to do the verification during each one?
« Last Edit: November 20, 2013, 06:35:39 pm by pclausen »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Setting up very large system (24 x 1TB and 24 x 2TB)
« Reply #3 on: November 20, 2013, 07:07:39 pm »
The pool is per array. I might consider allowing a spool that spans arrays, but that will be for later builds.

You don't need drive letters to configure thing in tRAID.


Offline pclausen

  • Jr. Member
  • **
  • Posts: 96
  • Karma: +0/-0
    • View Profile
Re: Setting up very large system (24 x 1TB and 24 x 2TB)
« Reply #4 on: November 21, 2013, 06:05:55 pm »
Cool.  I went ahead and configured a tRAID with the 6 spare drives.  Here are they as pass through on the controller:



And here's the parity getting created on my test tRAID.  Throughput seems fairly decent at 167 MB/s.



I never bothered assigning drive letters, so I'm not seeing the drives presented to the OS, which it how I wanted it.

Once the RAID is complete, I plan to do some major stress testing copying stuff my existing RAIDs.  So far so good!

Offline pclausen

  • Jr. Member
  • **
  • Posts: 96
  • Karma: +0/-0
    • View Profile
Re: Setting up very large system (24 x 1TB and 24 x 2TB)
« Reply #5 on: November 22, 2013, 07:58:06 pm »
Copying about 1.4TB over to the tRAID.  Seems a bit pokey at 25 MB/s.  Of course I haven't done any tuning yet.  Will let it complete, then tweak stuff, and try again.  I would have expected around 80-100 MB/s.


Offline pclausen

  • Jr. Member
  • **
  • Posts: 96
  • Karma: +0/-0
    • View Profile
Re: Setting up very large system (24 x 1TB and 24 x 2TB)
« Reply #6 on: November 23, 2013, 02:10:58 pm »
When I stop the array and then start it again, I get an error message that I need to reboot the server.

Is this normal behavior?

I got the same thing after the array had initially created the parity and I went to start it for the first time.  I could see it maybe needing it then, but having to reboot the entire system each time I tweak a setting the requires a restart of the array, is going to put a lot of strain/stress on the hardware.

Offline dscline

  • Sr. Member
  • ****
  • Posts: 272
  • Karma: +6/-0
    • View Profile
Re: Setting up very large system (24 x 1TB and 24 x 2TB)
« Reply #7 on: November 28, 2013, 04:40:22 pm »
Yes, tRAID is currently limited to 2 PPUs. The plan is to enable more PPUs later.

Just to confirm, the plan is for more PPUs to be enabled with the current "product", as opposed to it becoming available in some future offering that would require additional investment?  Would it require rebuilding ALL the parity (i.e., data at unprotected during a rebuild), or could the existing array be expanded to incorporate additional PPUs?
« Last Edit: November 28, 2013, 06:19:22 pm by dscline »
WHS 2011
tRAID final 23 DRUs 2PPUs
Supermicro C2SEA, Q9505s (stock), 4GB
Supermicro AOC-SASLP-MV8
IBM m1015 flashed to LSI 9211-8i/IT + HP SAS Expander
Generic SiI3132

Offline terrastrife

  • Full Member
  • ***
  • Posts: 235
  • Karma: +7/-0
  • 38TB FlexRAID
    • View Profile
Re: Setting up very large system (24 x 1TB and 24 x 2TB)
« Reply #8 on: November 29, 2013, 01:13:17 am »
Your write speeds are I'm partially guessing due to parity disk performance. Your quickest 2TB would probably be the Deskstar 7K2000 which I found is about half the performance for parity compared to a ST4000DM001 (5900RPM 4TB).

Slowest disk will also impact overall performance during tasks, for example, my tasks are ~1000MB/sec but I'm being held back by a couple of older 2TB drives. With 6 disks I would guess your task throughput should be closer to 450MB/sec. You have a bottleneck somewhere.

If you even remotely get close to changing/viewing advance settings between stopping and starting the array it will ask to reboot.

Offline dscline

  • Sr. Member
  • ****
  • Posts: 272
  • Karma: +6/-0
    • View Profile
Re: Setting up very large system (24 x 1TB and 24 x 2TB)
« Reply #9 on: December 01, 2013, 08:26:54 pm »
Just to confirm, the plan is for more PPUs to be enabled with the current "product", as opposed to it becoming available in some future offering that would require additional investment?  Would it require rebuilding ALL the parity (i.e., data at unprotected during a rebuild), or could the existing array be expanded to incorporate additional PPUs?
Bumping the question... is Tx going to be incorporated into tRAID any time soon?
WHS 2011
tRAID final 23 DRUs 2PPUs
Supermicro C2SEA, Q9505s (stock), 4GB
Supermicro AOC-SASLP-MV8
IBM m1015 flashed to LSI 9211-8i/IT + HP SAS Expander
Generic SiI3132

Offline pclausen

  • Jr. Member
  • **
  • Posts: 96
  • Karma: +0/-0
    • View Profile
Re: Setting up very large system (24 x 1TB and 24 x 2TB)
« Reply #10 on: July 05, 2014, 08:32:42 pm »
Resurrecting this old thread of mine.

Is there still a 2 PPU limit?

After almost loosing one of my RAID6 arrays again, I've finally been pushed over the edge to move to tRAID, but was curious what enhancements may have taken place with regards to setting up large arrays.

I also lost one of my SAS backplanes, so I'm upgrading it to a SAS2 (supports 6Gbs) so that should help take care of some of the write performance issues I had.  I see that a "landing disc" is now supported, which is great news for "normal" copying.  In my case, I'll be doing some not so normal copying when transferring 20TB and then 30TB worth of data to tRAID.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Setting up very large system (24 x 1TB and 24 x 2TB)
« Reply #11 on: July 06, 2014, 05:36:58 am »
New build next week. However, the limit on PPUs is still at two.
There is going to be a special focus on RAID engines in a few months.

Offline pclausen

  • Jr. Member
  • **
  • Posts: 96
  • Karma: +0/-0
    • View Profile
Re: Setting up very large system (24 x 1TB and 24 x 2TB)
« Reply #12 on: July 07, 2014, 09:47:38 am »
Ok thanks.  I'll hold off on downloading until the latest build is out.

In your initial reply, you stated if I trusted my drives, I could potentially configure just 2 RAID sets with 24 drives in each.

What are your thoughts on the following config:

RAID1 22 x 1TB DRU + 2 4TB PPU
RAID2 22 x 2TB DRU + 2 4TB PPU

Over time as the 1TB and 2TB drives die, I would replace them with 4TB units.

Once more than 2 PPUs are supported, I could see adding a 3rd one to each RAID array, especially as the DRUs are being replaced with larger capacity drives.

One thing that I'm not clear on, is how many DRU drives can I loose before I reach the point where there's not enough parity information available to reconstruct the data?  Does this change as the capacity of the DRUs increase?

With RAID6 it was pretty straight forward in that I could loose 2 drives at once with no data loss.  Of course as the capacity of each drive went up, the probability of loosing a 3rd drive during the rebuilt process went up as well.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Setting up very large system (24 x 1TB and 24 x 2TB)
« Reply #13 on: July 07, 2014, 12:02:17 pm »
How many drives you should have per array varies per setup and level of paranoia. ;)

Your proposed setups should be fine unless you are unlucky to lose 3 drives at once from the same array.

The protection rules with 2 PPUs in tRAID are the same as in RAID 6 except better (no stripe dependency).

Offline pclausen

  • Jr. Member
  • **
  • Posts: 96
  • Karma: +0/-0
    • View Profile
Re: Setting up very large system (24 x 1TB and 24 x 2TB)
« Reply #14 on: July 07, 2014, 12:23:34 pm »
Got it thanks.

I think I got my migration plan figured out, just need to verify a few assumptions:

1. I can upgrade a PPU from 1TB to 2TB and then later from 2TB to 4TB, correct?

2. I can upgrade a DRU from 1TB to 2TB and then later from 2TB to 4TB and once the data has been restored from parity, the extra additional storage capacity will be available, right?