Author Topic: Large file handling and proper sizing.  (Read 1802 times)

Offline NLS

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1,018
  • Karma: +29/-4
  • Look ma, no hands!
    • View Profile
    • iLogic
Large file handling and proper sizing.
« on: November 03, 2011, 01:28:33 pm »
OK since the title doesn't say much, I have to explain.

I've noticed that in Real-Time Parity we have to set parity size. It doesn't automatically use the space of the volume specified. This is ok if the PPU is NOT a volume but maybe a common folder, but not very intuitive if you use a full volume (or more full volumes).
Also I've noticed that contrary to snapshot, it creates a MONSTER file with the size of the requested parity.

Now the questions risen from this are those:

First of all how the OS handles such a huge file? I suspect it might be a problem exp. with older versions like XP (and WHS). I also wonder if this file is actually fragmented and I doubt it can be defragmented with any tools. This is my first "question" and I move to the second that was the actual reason I made the post.

Second and more affecting the end user, is what kind of measure system is used and should be used. I am an old school computer guy and I see things as powers of 2. As we know though, drive manufacturers use the language of "stupid consumer" (which also helps them cheat as a 2TB drive is NOT a 2 TERABYTE drive - note I never accepted the new definitions of mibobytes and such nonsense).

Anyway this would be a nice coffee table discussion but it actually affects us in using FlexRAID Real-Time Parity.

So let say we have a 2TB PPU drive. As hard disk manufacturers define 2TB. So in parity size what we say? (supposing we want to use the whole thing) 2TB? Maybe 1.9TB? Maybe 2000GB? Or 1999GB? Or 2047GB? (in the latter two, I left 1GB out for various things - and have to make sure to also not grow my DRUs beyond this size) ...what is the "proper" way to use this setting? (again: supposing we want to give the whole volume as PPU)

PS. I wouldn't be NLS if there wasn't a hidden feature request here: System should ask if we want to use the whole volumes and (ESPECIALLY in cruise control mode) set this setting automatically properly.

PS2. Those things above are the reasons that I don't actually use real-time parity (although I waited for this since I first heard of flexraid some years ago).
---
NLS
Production system: SBS2011 fully patched, intel Core2 Quad, 8GB, 12 disks (1 system IDE, 1 backup IDE, 10 for array and parity most SATA3), parity is 3TB, largest data disk is 3TB, millions of smaller files, common browser Chrome latest.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Large file handling and proper sizing.
« Reply #1 on: November 03, 2011, 01:49:22 pm »
The default is to use the whole drive unless the parity size is explicitly specified.
You do not need to set the parity size. If it did not behave that way, then it is a bug.
In fact, users should not set the parity size unless under specific circumstances as FlexRAID will automatically determine what the appropriate size should be.

Setting the parity size only makes sense in Expert mode and when the parity drive is much larger than the largest DRU.
So, if all your DRUs are 2TB and you have a 3TB PPU and know that none of your DRUs will ever grow to 3TB, then you can explicitly set the parity size to 2TB and use the remaining 1TB off the PPU for other purposes.

The parity file size is not an issue. It is intentional and has several technical benefits, which I won't go into. So, don't worry about it.
Obviously, FAT32 drives are not supported as PPU drives.

When explicitly specifying the parity size, users should always leave some extra space for the metadata information.
Since there is no way to determine the exact extra space that needs to be reserved, we will just use a simple formulat: 100MB * Number of DRUs.