Author Topic: FlexRAID in Cruise Control  (Read 20739 times)

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
FlexRAID in Cruise Control
« on: September 30, 2011, 12:12:19 pm »
So, I am working on creating a "Cruise Control" mode for FlexRAID.

In this mode:
- Zero configuration outside of adding and removing drives and choosing a protection scheme
- FlexRAID will take ownership of the user's drives and will remove them from his/her view
- The source drives will NOT be accessible except through the storage pool

More:
- For protection scheme, users will be able to choose between Snapshot RAID and Real-Time RAID.
- When adding a drive, the user will state whether the drive is to be used as DRU, PPU, or URU (if applicable).
- FlexRAID will take care of scheduling the various tasks for Snapshot RAID since it can now automatically detect changes.
- Of course, users will have the option of overriding a few options, but customization will be kept to a minimal.

All the low levels to make this work are done. What's left is the UI.

Any opinion, remark, or suggestion? :)


Offline timotl

  • Newbie
  • *
  • Posts: 4
  • Karma: +1/-0
    • View Profile
Re: FlexRAID in Cruise Control
« Reply #1 on: September 30, 2011, 01:46:48 pm »
These sound like great ideas.
Would this replace the normal interface or is it more of a wizard type approach?

How would the source drives be accessed of FR is stopped or removed?
Will you just be removing the mount points for them or blocking access some other way?

-timotl


Offline jasonwc

  • Full Member
  • ***
  • Posts: 197
  • Karma: +5/-0
    • View Profile
Re: FlexRAID in Cruise Control
« Reply #2 on: September 30, 2011, 01:54:55 pm »
Thoughts:

- I think such a feature would really make FlexRAID available to a larger number of users by simplifying configuration and maintenance.  While you can achieve the same maintenance-free outcome by setting up the RAID, storage pool, and schedulersettings independently, this will make setup significantly easier.

- It makes sense to merge the storage pool functionality and Snapshot RAID for a "cruise control" solution. I've seen a lot of posts asking why they don't have a single pool after setting up RAID, not realizing the two features were independent.

- While I like the idea of removing the individual drives from Explorer (right now this has to be done manually by setting NTFS folder paths), what steps will be necessary to regain access to the individual drives if something goes wrong? Will there be a special procedure to restore access or will it be automated by an uninstall of FlexRAID?

Questions:

- Will users be given any option to control the frequency of Snapshot RAID updates or will it be completely automatic?

- I take it that adding drives will be done through the drag-n-drop interface currently used for RAID, so you won't have to manually add the drive/folder paths like you currently must do for Storage Pooling?

- What merge mode will be used for the storage pool? Will this be configurable?

- How will this option be presented to the user? In other words, will this be presented as just another option in the Add Configuration window (Storage Pool/Snapshot RAID/Realtime RAID/Cruise Control Mode", or will it be presented as a "basic" vs. "advanced" configuration? I can see some confusion arising out of this additional option due to the number of potential FlexRAID configurations (Snapshot only, storage pool only, Realtime (combined), Cruise control (combined /w either snapshot or realtime RAID).

EDIT: Apparently Tim and I had some of the same questions. I saw his comments after posting mine.
« Last Edit: September 30, 2011, 02:04:14 pm by jasonwc »
Norco RPC-4224 4U Rackmount Case (24 Hotswap Bays) | Asus P8B Workstation Motherboard (four PCI-E 16x slots @ 8x/8x/4x/4x and dual 82574L Intel server NICs) | Core i3-2100 | Supermicro AOC-SASLP-MV8 SAS Controller | Corsair TX650v2 PSU | Currently 16 TB array, 2 TB parity | WHS 2011

Offline NLS

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1,018
  • Karma: +29/-4
  • Look ma, no hands!
    • View Profile
    • iLogic
Re: FlexRAID in Cruise Control
« Reply #3 on: September 30, 2011, 02:02:10 pm »
I am thinking that maybe after this you should stop adding things for a while and debug everything first.
I wanted to see the pool code re-write you mentioned few days ago and instead you plan something else. Is this in addition to the new pool code?

As an idea, what you mentioned is where the software SHOULD go anyway for the mean user (so be the default) and maybe let all the other tweaks for some advanced mode or something.

---
NLS
Production system: SBS2011 fully patched, intel Core2 Quad, 8GB, 12 disks (1 system IDE, 1 backup IDE, 10 for array and parity most SATA3), parity is 3TB, largest data disk is 3TB, millions of smaller files, common browser Chrome latest.

Offline jasonwc

  • Full Member
  • ***
  • Posts: 197
  • Karma: +5/-0
    • View Profile
Re: FlexRAID in Cruise Control
« Reply #4 on: September 30, 2011, 02:09:46 pm »
I am thinking that maybe after this you should stop adding things for a while and debug everything first.
I wanted to see the pool code re-write you mentioned few days ago and instead you plan something else. Is this in addition to the new pool code?

As an idea, what you mentioned is where the software SHOULD go anyway for the mean user (so be the default) and maybe let all the other tweaks for some advanced mode or something.

I agree with this. Right now, my only issues with FlexRAID are related to the storage pool. Snapshot RAID was easy to setup using xliv's guide, and the scheduler is dead simple. The RAID function seems really stable as well.  As the re-implimentation of the storage pool will fix the nasty delete bugs and provide full support in WHS 2011, I think that should be the priority before adding a new mode that can already be achieved with the current version of FlexRAID. Just my 2 cents.
Norco RPC-4224 4U Rackmount Case (24 Hotswap Bays) | Asus P8B Workstation Motherboard (four PCI-E 16x slots @ 8x/8x/4x/4x and dual 82574L Intel server NICs) | Core i3-2100 | Supermicro AOC-SASLP-MV8 SAS Controller | Corsair TX650v2 PSU | Currently 16 TB array, 2 TB parity | WHS 2011

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: FlexRAID in Cruise Control
« Reply #5 on: September 30, 2011, 02:30:31 pm »
@timotl
The old interface will stay as-is for advanced users. There will be a simpler interface for the Cruise Control functionality.

@jasonwc
Agreed on all thoughts.

1. Users will be able to regain control of their drives very easily if they wish to. Single button click. :)
2. If there is a spare, restoration will be automatic. If not, the user will be taken to a wizard for the process.
3. Yes, as mentioned above, users will have a few options to customize the Cruise Control mode like changing some aspect of a Snapshot update schedule and criteria.
4. There will be a new interface for all Cruise Control operations. I will work on it over the weekend since I have the design fresh in mind.
5. I was thinking of presenting this as: Cruise Control Mode versus Expert Mode, where Expert Mode is the current way of doing things.

@NLS
In principle, yes. However, now that the storage pool is much improved, I need to get the Real-Time RAID production ready, and this new mode will allow me to do just that.
In fact, my plan is to support the Real-Time RAID in Cruise Control mode only.
Real-Time RAID will be available in Expert mode, but that will be considered experimental.



Offline jasonwc

  • Full Member
  • ***
  • Posts: 197
  • Karma: +5/-0
    • View Profile
Re: FlexRAID in Cruise Control
« Reply #6 on: September 30, 2011, 03:16:03 pm »
2. If there is a spare, restoration will be automatic. If not, the user will be taken to a wizard for the process.

This raises the question regarding how FlexRAID will handle a situation where the drive has not failed but rather has been temporarily disconnected or turned olff, as is often the case with external hard drives. Unlike other RAID solutions, FlexRAID doesn't require every DRU to be always active, since you can control when updates occur. So, if one of the DRUs contains a removable drive, and the drive has been turned off/disconnected, will this result in a restore operation?
« Last Edit: September 30, 2011, 03:41:00 pm by jasonwc »
Norco RPC-4224 4U Rackmount Case (24 Hotswap Bays) | Asus P8B Workstation Motherboard (four PCI-E 16x slots @ 8x/8x/4x/4x and dual 82574L Intel server NICs) | Core i3-2100 | Supermicro AOC-SASLP-MV8 SAS Controller | Corsair TX650v2 PSU | Currently 16 TB array, 2 TB parity | WHS 2011

Offline xliv

  • Global Moderator
  • Full Member
  • *****
  • Posts: 208
  • Karma: +7/-1
    • View Profile
Re: FlexRAID in Cruise Control
« Reply #7 on: September 30, 2011, 06:18:42 pm »
I love it! Looks great!
(well apart from the fact that we'll need to redo all guides...)

I'm not sure I understand the difference btw snapshot and RT in this case, if this new mode can track the change and do the updates automatically?
Server HW: ASUS P8B-E/4L + PIKE 2008 SAS | Xeon E3-1230 | 8GB DDR3 ECC 1333 [Kingston - KVR1333D3E9SK2/8G] | Intel 320 - 80 Go (OS)
Server OS: Win7 64b EN
FlexRAID 2.0 latest build, Cruise Control, 7 DRUs (6x3TB,1x2TB), 1 PPU (2x2TB)

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: FlexRAID in Cruise Control
« Reply #8 on: September 30, 2011, 06:33:48 pm »
I love it! Looks great!
(well apart from the fact that we'll need to redo all guides...)

I'm not sure I understand the difference btw snapshot and RT in this case, if this new mode can track the change and do the updates automatically?

Changes are not tracked in Snapshot mode. It just will know that there are changes and an update will need to be scheduled.

Offline Gaevonn

  • Newbie
  • *
  • Posts: 29
  • Karma: +0/-0
    • View Profile
Re: FlexRAID in Cruise Control
« Reply #9 on: October 02, 2011, 10:35:48 am »
This sounds like a great idea.  One question though and sorry if it was in the others and I missed it.  What about people like myself, whose media is mixed between drives already?  Will the Cruise Control system recognize this media and setup folders accordingly, or woiuld this be best left to people starting fresh.  Thanks again for the great product.
I Work; therefore, I Live.
;-)

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: FlexRAID in Cruise Control
« Reply #10 on: October 02, 2011, 01:52:07 pm »
This sounds like a great idea.  One question though and sorry if it was in the others and I missed it.  What about people like myself, whose media is mixed between drives already?  Will the Cruise Control system recognize this media and setup folders accordingly, or woiuld this be best left to people starting fresh.  Thanks again for the great product.

The Cruise Control mode uses the Auto-Folder merge mode (the other modes might be offered later).
So, the existing data must support being merged automatically.

Offline S_F

  • Newbie
  • *
  • Posts: 49
  • Karma: +0/-0
    • View Profile
Re: FlexRAID in Cruise Control
« Reply #11 on: October 03, 2011, 07:11:50 am »
the existing data must support being merged automatically.

Could you please elaborate on what exactly this means?

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: FlexRAID in Cruise Control
« Reply #12 on: October 04, 2011, 07:05:16 pm »
Update:

This feature is turning out to be a lot more work than I bargained for. :(
The only thing that keeps me going is that it will be the most used feature without any doubt.
Most users don't want to be experts or tinker with advanced features. I know some users don't agree, but we will see.

Anyway, coming up with a simplistic UI that just makes sense is not easy. Not at all.  :-\
As always, the simpler the interface, the more complex the background execution needs to be.

Offline Sushi6

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: FlexRAID in Cruise Control
« Reply #13 on: October 04, 2011, 08:43:56 pm »
Thanks Brahim, Cruise control sounds great.
I'm so looking forward to using FlexRAID real time on a dual boot Ubuntu/winpoo7 machine. I have not installed FR... yet. With Cruise control working in linux, I'd install it instantly.

My suggestion is for Linux and that the disks in the storage pool only mount when you use them and you can umount them when you want but also have an option to automount on startup.
 
extra note off topic but.... I've waited years for something like FlexRAID. I've done the CD then DVD with par files. Then later  used DVDisaster which is painfully time consuming. To think with FR all I have to do is buy a new larger HDD every few years is a dream come true. I think people are archiving more and more and done properly, if you can combine it with many different systems, FlexRAID will change the way the average user stores data... and then learns to create a media server.
« Last Edit: October 04, 2011, 09:06:41 pm by Sushi6 »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: FlexRAID in Cruise Control
« Reply #14 on: October 07, 2011, 05:40:31 am »
Update:
On the plus side of things, the Real-Time RAID functionality is now feature complete. :)
Not sure why it took this long, but RAID expansion and contraction are now possible without having to re-create the Real-Time RAID. So now, not only users can create the RAID with drives containing existing data, but they can also add new drives with existing data onto the array after it has been created. Similarly, drives with or without data that are part of the array can be removed.
After a drive has been removed, its data can be returned to the pool and allocated to the remaining drives automatically or at a later time.

The core interface for the drive manager is complete but there are a lot of intricacies in the background that I still need to code for.