Author Topic: Storage Pool with Explicit Merge Questions  (Read 3256 times)

Offline Gaevonn

  • Newbie
  • *
  • Posts: 29
  • Karma: +0/-0
    • View Profile
Storage Pool with Explicit Merge Questions
« on: September 26, 2011, 09:50:00 pm »
Hello and thanks for a great program.  Brahim, I have been looking for a program that will provide me Drive Pooling with Failure recovery and this certianly looks like it.  I have been working to migrate from WHS V1 over to this and have run into a snag.  I will preface this with, I know that Brahim and others are quite busy and like to ensure that we have tried reading all the documents ahead of time and tried (even if failed) first, before posting this type of question.  I will say that I have been working this and reading the old forum for the last week, but am still running into the same issue.  It must be a simple "Squiggle in the wrong spot" issue, but I have yet to figure it out.  If at all possible, could someone look at my configuration and drive setup and see what it is I am missing.  Thank You.

System setup is a Test Machine with dual core AMD Phenom II X2 555, running Windows 7 PRo SP1 and 16GB of Ram.  I have 1 OS Drive and DVD-Rom on the onboard SATA and 8 HDDS on a Supermicro AOC-SASLP-MV8 PCIe controller running in non-RAID mode.  HDDs 1-4 are 2TB Hitachi and 5-8 are 1TB WD Black.  I am using the FlexRAID-2.0-PreviewXII.


Config File:
*****Contents from the Runtime Properties Section *********
unique=C:\Storage\Disk1;C:\Storage\Disk2;C:\Storage\Disk3;C:\Storage\Disk4;C:\Storage\Disk5;C:\Storage\Disk6;C:\Storage\Disk7;C:\Storage\Disk8
restrict=
thread=5
bufferSize=1MB
reserve=25GB
removable=false
syncFolderLastMod=false
filesystem=FlexRAIDFS
*****Contents from the Runtime Properties Section *********

DRIVE=E   

-E:\   
|-<Movies>   
|-<TV_Shows>   

-E:\Movies   
|-*C:\Storage\Disk1\Movies
|-*C:\Storage\Disk2\Movies
|-*C:\Storage\Disk3\Movies
|-*C:\Storage\Disk4\Movies

-E:\TV_Shows   
|-*C:\Storage\Disk5\TV_Shows
|-*C:\Storage\Disk6\TV_Shows
|-*C:\Storage\Disk7\TV_Shows
|-*C:\Storage\Disk8\TV_Shows


Configuration of Drives: Using NTFS Folders instead of Drive Letters.  Drives 1-4 have a "Movie" Folder and Drives 5-8 have a "TV_Shows" and all folders have files, but none of the drives are full.

Need:  For me to be able to copy additional files to the E:\Movies and as Disk1 fills, it continues on to Disk 2.  Same for E:\TV_Shows; in that, as Drive5 fills, it moves to Drive6.  Continuing to each drive as necessary, when they get full, was the idea, but keeping the Movie and TV_Shows seperated to different drives.

The Issue:  The configuration (and I have tried it with and without virtual folders) seems to work fine for E:\Movies, where the files start in Drive1\Movies and continue to Drive2 (understanding that they do not appear in the actual Disk2\Movies folder, but the hidden folder).  The issue is the TV_Shows folder.  The files always copy to the Drive5\TV_Shows folder and then once Drive5 is full, the files continue copying to the hidden folder on Drive1.  I cannot get the files to continue to the Drive6, drive7, or drive 8, so long as Drive1-4 are present in the configuration. 

I have read and re-read numerous posts an explicit merge and I am positive that this is the mode I will need to use, so that i can continue to keep the two folders on seperate drives.  I have tried not using Virtual Folders, but achieved the same result.  It seems like what I want to do is possible, based on others accounts, but I have yet to find a method that seems to work.  I have tried to use "Auto-Merge" modes, but the Space Priority only moved data across 4 drives and not 8, but also spread out the data strangely.  Using the Folder Priority, everything starts on Disk1.  Any help would be greatly appreciated.

One final question: Will the storage pools with explicit merge ever allow flow over into identical folders on the Merged drives, or can the Auto-Merge options be set to force specific drives to be used for specific virtual folders?  My data will require "moving from the WHS to the final Storage box and currently it is not seperated on the drives, but my goal to to seperate, so that I can grow the storage pools for only what I need and the apply Snap-Shot later.
I Work; therefore, I Live.
;-)

Offline xliv

  • Global Moderator
  • Full Member
  • *****
  • Posts: 208
  • Karma: +7/-1
    • View Profile
Re: Storage Pool with Explicit Merge Questions
« Reply #1 on: September 27, 2011, 02:54:15 am »
Hi,

Unfortunatley I'm not using the explicit merge configuration, as the auto-mode are actually sufficient for my needs.
I used to want to group things together, but I figured I should let go, and allow FlexRAID to do its job.

In your case, your configuration seems correct. But again, not a specialist of the explicit merge.
Do the folders TV_Shows and Movies exist on all your drives? auto modes are creating automatically the root folders, but explicit does not.

It seems like a bug - if you can't find your way around it, enter it in bug.flexraid.com, attaching the logs (log.log in the installation folder, and FlexRAID.sys.log and FlexRAID.DM.log in the user root or your OS drive root).

Or you could let go, and leave FlexRAID to decide where to store your files, merging your movies and TV shows  ;D
Server HW: ASUS P8B-E/4L + PIKE 2008 SAS | Xeon E3-1230 | 8GB DDR3 ECC 1333 [Kingston - KVR1333D3E9SK2/8G] | Intel 320 - 80 Go (OS)
Server OS: Win7 64b EN
FlexRAID 2.0 latest build, Cruise Control, 7 DRUs (6x3TB,1x2TB), 1 PPU (2x2TB)

Offline NLS

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1,018
  • Karma: +29/-4
  • Look ma, no hands!
    • View Profile
    • iLogic
Re: Storage Pool with Explicit Merge Questions
« Reply #2 on: September 27, 2011, 10:30:20 am »
I thought explicit merge was disabled for now.
---
NLS
Production system: SBS2011 fully patched, intel Core2 Quad, 8GB, 12 disks (1 system IDE, 1 backup IDE, 10 for array and parity most SATA3), parity is 3TB, largest data disk is 3TB, millions of smaller files, common browser Chrome latest.

Offline xliv

  • Global Moderator
  • Full Member
  • *****
  • Posts: 208
  • Karma: +7/-1
    • View Profile
Re: Storage Pool with Explicit Merge Questions
« Reply #3 on: September 27, 2011, 11:32:32 am »
Ha! Indeed, I remembered seeing something during one of the preview releases, but I assumed it was reenabled later on. That would require Brahim's confirmation I guess.

But the only reason why Gaevonn is willing to use the explicit merge is the ability to separate his TV shows from movies. Maybe the best solution for this would be the ability to create several pools with different drives. I don't know if that will be possible with the next implementation, maybe should be entered as a feature request.
Server HW: ASUS P8B-E/4L + PIKE 2008 SAS | Xeon E3-1230 | 8GB DDR3 ECC 1333 [Kingston - KVR1333D3E9SK2/8G] | Intel 320 - 80 Go (OS)
Server OS: Win7 64b EN
FlexRAID 2.0 latest build, Cruise Control, 7 DRUs (6x3TB,1x2TB), 1 PPU (2x2TB)

Offline Gaevonn

  • Newbie
  • *
  • Posts: 29
  • Karma: +0/-0
    • View Profile
Re: Storage Pool with Explicit Merge Questions
« Reply #4 on: September 27, 2011, 12:53:06 pm »
Thanks for the responses, thus far.  I had read that Explicit mode merging was disabled for Real-time RAID, but i am not using either RAID mode right now.  Maybe that is my issue?  I assumed I could use this for storage pooling only and then once I got that worked out, move to a RAID with Storage pool mode.  It seems strange that the data moves to the first drive properly, but once that drive is full and then it moves to whatever the first drive in the configuration is (unless that is full too, in which it still moves to Drive2 and not Drive6, as I need it to).  It may be wishful thinking on my part.  most of the Forum info on explicit mode is from the earlier versions.  while i agree the "Auto merge" options are working, I would really like to keep the data seperated to the two drive groups, if possible.  I like xliv's idea of multiple pools, but I figure that is too much work.  Brahim is quite busy with an update, if i am not mistaken.  I'll continue to throw diff configs at it and see where I get, but any suggestions are greatly appreciated.
I Work; therefore, I Live.
;-)

Offline Gaevonn

  • Newbie
  • *
  • Posts: 29
  • Karma: +0/-0
    • View Profile
Re: Storage Pool with Explicit Merge Questions
« Reply #5 on: September 27, 2011, 01:00:26 pm »
Xliv,  thanks for the suggestion.  My folders do exist on all the drives and each have some data in them (though I tried this with empty folders too).  I was holding off on a Bug Report, until someone with Explicit mode merging experience, or Brahim could confirm that based on what I wanted to see, the config looked right.  If I cannot work it out by the end of the week, I will do as you suggest and open a bug report.  I just don't want to appear a pest, if this was something right in front of my face.  Thanks 8)
I Work; therefore, I Live.
;-)

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Storage Pool with Explicit Merge Questions
« Reply #6 on: September 27, 2011, 02:10:40 pm »
@Gaevonn
xliv is correct. It is a bug.
It should have behaved the way you originally thought. I will fix that in the next release.

The Explicit Merge mode is only disabled for the Real-Time RAID feature since it is less forgiving to configuration mistakes.

Offline Gaevonn

  • Newbie
  • *
  • Posts: 29
  • Karma: +0/-0
    • View Profile
Re: Storage Pool with Explicit Merge Questions
« Reply #7 on: September 27, 2011, 03:45:58 pm »
Brahim, thank you.  This gives me renewed hope.  I am very excited about the work being done on this project.  Question?? How does one "donate" to your cause?  I am more than willing to pay for this software.  I noticed your "Thank You" to the donators, but was unable to see how.  thanks again.  i await the updated release.  hate to ask, but do you have a timeline that you can share? ;D
I Work; therefore, I Live.
;-)

Offline xliv

  • Global Moderator
  • Full Member
  • *****
  • Posts: 208
  • Karma: +7/-1
    • View Profile
Re: Storage Pool with Explicit Merge Questions
« Reply #8 on: September 27, 2011, 04:51:55 pm »
Hi - Brahim does not like to answer to those questions  8)
See here: http://www.openegg.org/forums/forums/list.page

At the bottom of the page, there is a paypal button.
Server HW: ASUS P8B-E/4L + PIKE 2008 SAS | Xeon E3-1230 | 8GB DDR3 ECC 1333 [Kingston - KVR1333D3E9SK2/8G] | Intel 320 - 80 Go (OS)
Server OS: Win7 64b EN
FlexRAID 2.0 latest build, Cruise Control, 7 DRUs (6x3TB,1x2TB), 1 PPU (2x2TB)