Author Topic: Problems downsizing array of Spanned DRU's  (Read 1305 times)

Offline Mattzz

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Problems downsizing array of Spanned DRU's
« on: October 26, 2015, 06:55:33 am »
Hello,


My setup (numbers in tb)
-PPU
6
-DRU
6
3+2
3+2
4


My ultimate goal is to downsize the array to fewer DRU, leaving some of the smaller hDDS outside of the raid - the problem is how to handle the spanned DRU. The "perfect data copy" returns no errors yet moved no data:


"
The [<[copy-data]>] task has successfully initiated... Process number: 7

Created 0 directories out of 23.143K total...
Created 0 files out of 166.269K total...
Copied 0.0 byte out of 1003,3 GB total...
"

I tried this on removing both hdds in of the DRU2. Perfect copy did not work on any of them.


What is the correct approach to downsize the array?


BR Mattzz

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Problems downsizing array of Spanned DRU's
« Reply #1 on: October 26, 2015, 07:15:30 am »
You need to first remove the desired disks from the array (RAID contraction).
Do that through the Disk Manager.
After the desired disks have been removed, then you can Perfect Data Copy them into the pool.

Offline Mattzz

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: Problems downsizing array of Spanned DRU's
« Reply #2 on: October 26, 2015, 07:36:16 am »
Thanks for the quick reply!

I am in Cruise Control mode.
The process I followed was:

1) Remove Spanned hdd2tb from DRU2 (3+2tb) via drive manager
The drive gets a new mounted letter, F: working fine. The drive looks empty, but has some 250mb used space.
2) Perfect Data Copy option, selecting destination folder to V:\  as the drivepool.
3) No getting the 'type of error' shown above.
4) repeating the process for the hdd3tb in the same DRU. Different drive letter, E:\
5)  this time the availible space in windows indicates the drive has 1.79tb material stored.
6) Opening the hdd, enabling "show hidden files and folders" - nothing appears.
7) Perfect Data Copy completed within seconds, claiming no data was moved.

How to proceed? I've the feeling the data never got copied to the pool, making my uncertain about the integrity of the data that is now on the array, or the removed hdds.
« Last Edit: October 26, 2015, 07:40:42 am by Mattzz »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Problems downsizing array of Spanned DRU's
« Reply #3 on: October 26, 2015, 08:02:38 am »
You need to enable the viewing of System files to see the hidden content.
Change the options on the hidden folders to be normal folders and not system folders.

You also need to post the full logs. It seems that Perfect Data Copy is seeing the files but not copying them. Maybe the full logs will provide clues as to why.

Offline Mattzz

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: Problems downsizing array of Spanned DRU's
« Reply #4 on: October 27, 2015, 04:05:07 am »
Hello,

I kind of went into full panic mode, making all sorts of quick decisions that probably would not be recommended but worked out as of yet.
Luckily I've only smaller files, and a simple tree structure to keep track of, manually reconciling files between hdds is "an option".

Now- for the logs. I cut the logs since I fiddled around a bit more than necessary. However, I found some clues in the logs.

Here my suspicion of the failing copy command attempts shows:

[2015-10-26 12:53:48,629] INFO : [copy-data] started at: Mon Oct 26 12:53:48 CET 2015
[2015-10-26 12:53:49,900] INFO : [copy-data] completed at: Mon Oct 26 12:53:49 CET 2015
[2015-10-26 13:08:00,854] INFO : Executing update...
[2015-10-26 13:08:00,855] INFO : [update] started at: Mon Oct 26 13:08:00 CET 2015
[2015-10-26 13:08:37,629] INFO : Using RAID engine Tx
[2015-10-26 13:12:14,191] INFO : No data change detected! Aborting...
[2015-10-26 13:12:34,247] INFO : Successful UPDATE operation!


I managed to 'make files visible' through windows, enabling system file view, and then started to excavate the files from this DRU, the DRU2. (3+2tb)

Next I eliminated the 6tb single drive DRU, which was nearly empty in terms of data.

I then proceeded to experimenting with the Perfect Data Copy command. I used the 2tb drive and manually formatting both these drives.

Then I attempted to put only a set of testfiles. Then something interesting showed in the progress window:

Created 11 directories out of 14 total...
Created 86 files out of 88 total...
Copied 517,21 MB out of 517,21 MB total...
WARNING: Memory leak detected on native buffers! Leaks count = 15 out of 37


This is the logfilepart of the same incident:

[2015-10-26 16:55:01,468] INFO : Executing copy-data...
[2015-10-26 16:55:01,470] INFO : [copy-data] started at: Mon Oct 26 16:55:01 CET 2015
[2015-10-26 16:55:10,817] INFO : [copy-data] completed at: Mon Oct 26 16:55:10 CET 2015
[2015-10-26 16:55:11,220] WARN : Memory leak detected on native buffers! Leaks count = 15 out of 37


This seems to be a HBA problem related issue, potentially drivers(?). The Dru2 (2+3tb) and the 6tb are on at least 2 different HBA, perhaps even 3.
However the files got copied and were functional. The command finally functioned, at least under some circumstances.

During the past night I've mustered to remove a single 4tb alone ine DRU(4?). About 15hrs for 2tb data, only for the remove from storage command. Excluding the Perfect-Data-Copy. Is that reasonable? I might add that this time, the files on the drive had 'normal attributes' and were not tucked away in a hidden file structure - which happened during my spanned-drive-dru removal... Something did not went right when removing the first spanned DRU's.

If I intend to just ASAP evaquate from FlexRaid without intention to resurrect the particular raid (planning to start over after reinstall OS), would it be possible to 'shut the syncing off' and then manually all files from the hdds?

BR / MAttzz
« Last Edit: October 27, 2015, 06:22:06 am by Mattzz »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Problems downsizing array of Spanned DRU's
« Reply #5 on: October 27, 2015, 07:09:07 am »
1. If you are doing that restructuring, you are much better off deleting the configuration and reconfiguring things anew.
After you delete the config, you would basically regain control of your disks, organize and clean things up as needed, and then setup a new config. That whole process will take much less time that the course you are on.

2. The memory buffer leak in the Perfect Data Copy task could be a bug in that it is not closing things up properly.
Please log a bug report on bug.flexraid.com and this will be looked into. Thanks.

Offline Mattzz

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: Problems downsizing array of Spanned DRU's
« Reply #6 on: October 27, 2015, 10:54:15 am »
Hello,

I choose to delete the array, then looked over all the files left, including the ones in a hidden directory in the former PPU drive. Then I clear out the flexraid files on the PPU which are not recognizable as my older files.

I sense somethings are missing just by looking at 'My computer'.
Upon closer inspection, to the very LEAST 5tb is gone. Chunks out of different 'root folders' are gone.

Is there ANYTHING in these logs that indicate the complete demise of enough data to EAT +5tb data?

I am currently in of chock I'm ...border line between bummed out and apathy.


 

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Problems downsizing array of Spanned DRU's
« Reply #7 on: October 27, 2015, 11:22:57 am »
Quote
Is there ANYTHING in these logs that indicate the complete demise of enough data to EAT +5tb data?
Quote
I then proceeded to experimenting with the Perfect Data Copy command. I used the 2tb drive and manually formatting both these drives.
Only you can say. Snapshot RAID only does read operations on your data. So, there is nothing that would delete it.

1. What version of RAID-F are you running?
2. You mentioned hidden data on some of the drives, what is the path of that data?

Offline Mattzz

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: Problems downsizing array of Spanned DRU's
« Reply #8 on: November 03, 2015, 03:22:21 am »
Only you can say. Snapshot RAID only does read operations on your data. So, there is nothing that would delete it.

1. What version of RAID-F are you running?
2. You mentioned hidden data on some of the drives, what is the path of that data?

Hi
1. The version availible for trials, about 10 days ago.
2. Do not remember for sure.

This post became slightly longer than anticipated, it is for a good cause.

Here is what I think happened. The data was already lost when I went into flexraid, except I did not notice right away.

I did some experiementing with other softwares (Drivebender/ StableBit) just before trying FlexRaid.
I've spent some time recreating possible scenarios of what could've gone wrong, once more on limited amounts of data, finally I analyzed the remains of my files to figure out what was really gone. (unfortunately a lot more is unusable than actually lost TB's)
The softwares probably did what they should in the first place, where an unfortunate 'balancing plugin setting' started to equalize disk usage among the disks in the setup that caused my files to get 'spread out' in undesirable ways. So far, no data was lost.
However, the action caused my data to become way more vulnerable to hdd failure.
I think the moments when data really got to the point where the fire could be 'handled' and kept from 'burning the house down' was during the processes of 'removing hdd from array' which at some occasions did not went as smoothly as expected at some times during my testing. Probably, this was the cause for files being dropped from the hdpool.

Adding to the potential for fucking up (not appreciating files are actually on really thin ice, and in need of baby sitting the software to ensure everything checks out) is that the software (at least StableBit) hide files on the hdd when subjected to a drivepool. If this concided with some misfortunate 'shrinking of array' causing the files to get lost for sight and somewhere along the lines my hdds showed up max free space. I expanded and shrunk the pools of both drivebender and stablebit several times durign their respective test runs. I can not tell for sure when the problems happened.
In retrospect, there were a number of instances during this process that could've made the files salvageble, but for unfortunate reasons, did not tick in place.

I even did some 40hrs runs for some the hdds that were likely to have at one point at least, have been responsible for the majority of the most important files. None of the recovered bundles were CRC checking correct. :(

Conclusion and shortversion:
-FlexRaid did not fuck up.
-Im more interested in parity functionality now than ever.