FlexRAID Support Forum

General => News and Updates => Topic started by: Brahim on January 28, 2017, 01:03:04 pm

Title: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on January 28, 2017, 01:03:04 pm
Alright guys, time for some releases. :)

We'll start with tRAID and follow suit with RAID-F later. Also, Windows builds will be out first with Linux builds following next.

Please post in this thread if you would like early access to tRAID version 1.1.


Changes:
Code: [Select]
- Lots of performance and stability improvements

- Fixed bugs as reported on http://bug.flexraid.com:
0000447 Starting of storage pool end-up in a blocking loop if regular expression for landing-disk is invalid
0000441 Mercurial still not working correctly (Permission denied)
0000442 Cannot delete Log-Raid fhd file
0000440 File statistics broken after applying Mercurial fix
0000433 Cannot create a Shortcut on a Storage Pool with T-Raid...
0000434 Mercurial push does not work
0000417 tRAID crashes when writing a file that over flows the first drive in the array.
0000416 Raid monitoring plugin, file statistics, and forensic give errors
0000418 Failed File and RAID statistics
0000423 Raid Monitoring Tab has no Data - ErrorCode=31
0000436 No individual file statistics for old drives in Dashboard
0000407 Issue with Physical Drives List
0000425 Recycle bin feature can cause non-empty folders to be deleted

Download:
Host: http://dl.flexraid.com/NZFS-TRAID-HOST-1.1_Preview_2017-01-28.zip
Client: http://dl.flexraid.com/NZFS-CLIENT-1.1_Preview_2017-02-11.exe
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on January 30, 2017, 04:38:26 am
I am interested, provided you consider it stable enough to use it on my "productive" system with real data.  :)
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: hellgate on January 30, 2017, 08:17:47 am
what ver. off Linux will you bee supporting with tRAID version 1.1.

Thanks..
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: JasonX on January 30, 2017, 08:50:24 am
Ohhh I would be interested in Version 1.1!
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: hellgate on January 31, 2017, 09:19:32 am
well I'M interested on the Linux builds..!!

Thanks..
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 01, 2017, 03:45:57 pm
well I'M interested on the Linux builds..!!

Thanks..
Waiting on 16.04.2 for the Linux release. Should be out any day now.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: hellgate on February 01, 2017, 05:47:07 pm
Waiting on 16.04.2 for the Linux release. Should be out any day now.

Thank you very much..
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: d2dyno on February 03, 2017, 08:18:54 pm
Can this be used in a production environment/live data, or not recommended?
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 04, 2017, 09:05:17 am
Alright, added the download links to the first post.
Please install, monitor, and report. Thanks.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 04, 2017, 11:26:16 am
How should an "update" be performed? Uninstall the old version and install the new? Or is it sufficient to just stop the services and copy the contents/install the exe?
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 04, 2017, 03:15:22 pm
How should an "update" be performed? Uninstall the old version and install the new? Or is it sufficient to just stop the services and copy the contents/install the exe?

http://wiki.flexraid.com/2013/06/27/uninstalling-or-upgrading-the-transparent-raid-host-package/
http://wiki.flexraid.com/2013/06/27/uninstalling-or-upgrading-the-transparent-raid-web-client-package/
Title: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Wilky13 on February 05, 2017, 12:16:29 am
Yes please.  Running Windows 7 Home Edition. 


Sent from my iPad using Tapatalk
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 05, 2017, 04:04:11 pm
Thanks for the preview.

Feedback so far:
1. After login to the web ui, I (always) get "com.google.gwt.user.client.rpc.StatusCodeException: 500 ..." (see screenshots and log)
2. In the Web UI log, I get a multitude of warnings and specifically a repeated error: "RegisteredDisk invalid search ID=0" (attached the complete log after installation)
3. Dashboard is completely empty (would expect CPU/mem meter and files statistics there)
4. The array itself seems to be running fine so far.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 05, 2017, 07:55:52 pm
Fixed for the next build.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: JasonX on February 06, 2017, 05:18:48 am
I also have running the new version for round about 2 days.
I noticed no problems. The Web-UI problem is not present for me.

I noticed:
1) In the Log-File the old version is shown "v1.0.0"
2) I still going crazy with the regular expression stuff. The lock is solved, but I still get no proper exclusion-function. I will reply on the original post, if I have time to try some more things.
3) The parity calculation seems to be SIMD/SSE enhanced now!? Works faster.
4) I noticed, also on the previous version, the SMART monitor option to "Enable" the Monitoring on a specific drive sometimes disapear. (see Attachment) If once configured all drives to enable this function, after a time some are not enabled anymore and I don't know why. Most of the times only my SSD keeps the option enabled, for all other drives the option is gone later the day. Currently all options are enabled as expected, but only on one HDD drive it is gone.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 06, 2017, 07:27:53 am
Typos:
For the SIMT feature tooltip, I think it should be "past" instead of "passed" (brackets at the end)?
For the CQ Depth tooltip (about in the middle of the text): "accessed" instead of "access".

For TCQ the default depth is 8, but the tooltip says it's 32. (Which one would you actually recommend as default?)
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 06, 2017, 08:18:34 am
Will resolve all posted issues and will push a new build later today.

FYI, the SIMT feature will be removed from the UI as it is a major feature I did not intend to have in this release. Plus, only part of it is being enabled in this build. The full stack is rather cool, and we will discuss of that later. I did not mean to let this cat out of the bag. ;)
This release is really supposed to be a bug fix release with no major feature.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 06, 2017, 08:21:17 am
Okay, I disable it again ... ;-)
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 07, 2017, 04:07:13 am
I'm not sure whether the data I get in the RAID Monitoring tab is reasonable.

I've added a screenshot of a disk where I've definitely written to since the update. Nevertheless, I see low write speed and 0 % Write Update.

The other disks are similar, but to those I may have not written so far.

However, as I don't know which set the median is deduced from, I cannot judge this further.

BTW: What's the "last update" in the "Since last update" option.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 07, 2017, 11:02:59 am
"Since Last Update" is since the time when the last statistic was collected. It is usually in seconds. I can't remember the exact duration, but is should be 15 seconds or less. It is supposed to represent the current (as in within the last few seconds) performance metrics.

Otherwise, the "Median" values are averages since the array has been up.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 08, 2017, 06:00:29 am
Looks like the preview resolved my issue with files being deleted when moved (http://bug.flexraid.com/view.php?id=431).

I will investigate this further with the new build and report again ...
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 08, 2017, 09:06:04 am
Looks like the preview resolved my issue with files being deleted when moved (http://bug.flexraid.com/view.php?id=431).

I will investigate this further with the new build and report again ...
With ReFS?
The defect is still open only because of the issue moving from the pool to disk outside the pool on ReFS.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 08, 2017, 09:09:16 am
Yes, with ReFS. Just realized it, did not have time to evaluate further. And obviously I do not have any idea why it seems to work now...
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 08, 2017, 09:15:52 am
Just checked: I also have exactly the same settings for array and pool. Must be some change in the preview. Will post again as soon as I've tested more extensively.

Despite that, I still have severe problems with Mercurial. Trying to narrow it down to something reproducible/evaluable at moment ...
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: d2dyno on February 08, 2017, 08:34:54 pm
Will resolve all posted issues and will push a new build later today.

FYI, the SIMT feature will be removed from the UI as it is a major feature I did not intend to have in this release. Plus, only part of it is being enabled in this build. The full stack is rather cool, and we will discuss of that later. I did not mean to let this cat out of the bag. ;)
This release is really supposed to be a bug fix release with no major feature.
Where can I get the latest build then?
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 09, 2017, 05:43:32 am
Current state on the moving files problem:

Scenario 1: Accessing pool via network share and moving folder to local drive (drag and drop)
-> Folder structure is preserved during move (=successful)
-> moved files do NOT show up in the URB (which I would call reasonable behavior)
= desired behavior

Scenario 2: Accessing pool locally (via Remote Desktop) and moving folder to non-FlexRaid drive (drag and drop)
-> Target folder is empty
-> moved folder and subfiles DO show up in the URB
= earlier behavior

Scenario 3: Accessing pool locally as in Scenario 2 but moving files with PowerShell's MoveItem
-> Target folder is created and complete
-> the URB only contains the moved folder without subfiles/-folders
EDIT:
-> the URB contains all subfiles and -folders as individual entries, indicating that they are moved and deleted sequentially by the command

In any case, although the current behavior is somewhat erratic, it is much preferable to the previous situation, as network share access is the most common usage scenario for me and no data is lost in any configuration.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 09, 2017, 09:31:01 am
Thanks for the update.

Scenario 3 is to be expected for applications that choose to have atomic operations on the individual files. This is what happens when applications code strictly against a known behavior rather than a contract.
The contract is such that folders and files can be deleted. Whether there are possible constraints such as ACL or certain file systems requiring that folders be empty first is a separate concern. A proper approach would be to try to delete the folder first (whether empty or not) and then delete the sub-entries if there is a failure that dictates so.

For scenario 1, what is the OS of the client machine? I am a bit worried that later versions of SMB against a host running ReFS might lead to the same behavior as scenario 2.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 09, 2017, 09:39:27 am
Scenario 3 was only added for completeness and for your information, since it is not a usage scenario for me, but did work already before the preview.

For all current and earlier tests, client has always been Win 8.1 Prof x64 and server Win 2012 R2. If required, I also have a Win 10 client machine.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 09, 2017, 11:30:10 am
Scenario 3 was only added for completeness and for your information, since it is not a usage scenario for me, but did work already before the preview.
...
Yep, got that... was only providing details whether useful or not. :)
Now, are you saying the behavior in Powershell is different between the official release and this preview release? How so?

Quote
For all current and earlier tests, client has always been Win 8.1 Prof x64 and server Win 2012 R2. If required, I also have a Win 10 client machine.
A test using Win10 would be great. If we can validate the one ReFS odd behavior is limited to that one scenario, the greater the opportunity to formulate a workaround if not a fix.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 09, 2017, 11:37:49 am
Now, are you saying the behavior in Powershell is different between the official release and this preview release? How so?

No, PowerShell was the only way to move files successfully BEFORE the preview and has the same behavior since it still works (do not remember what happened to the URB then). If you look at the bug report, there was a point where I realized that although drag-and-drop did not work, I could move with PowerShell. However, this was already a test build provided by you, so I never tested this with the real release version. What's new in the preview is that moving from the network share is now possible, this was not the case in any previous build (except the compatibility mode of the URB).
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 09, 2017, 11:57:21 am
Ah, cool. Thanks for the clarification. :)
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: xexitenglishx on February 09, 2017, 05:04:03 pm
I would love early access to this preview.  I have been having issues lately and have been thinking over moving over to a qnap system..would love to not have to do that!
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 10, 2017, 04:23:14 am
Same behavior on Win 10 x64 as on Win 8.1 (accessing network share):

Folders are moved successfully, nothing shows up in the URB => desired behavior.

(Just for completion: Local target drive may be NTFS or ReFS, both works.)
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Reddwarfusa on February 10, 2017, 09:54:50 am
Ok here goes, installation on server 2012 R2.  Fingers crossed.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 11, 2017, 07:48:03 am
1) In the Log-File the old version is shown "v1.0.0"
...
Which log file?

Quote
4) I noticed, also on the previous version, the SMART monitor option to "Enable" the Monitoring on a specific drive sometimes disapear. (see Attachment) If once configured all drives to enable this function, after a time some are not enabled anymore and I don't know why. Most of the times only my SSD keeps the option enabled, for all other drives the option is gone later the day. Currently all options are enabled as expected, but only on one HDD drive it is gone.
SMART will auto disable if querying the drive for it fails more than 10 times.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 11, 2017, 07:29:48 pm
Update

A new version of the Web Client has been released (see first post).
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 12, 2017, 08:06:02 am
Very nice. So far, everything seems resolved except:

- In the dashboard (see attached screenshot), the individual files statistics of the discs have particular sectors permanently enlarged (e.g. the purple one for "HD103Server"). In contrast, the dynamic resizing occurring for the pool chart does not happen here.
- The readability of the individual files statistics is worse compared to the pool chart (maybe use the same design? This is not a bug, but a suggestion).
- After I had collapsed the chart for "HGST3", uncollapsing it gives my an empty panel as seen in the screenshot.

Sidenote:
The log file with the old version number is the NZFS Service Log:
"NZFS Service Start v1.0.0 2017.01.28"
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 12, 2017, 08:13:58 am
I also still see slight rearrangements on the dashboard as described in http://bug.flexraid.com/view.php?id=388.

Despite that, as reported earlier, the data on the RAID Monitoring tab does not look reasonable to me, for I have mostly Read Update and always 0 percent Write Update.

Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 12, 2017, 01:22:45 pm
...
- In the dashboard (see attached screenshot), the individual files statistics of the discs have particular sectors permanently enlarged (e.g. the purple one for "HD103Server"). In contrast, the dynamic resizing occurring for the pool chart does not happen here.
...
The charts are all different and convey different information. This is on purpose.
The disk charts show both data size as pie chart percentage and file count as relative wedge size.

For the pool disk, the pie is different as it only shows data size percentages.

Quote
- The readability of the individual files statistics is worse compared to the pool chart (maybe use the same design? This is not a bug, but a suggestion).
...
There is no way to control that on small screens. Readability improve with screen size.

Quote
- After I had collapsed the chart for "HGST3", uncollapsing it gives my an empty panel as seen in the screenshot.
...
Please add a bug report to the bug tracker.

Quote
...
Sidenote:
The log file with the old version number is the NZFS Service Log:
"NZFS Service Start v1.0.0 2017.01.28"
Fixed. Thanks.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 12, 2017, 01:29:35 pm
I also still see slight rearrangements on the dashboard as described in http://bug.flexraid.com/view.php?id=388.
...
Yeah, still unresolved.

Quote
...
Despite that, as reported earlier, the data on the RAID Monitoring tab does not look reasonable to me, for I have mostly Read Update and always 0 percent Write Update.
That's a good thing. ;)
You want the least amount of Write Update.

A Write Update is one where the same data is being rewritten due to parity stripe alignment.
http://wiki.flexraid.com/2013/06/27/understanding-the-raid-monitoring-plugin/
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 12, 2017, 02:17:45 pm
Just found a folder with 200-300 GB of files to be just empty. Any ideas on this or earlier posts regarding the same issue? Nothing in any bin and I surely didn't delete it.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 12, 2017, 02:19:24 pm
Just found a folder with 200-300 GB of files to be just empty. Any ideas on this or earlier posts regarding the same issue? Nothing in any bin and I surely didn't delete it.
Details?
Don't remember nor understand this issue.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 12, 2017, 02:23:29 pm
A had a folder with some hundred GB of video files (only .mpg, no subfolders). Now the folder is just empty both when I access the individual tRaid drive and the pool.

However, if I sum up the amount of data in all folders, it's about 500 GB less than the reported used space of the tRaid drive. Looks like the data still is somewhere. Maybe I should look at the source disk?
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 12, 2017, 02:30:44 pm
Adding up the reported files after a file statistics run gives 2251 GB, the used space in the drive properties is 2.76 GB. The difference would be about the amount of files missing.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 12, 2017, 02:34:30 pm
Just did a scan with ReclaiMe File Recovery: This tool shows all the files (526.8 GB) in the correct spot.

tRaid Log show does not show anything; permissions issue should also be excludable, as I'm using an administrator account and accessed the data just yesterday.

Note that filesystem is ReFS.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 12, 2017, 02:37:46 pm
Would it be possible to just reinstall the 1.0 versions and see whether files come back?
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 12, 2017, 02:43:08 pm
Just mounted the source drive: Files are still not shown, but used space still includes them. So FlexRaid is not hiding them, but must have made them invisible somehow in the first place.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 12, 2017, 02:46:26 pm
Despite that, during every second/third mount of the tRaid drive the mount fails and gives my the appearance of an MBR-formatted drive in Disk Management (see screenshot). When I stop/start the array again, the drive is just mounted fine.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 12, 2017, 03:03:52 pm
Any ideas how to recover the data in this case? I am not very keen on spending 50-100 Euros for a ReFS recovery software.

Quite strange BTW:
Only this ReclaiMe software display the missing files (looks like it does a complete scan of the drive in the background).
RStudio is not even able to correctly show me the existing files!!
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 12, 2017, 11:13:44 pm
Any ideas how to recover the data in this case? I am not very keen on spending 50-100 Euros for a ReFS recovery software.

Quite strange BTW:
Only this ReclaiMe software display the missing files (looks like it does a complete scan of the drive in the background).
RStudio is not even able to correctly show me the existing files!!
I truly have no idea on this one. Corrupted file system maybe?
Did you check the Windows event logs for possible file system error notifications?
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 13, 2017, 12:30:15 am
Should look in the event log more often:

Got four Errors 133 (two times each):
The file system detected a checksum error and was not able to correct it. The name of the file or folder is "\\?\Volume{ID}\<unable to determine file name>".
The file system detected a checksum error and was not able to correct it. The name of the file or folder is "\\?\Volume{Other ID}\FilmeHD\_InputHD"

So, I've checked and I have exactly the same issue for another folder on another disk (!). While for one disk I would have blamed ReFS, in this case I doubt it's a coincidence.

So it narrows down to two question:
1. How/why did the file system got corrupted?
2. How can one deal with a corrupt ReFS file system other than reformatting?

Both folder were not accessed by me at the specified time and have definitely not been written to.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 13, 2017, 04:46:02 am
Despite what caused my problems yesterday, I consider leaving ReFS and going back to NTFS for my array (while I still can). Therefore, I would like your advice on two aspects:

1. If you do have one, I would be interested in your opinion on NTFS vs ReFS (1.2)

2. From a technical perspective, I would use the following procedure to migrate my whole array to NTFS with uninterrupted parity protection (do you confirm that?):
   a) disable pool and mount tRaid drives
   b) move data off one of the tRaid disks to external drives
   c) reformat the empty drive to NTFS (SWO/TCQ off?)
   d) move data from the second (refs) drive to the first, now empty, drive (NTFS)
   e) reformat second drive, move 3->2, reformat 3rd, move 4->3, reformat 4th
   f) move the data from the external drive to the, now empty, 4th drive

If I do that, the parity should protect everything on the pool, while the only data at risk are the files on the external drives? Without the array, mixed mode of NTFS/ReFS should not be a problem, will it?
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: JasonX on February 13, 2017, 05:08:49 am
Which log file?

This is answered by adridolf already: FlexRAID.nzfx.log

SMART will auto disable if querying the drive for it fails more than 10 times.

Of course, this makes sense, since the drives go into spin-down after some time. What does not makes sense: The second option to "skip SMART monitoring" if the drives are in Spin-Down. As of my understanding: The combination of both parameters make sure the SMART monitoring is enabled but will not spin-up any drive, so the monitoring is done once the drive is online again.
Does your answere mean, the option is somehow useless? I think the monitoring should be still keep enabled in this combination!?

But I've checked this: After drives fall into spin-down mode, the option is gone after some time.

Is it possible to implement a better/smarter handling of this? Even by a third option, maybe "ignore communication errors" or something!?
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 13, 2017, 11:15:02 am
...
1. How/why did the file system got corrupted?
...

Do you have caching enabled on the pool? There is a limitation that the tRAID disks should NOT be be accessed (they should not have a drive letter) while the pool is running with caching enabled.

Despite what caused my problems yesterday, I consider leaving ReFS and going back to NTFS for my array (while I still can). Therefore, I would like your advice on two aspects:

1. If you do have one, I would be interested in your opinion on NTFS vs ReFS (1.2)

2. From a technical perspective, I would use the following procedure to migrate my whole array to NTFS with uninterrupted parity protection (do you confirm that?):
   a) disable pool and mount tRaid drives
   b) move data off one of the tRaid disks to external drives
   c) reformat the empty drive to NTFS (SWO/TCQ off?)
   d) move data from the second (refs) drive to the first, now empty, drive (NTFS)
   e) reformat second drive, move 3->2, reformat 3rd, move 4->3, reformat 4th
   f) move the data from the external drive to the, now empty, 4th drive

If I do that, the parity should protect everything on the pool, while the only data at risk are the files on the external drives? Without the array, mixed mode of NTFS/ReFS should not be a problem, will it?
1. I don't use ReFS. File systems from my experience take a very long time to fully mature.

2. You could do as you outlined. Things would go much faster if you were to delete the config, regain control of your disks, and do the migration offline.

Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 13, 2017, 11:43:29 am
Do you have caching enabled on the pool? There is a limitation that the tRAID disks should NOT be be accessed (they should not have a drive letter) while the pool is running with caching enabled.

What caching do you mean precisely?
If have OS caching disabled and SWO/TCQ enabled. Storage pool caching has always been disabled for this particular reason.
However, during the time when the file system failed only the pool was mounted and none of the individual drives.

During my own research today I've found several people referring to problems with ReFS and RAID systems in general, e.g. the last post on https://social.technet.microsoft.com/Forums/windowsserver/en-US/171a1808-157e-4ef9-b0dd-8a507ff6fcef/refs-corruption-when-filled-to-capacity?forum=winserver8gen

However, everything I found was very vague.

I am done with ReFS and will switch to NTFS now.

(Since I don't think this is particularly related to the preview, you may move the ReFS failure posts to a different topic if you like, since a cannot do that.)
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 13, 2017, 11:45:16 am
2. You could do as you outlined. Things would go much faster if you were to delete the config, regain control of your disks, and do the migration offline.

Thanks for the response.

I agree. However, in this case, I would not have the possibility to recover any data in case of a drive failure during migration until the parity is recomputed, would I?
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 13, 2017, 07:06:28 pm
Thanks for the response.

I agree. However, in this case, I would not have the possibility to recover any data in case of a drive failure during migration until the parity is recomputed, would I?
I worry more about you formatting the wrong disk(s) and other user errors more so than a disk failing in that window. The shorter the window the better - and, doing it offline will go faster.

Parity won't help you much if you format the wrong disk or corrupt a disk. By doing it outside of the RAID, you are also not stressing the parity disk.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 14, 2017, 05:12:04 am
There is no way to control that on small screens. Readability improve with screen size.

I was referring to the white text and the white border of the pie sectors: I would consider it more readable with black text and without the white borders, like for the pool chart.

That's a good thing. ;)
You want the least amount of Write Update.

A Write Update is one where the same data is being rewritten due to parity stripe alignment.
http://wiki.flexraid.com/2013/06/27/understanding-the-raid-monitoring-plugin/

I read the article before and understood it differently then. But if the screenshot does not look odd to you, it's okay.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 14, 2017, 05:17:07 am
When reformatting from ReFS to NTFS, I had to switch from PERFORMANCE to ENERGY_EFFICIENCY to be able to format the disk (in addition to switching off TCQ and SWO).

This has not been necessary earlier, when I formatted new disks during Expansions. Thus, this could be either specific to NTFS (earlier formats were ReFS) or to the preview.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 16, 2017, 10:35:20 am
Current state on the moving files problem:

Scenario 1: Accessing pool via network share and moving folder to local drive (drag and drop)
-> Folder structure is preserved during move (=successful)
-> moved files do NOT show up in the URB (which I would call reasonable behavior)
= desired behavior

Scenario 2: Accessing pool locally (via Remote Desktop) and moving folder to non-FlexRaid drive (drag and drop)
-> Target folder is empty
-> moved folder and subfiles DO show up in the URB
= earlier behavior

Scenario 3: Accessing pool locally as in Scenario 2 but moving files with PowerShell's MoveItem
-> Target folder is created and complete
-> the URB only contains the moved folder without subfiles/-folders
EDIT:
-> the URB contains all subfiles and -folders as individual entries, indicating that they are moved and deleted sequentially by the command

In any case, although the current behavior is somewhat erratic, it is much preferable to the previous situation, as network share access is the most common usage scenario for me and no data is lost in any configuration.

Exactly the same behavior after having changed to NTFS (this issue is not ReFS-specific)
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 16, 2017, 11:28:01 am
Exactly the same behavior after having changed to NTFS (this issue is not ReFS-specific)
Broker logs will be needed here to substantiate.
The fix was tested to work perfectly for NTFS.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 17, 2017, 04:55:38 am
I have exactly the same behavior as I reported for ReFS. I appended a log file where I moved three folders named scenario1, scenario2 and scenario3 according to the earlier descriptions.

You have to scroll to the section from 2017-02-17 at about 12 o'clock. Normal recycle bin is disabled.

Still, this is perfectly acceptable for me, I just want to report.

To add something positive:
Note that through my whole migration process of about 12 TB not a single log entry has been created and the parity seems to be perfectly fine. The few remaining log entries are from the Mercurial issue (code=5) and my attempts to mount as removable (Code=87).
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 17, 2017, 04:59:22 am
The issue with the collapsed drive file statistics chart was actually my own stupidity: After uncollapsing, it additionally had to be resized.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 18, 2017, 06:53:23 am
I have exactly the same behavior as I reported for ReFS. I appended a log file where I moved three folders named scenario1, scenario2 and scenario3 according to the earlier descriptions.

You have to scroll to the section from 2017-02-17 at about 12 o'clock. Normal recycle bin is disabled.

Still, this is perfectly acceptable for me, I just want to report.

To add something positive:
Note that through my whole migration process of about 12 TB not a single log entry has been created and the parity seems to be perfectly fine. The few remaining log entries are from the Mercurial issue (code=5) and my attempts to mount as removable (Code=87).
Quite strange.

Scenario 1 & 3, the folder is opened without intent for deletion but gets deleted. Why Explorer would do that is a mystery.
In scenario 2, the empty folder gets opened for deletion and gets deleted. Another mystery is that was not the intended behavior.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 20, 2017, 06:07:38 am
Still no issues with the latest preview. Seems to work well. (Except the strange moving files issue)

Looks like NTFS also resolved my remaining Mercurial issues. At least I was not able to trigger any repository corruption so far during my tests ...

Only a slight remark: I occasionally get warnings "Exception while dispatching incoming RPC call org.eclipse.jetty.io.EofException" etc. in the Web UI log. This is typically accommodated by several "stop(41)" info entries during START of the Web UI (at least that's what the timestamp says). Looks like something get's stuck during shutdown there. I appended some excerpts of the web ui log to clarify. However, I do not detect any impact on the behavior of the Web UI itself.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 20, 2017, 09:09:48 am
...
Only a slight remark: I occasionally get warnings "Exception while dispatching incoming RPC call org.eclipse.jetty.io.EofException" etc. in the Web UI log. This is typically accommodated by several "stop(41)" info entries during START of the Web UI (at least that's what the timestamp says). Looks like something get's stuck during shutdown there. I appended some excerpts of the web ui log to clarify. However, I do not detect any impact on the behavior of the Web UI itself.
There is a file named log4j.properties in the installation directory.
There should be a line that reads "log4j.logger.org.eclipse.jetty=ERROR".
Do you have that line?
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 20, 2017, 09:18:35 am
Yes:

[...]

log4j.logger.com.tchegbe=INFO
log4j.logger.org.eclipse.jetty=ERROR
log4j.logger.org.quartz=ERROR
log4j.logger.com.google.gwt=ERROR
log4j.logger.com.metaparadigm.jsonrpc=ERROR
log4j.logger.java.lang.Thread=ERROR
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 20, 2017, 12:44:08 pm
Please update that entire section to:
Code: [Select]
log4j.logger.com.tchegbe=INFO
log4j.logger.org.eclipse.jetty=ERROR
log4j.logger.org.quartz=ERROR
log4j.logger.com.google.gwt=ERROR
log4j.logger.com.metaparadigm.jsonrpc=ERROR
log4j.logger.java.lang.Thread=ERROR
log4j.logger.javax.servlet=ERROR

Changes have been made for the next build.
Thanks.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 20, 2017, 01:54:28 pm
Strange:

After I had updated the log properties file and restarted the server, the success e-mail message of the last Verify Sync was resent (it initially arrived 5 hours ago).
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 20, 2017, 03:41:38 pm
Confirmed. Please add to bug.flexraid.com as low priority for later.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 21, 2017, 12:32:46 pm
http://bug.flexraid.com/view.php?id=451
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 21, 2017, 12:36:16 pm
FYI:

With log properties file changed, I get the same warnings, but now during the shutdown period.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 21, 2017, 12:48:43 pm
FYI:

With log properties file changed, I get the same warnings, but now during the shutdown period.
As ERROR or WARNING.
The change is more of a log filter than resolving the error itself. It is actually not a defect, which is why suppressing the log is the proper thing to do.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 21, 2017, 01:55:01 pm
Warning. As before (log attached).
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on February 21, 2017, 04:13:16 pm
The log is exactly the same as before.
The recent entries show no issue. Meant to attach something different?
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on February 22, 2017, 03:54:29 am
The log is exactly the same as before.
The recent entries show no issue. Meant to attach something different?

Not exactly: The "Exception while dispatching incoming RPC call org.eclipse.jetty.io.EofException" is now actually occurring during shutdown (2017-02-20 23:18).

Earlier, it was logged only at machine startup. Maybe coincidence.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: e.Blue on March 08, 2017, 02:35:27 pm
I'm building a new server. My old server is running RAID-F. I'm planning on moving my drives over to tRAID and would like to use the v1.1 preview release. Are there any issues that I should consider before beginning the move?

-e.B

...also is there any discount for buying tRAID for longtime RAID-F owners? :D
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on March 09, 2017, 01:35:42 pm
Update

FYI, working on the Linux builds now that 16.04.2 is out.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: cogliostrio on March 10, 2017, 04:44:04 am
Update

FYI, working on the Linux builds now that 16.04.2 is out.

Yay  ;D
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on March 20, 2017, 10:45:42 am
While the preview is still running smoother than any version before, I found a new error in the WebUI log (it has occurred about every second day for three weeks, with the time looking random to me):

[2017-03-16 11:53:25,215] ERROR schedulerError(2337) - An error occured while scanning for the next trigger to fire.
org.quartz.JobPersistenceException: Couldn't acquire next trigger: null [See nested exception: java.lang.ArrayIndexOutOfBoundsException]
   at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2789)
   at org.quartz.impl.jdbcjobstore.JobStoreSupport$36.execute(Unknown Source)
   at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3763)
   at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2728)
   at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:264)
Caused by: java.lang.ArrayIndexOutOfBoundsException
   at org.neodatis.odb.core.layers.layer2.meta.NonNativeObjectInfo.setAttributeValue(NonNativeObjectInfo.java:313)
   at org.neodatis.odb.core.layers.layer3.Layer3ReaderImpl.internalFromBytes(Layer3ReaderImpl.java:213)
   at org.neodatis.odb.core.layers.layer3.Layer3ReaderImpl.metaFromBytes(Layer3ReaderImpl.java:107)
   at org.neodatis.odb.core.session.SessionEngineImpl.layer3ToLayer2(SessionEngineImpl.java:305)
   at org.neodatis.odb.core.session.SessionEngineImpl.getMetaObjectFromOid(SessionEngineImpl.java:520)
   at org.neodatis.odb.core.query.nq.NativeQueryExecutor.matchObjectWithOid(NativeQueryExecutor.java:57)
   at org.neodatis.odb.core.query.GenericQueryExecutor.executeFullScan(GenericQueryExecutor.java:291)
   at org.neodatis.odb.core.query.GenericQueryExecutor.execute(GenericQueryExecutor.java:169)
   at org.neodatis.odb.core.session.SessionEngineImpl.execute(SessionEngineImpl.java:753)
   at org.neodatis.odb.core.query.AbstractQuery.objects(AbstractQuery.java:140)
   at com.tchegbe.lib.ext.scheduler.NeoDatisQuartzDriverDelegate.selectTriggerToAcquire(NeoDatisQuartzDriverDelegate.java:73)
   at com.tchegbe.lib.ext.scheduler.ObjectDBQuartzDriverDelegate.selectTriggerToAcquire(ObjectDBQuartzDriverDelegate.java:1088)
   at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2746)
   ... 4 more
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on March 20, 2017, 10:59:09 am
While the preview is still running smoother than any version before, I found a new error in the WebUI log (it has occurred about every second day for three weeks, with the time looking random to me):

[2017-03-16 11:53:25,215] ERROR schedulerError(2337) - An error occured while scanning for the next trigger to fire.
org.quartz.JobPersistenceException: Couldn't acquire next trigger: null [See nested exception: java.lang.ArrayIndexOutOfBoundsException]
   at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2789)
   at org.quartz.impl.jdbcjobstore.JobStoreSupport$36.execute(Unknown Source)
   at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3763)
   at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2728)
   at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:264)
Caused by: java.lang.ArrayIndexOutOfBoundsException
   at org.neodatis.odb.core.layers.layer2.meta.NonNativeObjectInfo.setAttributeValue(NonNativeObjectInfo.java:313)
   at org.neodatis.odb.core.layers.layer3.Layer3ReaderImpl.internalFromBytes(Layer3ReaderImpl.java:213)
   at org.neodatis.odb.core.layers.layer3.Layer3ReaderImpl.metaFromBytes(Layer3ReaderImpl.java:107)
   at org.neodatis.odb.core.session.SessionEngineImpl.layer3ToLayer2(SessionEngineImpl.java:305)
   at org.neodatis.odb.core.session.SessionEngineImpl.getMetaObjectFromOid(SessionEngineImpl.java:520)
   at org.neodatis.odb.core.query.nq.NativeQueryExecutor.matchObjectWithOid(NativeQueryExecutor.java:57)
   at org.neodatis.odb.core.query.GenericQueryExecutor.executeFullScan(GenericQueryExecutor.java:291)
   at org.neodatis.odb.core.query.GenericQueryExecutor.execute(GenericQueryExecutor.java:169)
   at org.neodatis.odb.core.session.SessionEngineImpl.execute(SessionEngineImpl.java:753)
   at org.neodatis.odb.core.query.AbstractQuery.objects(AbstractQuery.java:140)
   at com.tchegbe.lib.ext.scheduler.NeoDatisQuartzDriverDelegate.selectTriggerToAcquire(NeoDatisQuartzDriverDelegate.java:73)
   at com.tchegbe.lib.ext.scheduler.ObjectDBQuartzDriverDelegate.selectTriggerToAcquire(ObjectDBQuartzDriverDelegate.java:1088)
   at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2746)
   ... 4 more

Please add to bug.flexraid.com and I will take a look.
Thanks.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: jeff on June 14, 2017, 10:54:31 pm
I'm in the process of setting up my system again after an SSD failure and wanted to check out the new preview version as it's been a while since the last tRAID update. I briefly had the latest release version installed, but didn't set anything up yet, uninstalled and installed the preview version in the first post (both host and client).

After rebooting, opening the web client, and logging in, I got this error: "com.google.gwt.user.client.rpc.StatusCodeException: 500 The call failed on the server; see server log for details"

I attached the NZFSWebUI.log which has a stack trace for com.google.gwt.user.client.rpc.SerializationException.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: jeff on June 14, 2017, 11:41:48 pm
I'm in the process of setting up my system again after an SSD failure and wanted to check out the new preview version as it's been a while since the last tRAID update. I briefly had the latest release version installed, but didn't set anything up yet, uninstalled and installed the preview version in the first post (both host and client).

After rebooting, opening the web client, and logging in, I got this error: "com.google.gwt.user.client.rpc.StatusCodeException: 500 The call failed on the server; see server log for details"

I attached the NZFSWebUI.log which has a stack trace for com.google.gwt.user.client.rpc.SerializationException.

Looks like the issue was related to something cached by the browser. I cleared the cache and cookies and the web client loads properly now. Probably something cached from the previous version that's not compatible. Good to find that is the solution, although it'd be good if the web client could handle this upgrade scenario gracefully.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on June 15, 2017, 06:35:48 am
Hello jeff,

I had this message when using the first preview version:

http://forum.flexraid.com/index.php/topic,49184.msg77024.html#msg77024

After having updated to the second preview, it vanished.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: harakiri on July 18, 2017, 01:01:14 pm
any news for the final version?
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: b-earl on July 23, 2017, 02:22:37 am
Thanks for a new preview Brahim!
Trying out on my new installed Windows Server 2016 with bitlocker on the data drives. At the moment the array is initializing. Will get back if something is strange.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: cogliostrio on August 05, 2017, 03:11:29 am
Update

FYI, working on the Linux builds now that 16.04.2 is out.

Any updates on this?
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: wdeydwondrer on August 13, 2017, 10:56:37 pm
Any news on the preview or final?
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on August 14, 2017, 07:57:39 am
Any news on the preview or final?

Still no ETA. Nonetheless, this would be the first thing to be pushed out.
There have certainly been long delays, but the hope is to have things resume this Fall.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: harakiri on August 22, 2017, 01:14:26 pm
Hi Brahim,
please keep us more informated. Actually I have no good feeling about the future. My tRAID running on a 2012 R2 but my hardware must be changed next year and there I have a decision for what I want to choose my storage software.  I want to migrate my data to a Server 2016, but if I have the feeling that tRAID is dead, I will make a Raid1 if there isn't any new news :( In the past I loved your software.

So PLEASE keep us more up2date or give a roadmap. The latest untested preview from tRAID is from 2017-01-28 :(
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: adridolf on November 23, 2017, 08:14:18 am
Is it safe to upgrade from the latest preview to the new release (2017.11.22)?

Upgrade is done by the normal procedure from the wiki?

http://wiki.flexraid.com/2013/06/27/uninstalling-or-upgrading-the-transparent-raid-host-package/

http://wiki.flexraid.com/2013/06/27/uninstalling-or-upgrading-the-transparent-raid-web-client-package/
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on November 23, 2017, 08:23:33 am
It is safe and recommended. :)
Also, hammer it and report any defect.
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: harakiri on November 24, 2017, 01:35:22 am
Brahim,
thank you for the latest version. Please tell us more news or a roadmap for the future. Actually I am unsure If I will keep tRaid on my next fileserver, beginning in 2018 or 2019. The forum and flexraid project looking dead, I don't know what exactly planning in the future.

I don't know how good the newest version is tested out to avoid critical errors with my 8 TB data  :-\
Title: Re: Transparent RAID (tRAID) Version 1.1 Preview release
Post by: Brahim on November 24, 2017, 08:47:21 am
Brahim,
thank you for the latest version. Please tell us more news or a roadmap for the future. Actually I am unsure If I will keep tRaid on my next fileserver, beginning in 2018 or 2019. The forum and flexraid project looking dead, I don't know what exactly planning in the future.

I don't know how good the newest version is tested out to avoid critical errors with my 8 TB data  :-\
Baby steps. ;)

In order of importance:
1. Refresher release updates (with all critical bugs resolved, new Linux kernels support, etc. - so we can pick back up where we left off)
2. Major improvement releases (already done - just need to be released)
3. Major re-architecture/redesign releases (to modernize all products)
4. Major product extension releases (more on this later)

ETA for refresher releases: happening now... tRAID for Linux being pushed out in the next few days
ETA for major improvement releases: likely January with beta builds out in the new few weeks...
ETA for major re-architecture/redesign releases: Spring 2018
ETA for major product extension release: no ETA... we'll see based on the beta releases

The current refresher releases were heavily tested both internally and externally by a number of users. So, these are the releases you want to be on. ;)