Author Topic: Bad performance? Post here!  (Read 44284 times)

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,541
  • Karma: +204/-16
    • View Profile
Bad performance? Post here!
« on: September 26, 2013, 10:59:11 am »
Purpose
The key purpose of this thread is to help users fine tune their systems and help resolve bad performance issues.
There is an important feature coming out soon that will boost the performance of Transparent RAID.
Nonetheless, it is important that the raw speeds (mostly write speeds) be adequate.

Expected performance
We define raw speed as the disk speed without the effect of OS caching.
When you copy a file, your OS will buffer the data into its cache and give you a speed measure that does not reflect of the true speed to the disk.
As shown in this post, your OS will continue copying the data to the disk in the background long after it gives you a completion notice on the file copy. So, your real disk performance is skewed as it is really much lower than reported.

Raw write speeds do take a hit under tRAID due to the parity updates cost.
Typically, sustained raw write speeds under tRAID should be around 1/3 and up to 65% of what the disk would do if it wasn't part of the RAID.

Getting your true disk speed
The Transparent RAID UI since RC9 now allows you to benchmark your disk.
To run a benchmark on a disk, simply right-click on the physical disk in the Web UI explorer panel and select "Benchmark".
What really matters in the benchmark is how your random figures compare to the sequential figures. A huge drop in random benchmark numbers imply some significant disk access time (latency), which hurt such disk when used in RAID mode.

Preliminary optimizations
Part1: http://forum.flexraid.com/index.php/topic,2560.0.html
Part 2: http://forum.flexraid.com/index.php/topic,2585.0.html
Part 3: Coming soon!

Also: Enabling NCQ

One of the most important optimization not discussed is disk defragmentation. Disk fragmentation will affect your disk performance as even sequential data access will translate into random disk access of the fragmented chunks.
So, do defragment your disks!

Reporting performance issues
If you are experiencing performance issues, post in this thread.

Things you will need to provide - each for TCQ and OS Caching (total 8 ):
  • RAW Disk Benchmarks: (array must be offline)
    • The tRAID benchmark of the source disk of the tRAID disk that will be used for the subsequent file copy test below
    • The tRAID benchmark of the source disk(s) of the parity disk(s)
  • File Copy Tests: (array must be online)
    • The result of copying a file that is at least twice the size of your RAM to the tRAID disk
      (note: do not copy to the pool! - just add a drive letter to the tRAID disk and access it directly)
    • The result of copying a file that is 10% the size of your RAM but no more than 1GB to the tRAID disk

For Windows 8 users, a screenshot of the file copy window showing the transfer chart will do (take the screenshot when the copy is 95% complete).

Other users should try to make a screen capture of the transfer.
If taking a screen capture is not an option, then take screenshots at different marks (~0%, 20%, 40%, 60%, 80%, ~100%).

Notes:
Quote
The tRAID benchmarks are extreme in that the algorithm forces the disk heads to move at their greatest amplitude.
For anyone wondering, the disk benchmarks are NOT affected by the RAID settings. So, no need to re-run them under different settings.
The disk benchmarks also do not involve the RAID system and are completely independent.

Only the file copy tests involve the RAID system and its settings.
« Last Edit: November 25, 2013, 07:42:05 am by Brahim »

Offline ninja6o4

  • Full Member
  • ***
  • Posts: 107
  • Karma: +0/-0
    • View Profile
    • My tRAID config
Re: Bad performance? Post here!
« Reply #1 on: September 30, 2013, 09:55:08 am »
Settings:


File copies prior to enabling OS Caching:



PPU Benchmark (DRU was same/similar):


Files copies after enabling OS Caching:

My tRAID config: http://i.imgur.com/6UL9GF4.png
Case: Supermicro SC846 24 Bay 4U
MB: Asus Q87E/CSM
CPU: Intel i7-4771
RAM: 4x 8GB
HBA: Dell H310, IT fw
HDD: 8x+2x 3TB WD Reds (tRAID), 2x Samsung 256 Pro SSD (Boot), 2x 1TB WD Blacks, 1x 640GB WD Black
PSU: Corsair HX750w
OS: Server 2008 R2 w/ HyperV

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,541
  • Karma: +204/-16
    • View Profile
Re: Bad performance? Post here!
« Reply #2 on: September 30, 2013, 10:51:10 am »
A few things:
1. Fix your RAID option screen
2. Attach your images rather than link them out
3. Now that I think about it and for users not on Windows 8 and who cannot do screen capture, screenshots at different marks would be better (~0%, 20%, 40%, 60%, 80%, ~100%)
« Last Edit: September 30, 2013, 12:35:01 pm by Brahim »

Offline johnzered

  • Jr. Member
  • **
  • Posts: 94
  • Karma: +0/-0
    • View Profile
Re: Bad performance? Post here!
« Reply #3 on: September 30, 2013, 02:08:53 pm »
The first image, local.png shows the copy process when done locally on the same machine where tRAID is installed (from a drive outside the tRAID array). The first row shows the 1GB copy and the second a 11GB copy.

The second, network.png shows the same copy process but this time over a wired network from a Windows 8 PC.

The third, shows the benchmark of my DRU1 and PPU1. And the last image is my tRAID config.

When copying the 11GB file locally with Windows caching enabled the speed fluctuated quite badly.
The copy process got even worse when I copied over network were the speed after 3-4gb was copied dropped dramatically.
I did try to copy it again a bit later and I experienced the same results.

Additional info of my setup can be found here: http://forum.flexraid.com/index.php/topic,2740.msg18748.html#msg18748
« Last Edit: September 30, 2013, 02:18:54 pm by johnzered »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,541
  • Karma: +204/-16
    • View Profile
Re: Bad performance? Post here!
« Reply #4 on: September 30, 2013, 02:25:12 pm »
@johnzered
Yikes! on the large file transfers and OS Caching on.  :o
Did you run the test several times to ensure this is not an odd thing?
Try rebooting and re-running the large file transfers.

Also, please describe your system specs.
« Last Edit: September 30, 2013, 02:30:49 pm by Brahim »

Offline johnzered

  • Jr. Member
  • **
  • Posts: 94
  • Karma: +0/-0
    • View Profile
Re: Bad performance? Post here!
« Reply #5 on: September 30, 2013, 02:44:24 pm »
My setup is specified in the provided link above.

I did the copy over network twice with the same result. I will do new tests tomorrow.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,541
  • Karma: +204/-16
    • View Profile
Re: Bad performance? Post here!
« Reply #6 on: September 30, 2013, 03:23:41 pm »
My setup is specified in the provided link above.

I did the copy over network twice with the same result. I will do new tests tomorrow.
Ok.
A few things to keep in mind:
1. Per your local 1GB file copy with OS Caching on, I would say the disk being copied form is the limit.
As per testing on other systems, you should normally see numbers close to the source disk speed until the cache is exhausted. So, if the source disk can do 300MB/s, you should see in the mid to high 200s MB/s. Do bear in mind where the file lies on the source disk if it is a spinning disk as that affects the read speed.

2. Your large file tests are just odd. The hiccups on the 1GB file transfer and drop on the large file are not making much sense.
Your speeds with OS Caching off are normal both on steadiness and throughput. So, I would expect much better performance with OS Caching on.
Please make sure nothing else is accessing either the source disk or target disk during the tests.

Offline johnzered

  • Jr. Member
  • **
  • Posts: 94
  • Karma: +0/-0
    • View Profile
Re: Bad performance? Post here!
« Reply #7 on: October 01, 2013, 01:33:39 pm »
1. Yeah the source drive could be limiting the results when copying locally, it's a Seagate 7200.11 1TB drive which is quite full so the source file could be in the slower part of the disk. Unfortunatelly I cannot easily add a SSD to it either. On the Windows 8 PC (network copy) though the source disk is a SSD so it should, based on what you say perform better than it currently does. I did a new test with the 1GB file over network and again the process was quite stable at 100 MB/s. I also did a new copy test from a Windows 8 PC to a Windows 7 PC over network, they both have 8 GB of RAM and both have SSD's; with the 1GB file the process was very stable at 111 MB/s and with the 11GB it was also very stable at 111 MB/s until 8-9 GB was copied then the speed dropped to (and stayed at) around 85 MB/s until it was finished.

2. I tried the large file copy both locally and over network as well and the results are the same as yesterday. So from my current testings for me it's better to just turn off Windows caching and use TCQ instead to get a more consistent transfer speeds or in some cases even better speeds.

Any new ideas?

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,541
  • Karma: +204/-16
    • View Profile
Re: Bad performance? Post here!
« Reply #8 on: October 01, 2013, 05:08:40 pm »
Well, 85MB/s on the large file transfer is much better than the 11MB/s the other system ended up with.

So, one system does perform as expected and the other doesn't.
Yeah, you might be better off using just TCQ on that other system if OS Caching does not play well with it.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,541
  • Karma: +204/-16
    • View Profile
Re: Bad performance? Post here!
« Reply #9 on: October 01, 2013, 08:28:43 pm »
@facke02
Your DB is not compatible with that release.
You will need to delete your DB and its log file and start anew.
DB compatibility is not guaranteed between RC releases in order to speed up development.

Offline gasutora

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: Bad performance? Post here!
« Reply #10 on: October 01, 2013, 11:11:22 pm »
I was about to do the tests after all arrived but there is something wrong going on, i ran the tests more than 1 time after seeing the values.



I have the same drives as John and he has much higher values, i added my ssd benchmark just to make sure it wasn't something in the ports or so.

My setup is as follows : Core i7 920, motherboard rampage III extreme, 6gb ram, SSD for os drive and the rest are WD 3TB reds, Operating system is Windows 2012 R2 (could this be the issue?) I formated the drives as ReFS but the parity is unformated anyway and the numbers were the same so guess that's not it, i'm using the ich10r ports on all drives including the ssd.

Ok i found a thread by johnzered and he had the same sort of performance in 2012 R2 with the same hard drives but with the disks set online he had better performance, did you run the tests with the drives online johnzered? Thanks (thread here http://forum.flexraid.com/index.php/topic,2740)

Any suggestions? Thanks
« Last Edit: October 01, 2013, 11:32:32 pm by gasutora »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,541
  • Karma: +204/-16
    • View Profile
Re: Bad performance? Post here!
« Reply #11 on: October 01, 2013, 11:49:46 pm »
@gasutora
Indeed, johnzered's results are only higher if he runs the benchmarks with the disks online.
Running the benchmark with the disks online is not valid as it is affected by the Windows' cache.

Please run all the tests as outlined as there is little sense to partial tests.

Offline johnzered

  • Jr. Member
  • **
  • Posts: 94
  • Karma: +0/-0
    • View Profile
Re: Bad performance? Post here!
« Reply #12 on: October 02, 2013, 12:08:07 am »
Well, 85MB/s on the large file transfer is much better than the 11MB/s the other system ended up with.

So, one system does perform as expected and the other doesn't.
Yeah, you might be better off using just TCQ on that other system if OS Caching does not play well with it.

Just to clear things up the 85MB/s transfer I got between a Windows 8 PC and a Windows 7 PC is just a "regular" file copy, tRAID is not involved as it's not installed on either of these machines. As the server where I have tRAID installed is a VM could that have something to do with it? Have you tried OS Caching inside a VM?

@gasutora
Indeed in this thread the provided benchmark is with the disks online and with a volume on it. In the other thread you mentioned, the first post is with the drive offline while the fourth post (eg. the one with "better" values) is with the disk online and a volume on it. But as Brahim explained it could be Windows cache kicking in if the disk is online. Will be interested to see your copy results.

Offline gasutora

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: Bad performance? Post here!
« Reply #13 on: October 02, 2013, 12:12:49 am »
Thanks yeah we can compare it then john, will leave the pool calculating parity etc and when it's done i'll run the copy tests etc.

Offline Skirge01

  • Full Member
  • ***
  • Posts: 141
  • Karma: +2/-0
    • View Profile
Re: Bad performance? Post here!
« Reply #14 on: October 03, 2013, 03:08:14 pm »
When you say that we should defrag our drives, you mean that we should defrag each transparent disk individually, not the storage pool itself, correct?