Author Topic: Transparent RAID Performance Thread - Part 2/4 (TCQ & OS Caching)  (Read 15169 times)

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,402
  • Karma: +201/-15
    • View Profile
Tagged Command Queuing (TCQ) vs OS Caching
TCQ and OS Caching are mutually exclusive. This is because TCQ is really a subset of what your OS does when caching.
[Edit] After further testing, it appears that the combo of TCQ + OS Caching achieves even greater performance!

TCQ is all about data prioritization while OS Caching does both data prioritization and write-through caching.
There is one limitation to OS Caching, which that an array cannot be stopped after being published and one has to reboot to stop the array.
Furthermore, Transparent RAID will have its own caching layer in the near term. This caching layer will be mutually exclusive to OS Caching but will support TCQ.
Last, some applications might choose to explicitly bypass the OS's cache forgoing the write optimization aspect too. You might find that such applications perform better under TCQ than with OS Caching on. Ultimately, you will need to test to see what works best for you.


Tagged Command Queuing (TCQ)
Quote
In tRAID, this option only applies to write operations and has the following traits:
  • It greatly improves on small size write command performance and reduces write latency.
  • It allows for the processing of multiple write requests at once and with greater effect on NCQ enabled systems.
  • It further improves on the performance of applications that buffer and write in burst mode.
  • Finally, TCQ reduces the negative side effect that S.W.O has on random writes but might slightly (up to 5%) reduce sequential write performance in turn.

This should only be enabled on systems that have been tested stable and fitted with an uninterruptible power supply (UPS) to avoid minor data loss on improper system shutdown.
We say minor data loss because what's at risk is the data in the write command queue buffer.
The default value is 32 and higher values use more memory.

TCQ is all about reducing write latency and smart write prioritization.
It does nothing to sequential write performance. In fact, it might slightly reduce sequential write performance by up to 5%.

Nonetheless, TCQ greatly boosts the performance of applications such as databases and those applications that write in small packets or in burst mode.
This is similar to how an SSD might provide greater system experience than a spinning disk with higher sequential transfer rates simply because of its greater performance in small write packet sizes.

Because the write commands are queued for optimization and prioritization, there is a chance of data loss if that queue is not committed before the system crashes or suffers a power loss.
As such, you should ensure to enable this option only if your system is stable.
Further, if you don't trust your power source, you should also plan to have the system on a UPS.

[Edit]
After further testing, it looks like TCQ might have a positive effect on sequential transfers.
Yes, there are a few cases where sequential transfers might take up to a 5% hit, but for most cases, sequential transfers seem to be gaining.
I will wait for user reports for a more conclusive summary effect of TCQ on sequential transfers.



OS Caching
OS Caching is available in RC10 and later releases.
As previously stated, OS caching is a grander feature than TCQ. It does the same data prioritization that TCQ does as well as write-through caching to boost the system's performance.
The reason OS Caching is being introduced after TCQ is because we really wanted users to focus on their raw disk speeds first.

OS Caching will skew your reported disk speeds. For data transfer sizes less than your available system memory, you will often see speeds greater than 100MB/s.
The reality is that your OS is simply buffering the transfer and completing it in the background. This is a great feature from a user experience perspective.
Nonetheless, your raw disk speeds are important as your disks should be able properly keep up with the cache. Else, if you were to transfer data twice the size of your system memory for instance, you will find that the throughput will dive at some point during the transfer and finish at a rather lower throughput than when the transfer started. The effect will largely depend on your raw disk speeds.
So, before enabling OS Caching, you should first enable TCQ and test that you are getting adequate performance. Once your raw speeds are acceptable and that you are happy with your overall configuration, then you can enable OS Caching.

The risk in turning on either TCQ or OS Caching
You should enable TCQ or OS Caching only if you trust the stability of your hardware.
TCQ and Windows Caching suffer from the same risk in that the queued data might not make it to the disk in the event of an abrupt power failure.
Ever had a "Delayed write failure" in Windows? That's ultimately the risk you run, and we have learned to accept that risk all these years running Windows. The fact that tRAID defaults to TCQ and OS Caching off is a unique feature for those that are strict on data integrity. So, turning either TCQ or OS Caching on is NOT an additional risk. Rather, it is a risk you've always had. Turning them off is a feature unique to tRAID for stricter data integrity.
« Last Edit: September 30, 2014, 06:06:20 am by Brahim »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,402
  • Karma: +201/-15
    • View Profile
Re: Transparent RAID Performance Thread - Part 2
« Reply #1 on: August 23, 2013, 01:44:55 pm »
Reserved.

Offline vletroye

  • Hero Member
  • *****
  • Posts: 714
  • Karma: +7/-0
    • View Profile
Re: Transparent RAID Performance Thread - Part 2/3 (updated with OS Caching)
« Reply #2 on: October 23, 2013, 04:20:05 pm »
Same question as the one I had regarding Direct I/O vs Buffered I/O and S.W.O:

Does TCQ and OS caching also improve the speed of the "parity creation" process ?
Or only the write operations of the data on the DRU ?

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,402
  • Karma: +201/-15
    • View Profile
Re: Transparent RAID Performance Thread - Part 2/3 (updated with OS Caching)
« Reply #3 on: October 23, 2013, 04:48:03 pm »
Those settings apply on to the deployed array and do not impact the tasks (Create, Verify, etc.).

Offline blaze2k

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Re: Transparent RAID Performance Thread - Part 2/3 (updated with OS Caching)
« Reply #4 on: November 13, 2013, 04:03:07 pm »
When I enable OS Caching my computer crashes and does all kinds of weird stuff. If i try to play a game from the array usually it freezes.
HTPC: 1xSSD + 3x1TB + 2x500GB in Thermaltake MAX-1542  / i5-3450S / Asus P8H77-I / 2x4GB Corsair 1600 / Radeon 7850 2GB / Cooler Master Elite 361 + Seasonic M12II 520

Offline vletroye

  • Hero Member
  • *****
  • Posts: 714
  • Karma: +7/-0
    • View Profile
Re: Transparent RAID Performance Thread - Part 2/3 (updated with OS Caching)
« Reply #5 on: November 13, 2013, 05:37:07 pm »
As it's using a lot your RAM, I would suspect that this one as an issue...

Test it with MemTest86+
You can easily create a bootable usb key with MemTest86+ using this installer

Did you by any chance o.c. you RAM/voltage ?

Offline blaze2k

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Re: Transparent RAID Performance Thread - Part 2/3 (updated with OS Caching)
« Reply #6 on: November 14, 2013, 12:47:25 am »
Haven't checked the ram but I have 8GB and I'm not OC anything, all stock. Resets, freezes amd all kinds of crap
HTPC: 1xSSD + 3x1TB + 2x500GB in Thermaltake MAX-1542  / i5-3450S / Asus P8H77-I / 2x4GB Corsair 1600 / Radeon 7850 2GB / Cooler Master Elite 361 + Seasonic M12II 520

Offline vletroye

  • Hero Member
  • *****
  • Posts: 714
  • Karma: +7/-0
    • View Profile
Re: Transparent RAID Performance Thread - Part 2/3 (updated with OS Caching)
« Reply #7 on: November 18, 2013, 03:18:49 pm »
I got 8 new GB (4x2GB) DDR2 from G.Skill one week ago and my Server with tRAID started to crash...
I did run memtest86+ (for a whole night. It did 4 complete passes) and 1 module appeared to be "corrupted".

I did send the "corrupted" module back to G.Skill on last Monday and got today a new one. I will go to test it in a few minutes  :P

[EDIT] My Server runs now fine :p
« Last Edit: November 18, 2013, 03:41:07 pm by vletroye »

Offline SgtSiff

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: Transparent RAID Performance Thread - Part 2/3 (updated with OS Caching)
« Reply #8 on: December 09, 2013, 04:46:17 am »
So does OS Caching use RAM and not the SSD?

Will this mean that it will be useless for me as the great majority of my data is Blu-ray rips @ 25GB each.

I guess it will help with many small files?

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,402
  • Karma: +201/-15
    • View Profile
Re: Transparent RAID Performance Thread - Part 2/3 (updated with OS Caching)
« Reply #9 on: December 09, 2013, 08:23:01 am »
@SgtSiff
If you are going to be dealing with mostly large files, then yes, stick to TCQ as OS Caching will only give you great burst speed.

As stated before, tRAID will have a new massive caching layer that will boost performance even further. No detail yet. Lips sealed until it is ready.
Those that need to migrate large amount of data to tRAID would do best moving that data to the source disks before creating the RAID.
Transparent RAID support disks with data already on them.


Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,402
  • Karma: +201/-15
    • View Profile
Re: Transparent RAID Performance Thread - Part 2/4 (TCQ & OS Caching)
« Reply #10 on: September 30, 2014, 06:08:45 am »
Major Update!
Further testing is revealing that TCQ + OS Caching actually yields even better performance. So, the next release of Transparent RAID will enable the use of both features. :)

Offline adridolf

  • Jr. Member
  • **
  • Posts: 91
  • Karma: +0/-0
    • View Profile
Re: Transparent RAID Performance Thread - Part 2/4 (TCQ & OS Caching)
« Reply #11 on: February 07, 2017, 04:12:58 am »
For TCQ the default depth in the textbox is 8, but the tooltip says it's 32.

Which one would actually be recommended as default without testing?

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,402
  • Karma: +201/-15
    • View Profile
Re: Transparent RAID Performance Thread - Part 2/4 (TCQ & OS Caching)
« Reply #12 on: February 07, 2017, 11:27:00 am »
The default TCQ value is indeed 32. It is set to 8 when doing an Express Configuration. I think the idea was to go with milder setting when doing Express Config.
Thinking about it more, I am changing the Express Configuration to set it to 32 to be consistent.

Thanks.