Author Topic: All of a sudden: very slow copy speeds, slow verify sync  (Read 1250 times)

Offline Skirge01

  • Full Member
  • ***
  • Posts: 221
  • Karma: +5/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #15 on: February 12, 2019, 01:03:47 pm »
It's still verifying the same DRUs to the same PPUs, so a new config shouldn't be any faster.  Either the parity needs to be updated or it doesn't.  Now, if you start tweaking things according to the performance guides, THAT could improve performance.   :)

Offline pooler1

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #16 on: February 13, 2019, 02:52:34 pm »
It's still verifying the same DRUs to the same PPUs, so a new config shouldn't be any faster.  Either the parity needs to be updated or it doesn't.  Now, if you start tweaking things according to the performance guides, THAT could improve performance.   :)
ok thanks, yes that's what i figured.  5 days, about halfway through.  sheesh.

Offline pooler1

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #17 on: February 18, 2019, 06:19:37 pm »
my pool is back up, and i am doing the same kind of file operations that got me in the situation to see what is the issue.

I can duplicate the scenario...
I am using a file explorer on a client computer (traid is on the server) to MOVE a bunch of files from one folder to another.  I always see that traid struggles a little more with move operations vs copy.  So as it is going, the transfer speed slows to a crawl like 1 MBps and the lock override count of the array starts going up.  It went up to 11 before i cancelled the operation.  I am not sure what this means.  SKirge?

Offline Skirge01

  • Full Member
  • ***
  • Posts: 221
  • Karma: +5/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #18 on: February 19, 2019, 11:42:18 pm »
Are you still using the landing disk?  Have you tried completely removing it from the equation?  I'd start there.  Please don't skip that step because everything else we try could be wasted effort if the landing disk is the cause and we need to focus on that instead.  Once you're certain the landing disk is definitely not the issue...

Are you getting sync issues during the verify jobs?  If so, the issue may be related to the CQ Depth and/or the Salt settings.  (I'm leaning towards Salt.)  Have you tweaked either of those from default?  I believe they're normally 32 and 8, respectively, but I could be wrong as mine aren't at default.  I'm not sure which direction you'll need to head with CQ Depth, but I suspect you may need to increase the Salt.  Just an FYI that I'm "divining" these suggestions solely from the pop-up info boxes in the UI, not from some intimate knowledge of how tRAID works.  ;D

I'd also suggest you go through the 4 performance threads, in order, and follow Brahim's tips.  Especially make sure you're benchmarking your results after every change you make.  I actually created a spreadsheet with all the settings I was tweaking so I could keep track of the results.  It's a lot easier when you can clearly see the change (or lack thereof).

Offline pooler1

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #19 on: February 20, 2019, 12:44:12 pm »
Are you still using the landing disk?  Have you tried completely removing it from the equation?  I'd start there.  Please don't skip that step because everything else we try could be wasted effort if the landing disk is the cause and we need to focus on that instead.  Once you're certain the landing disk is definitely not the issue...

Are you getting sync issues during the verify jobs?  If so, the issue may be related to the CQ Depth and/or the Salt settings.  (I'm leaning towards Salt.)  Have you tweaked either of those from default?  I believe they're normally 32 and 8, respectively, but I could be wrong as mine aren't at default.  I'm not sure which direction you'll need to head with CQ Depth, but I suspect you may need to increase the Salt.  Just an FYI that I'm "divining" these suggestions solely from the pop-up info boxes in the UI, not from some intimate knowledge of how tRAID works.  ;D

I'd also suggest you go through the 4 performance threads, in order, and follow Brahim's tips.  Especially make sure you're benchmarking your results after every change you make.  I actually created a spreadsheet with all the settings I was tweaking so I could keep track of the results.  It's a lot easier when you can clearly see the change (or lack thereof).
ah yes!  ok forgot.  I will use the bypass landing disk option and report back.

Offline pooler1

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #20 on: February 20, 2019, 02:05:21 pm »
OK I removed the landing disk.  I do like this better.  I don't get that burst of speed in the beginning (sometimes it would exceed 200 MBps).  But, there are no more of those slowing to a crawl moments, and the speeds are stable.  I think this is the solution.

Offline pooler1

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #21 on: February 20, 2019, 08:01:37 pm »
skirge...helll yea, no landing disk for me.  THis is way better.  Sure, no super fast burst in the beginning and things are generally slower, but way more stable this way.  I am having no issues with this setup.

Offline Skirge01

  • Full Member
  • ***
  • Posts: 221
  • Karma: +5/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #22 on: February 20, 2019, 09:47:53 pm »
Awesome!  Very glad to hear that!  Keep an eye on your verify jobs, overall responsiveness, and any locks.  If those start appearing, then you may still need to tweak the Salt or the CQ.  Key thing is to change things one step at a time, monitor performance for a while, then tweak again.  It may take days or weeks of running before you're confident there haven't been any adverse effects from the change, so go slow.  (Personally, I give my tweaks 2 weeks.)  Once you've got no locks, verify jobs aren't finding anything, and responsiveness seems good, then you'll want to dig into those performance threads I mentioned.  Once you're done with those and things are smooth, fast, and reliable, you just might want to give the landing disk another shot... perhaps with some tweaks to that, as well.

Offline pooler1

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #23 on: February 26, 2019, 04:15:22 pm »
hmmm
still get lock counts up and slow to a crawl with move operations on that same directory.  weird.  i wonder...its one of the few directories that might exceed the unc limit of 255 characters for the path.  i wonder if thats it.

Offline Skirge01

  • Full Member
  • ***
  • Posts: 221
  • Karma: +5/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #24 on: February 27, 2019, 10:08:36 am »
You know the easy test then... shorten the path.  ;)  If it still happens after that, might be time to investigate changing the salt.

Offline pooler1

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #25 on: February 27, 2019, 02:29:20 pm »
You know the easy test then... shorten the path.  ;)  If it still happens after that, might be time to investigate changing the salt.
I'll do that, it will take a while as i sort through all the files.
Can you give me a quick take on what the salt is and it relates to traid?  i've read the documentation but it might help if i heard it in a different way.  thanks Skirge.

Offline Skirge01

  • Full Member
  • ***
  • Posts: 221
  • Karma: +5/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #26 on: February 27, 2019, 06:18:18 pm »
Like I said, my knowledge is only from what the popups tell me in the GUI.  If you've read that, you know exactly as much as I do.  :)  Maybe there's more stuff in those performance threads I mentioned or the wiki.

Offline pooler1

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #27 on: February 27, 2019, 07:24:19 pm »
Like I said, my knowledge is only from what the popups tell me in the GUI.  If you've read that, you know exactly as much as I do.  :)  Maybe there's more stuff in those performance threads I mentioned or the wiki.
ok thanks.  yes that is all i know as well.

Offline Skirge01

  • Full Member
  • ***
  • Posts: 221
  • Karma: +5/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #28 on: February 28, 2019, 02:25:44 pm »
You could also try searching these forums for the terms you're interested in and see if there's any other explanations from Brahim or others.  Good luck!

Offline pooler1

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: All of a sudden: very slow copy speeds, slow verify sync
« Reply #29 on: February 28, 2019, 10:24:06 pm »
ive read everything over and over, many times over the years lol.
this product is literally unique.  if it were not, i'd be super annoyed right now with the amount of time and effort i have spent trying to figure out what little i know now.  I'm not mad, i appreciate the product very much.  But it's a nightmare to tell what is going on, what the options really mean, etc.  Brahim, bless his heart, his explanations of things border on rude and vague.  But again, that is pretty normal for someone who is knee deep in such technical issues, and his product is good.  and then the disappearing bit as well, lol.

anyway, it is working for me.  I'd love to make it better, but it's like doing goddam phd thesis.  all these things written in the hints of the options, like long sentences and paragraphs, i have to hover over it multiple times lol. 

I've even offered to help (volunteer!) but he doesn't want it.  All we really need is all the options in a pdf explained in better terms, not crazy technical things that explain what its doing but doesn't help us figure out what the setting should actually be.  Most of the options are not discussed anywhere.  There are discussions in the forums if you search for the word, but most of them are not helpful in actually helping the end user understand what that option does.  It's just a discussion with that word or phrase in there. 

Anyway, its been good lately, ever since i took away the landing disk.  I'd like to get it faster only.  IF we can get our pools operating on standard hard drive speeds on avg like 70-200 MBps, that would be amazing.  that's my only goal.  if i cant get it, ill take these 20-70 speeds with stability, as it is right now.