Author Topic: Users with out of memory error, please report here.  (Read 10022 times)

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Users with out of memory error, please report here.
« on: January 17, 2014, 09:37:36 am »
If you are suffering from frequent out of memory errors, please report here so we can get to the bottom of this.
It is clear that only a handful of users are affected, but it seems pretty consistent for those affected.

What would be important is me getting access to your box to witness the issue first hand.
The issue is really external to RAID-F and resides in my compiling too. I need to gather enough information to get my software vendor to address this once and for all.

Thanks for your patience on this issue and effort in helping to resolve this.

So, please post in this thread and describe your scenario.

Offline Tsunamijhoe

  • Jr. Member
  • **
  • Posts: 84
  • Karma: +0/-2
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #1 on: January 17, 2014, 03:06:21 pm »
If you need anymore info from me Brahim, let me know. If the screenshots taken over 9 hours wasn't enough or if you would like remote access please advise on how to facilitate that for you..

I am glad i seemingly are not the only one having this problem,as it is a severe and cripling problem for the users who have it..

Thank you for adressing it

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #2 on: January 17, 2014, 04:48:32 pm »
What will be useful will be for users to recap the followings:

- OS?
- 32 bit or 64 bit?
- System memory
- Free system memory (system memory minus other applications)
- Rate of memory leak

I will ping some of you for remote access. Thanks.

Offline Tsunamijhoe

  • Jr. Member
  • **
  • Posts: 84
  • Karma: +0/-2
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #3 on: January 17, 2014, 07:01:53 pm »
- OS= Windows 7
- 32 bit or 64 bit?= 64bit
- System memory= 8GB
- Free system memory (system memory minus other applications)= Newly booted up system has approximatly 6GB free memory give or take, then flexraid slowly eats away until system runs out of memory
- Rate of memory leak= On my system, Flexraid will over a period of 8-9 hours increase memory usage from 150.000kb to over 4GB,and then it will crash with an outofmemory error (as screenshots showed)

If you need remote access,you will have to guide me through how to allow this traffic into my network,and how to setup a remote session on the box in question,and how to limit your access to only the box in question (i have 3 pc's as well as phones and other hardware on the network)

Offline karimkanji

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #4 on: February 08, 2014, 03:43:27 am »
Been using Flexraid (v2.0 update 12) for three months without major incident (had some issues with file permissions, high cpu utilisation and updates sometimes taking 12 hours despite very few new files but nothing i couldn't live with)

However, since upgrading to update 12b a week ago, I've had memory leaks which mean i need to restart flexraid evey 48 hours or so.

My details:

- OS? Windows 8.0
- 32 bit or 64 bit? 64 bit
- System memory? 10GB
- Free system memory (system memory minus other applications) Generally 6-7GB
- Rate of memory leak? When initially started, Flexraid uses <100MB,  after around 45 hours, Flexraid is using 3.3GB so approx 70MB/hr

Please let me know if you require further details. My plan is to roll back to update 12a. I'll report back if this fixes the problem.

Thanks
Karim

Offline simonsaysflex

  • Newbie
  • *
  • Posts: 9
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #5 on: February 08, 2014, 09:04:09 am »
This might not be why others have memory crashes but I guess it may help someone.

OK here's my input on one of the reasons for huge memory spikes. I've seen as high as 3.5GB spikes. FlexRAID memory usage goes from 100MB to 2.5 GB+ in an instant and crashes soon after. It happens when starting Update, Quick-Validate, Validate or Verify. Pretty much any operation that's dealing with PPU.

Not long ago I did a Verify for the 1st time after parity creation. I always used Validate until this point.
Validate never found any problems whatsoever. Verify on the other hand returned a bit-mismatch right away and quit. I checked the supposedly problematic files, they were in perfect shape (MD5 match, yes I also have MD5 hash for all my big static media files).

I then tried force-sync-verifying the affected byte range which completed successfully but upon another Verify there was yet another bit-mismatch. So I said screw it and force-snyc-verified the whole parity.

It completed successfully. Or so I thought?

After that my problems with FlexRAID crashing out of memory began. It happened EVERY time the second I clicked manual Update or any other operation. I tried all sorts of things. Different FlexRAID versions. Different configs. Nothing helped. I even reimaged my sytem partition to earlier time (yes I have those too for every week of the past year) and even that did not solve the crashing.

So at this point its either hardware problem, which it was not, or something fishy with parity. It seems that force-sync-verify completely borked it. As a last ditch attempt I deleted the whole parity and re-created it.

Problems gone.

I love FlexRAID but I'm left with a bad taste in my mouth and a few questions:

1. reason for Verify finding bit-mismatch? Files were OK, what wasn't?
2. reason why Validate never found any problems?
3. reason why force-sync-verify completely borked my parity? Or at last borked it enough so that FlexRAID completely lost it.

I need to know this things so that it doesn't happen again.
 

Offline RyC

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #6 on: February 08, 2014, 02:12:53 pm »
Updated to 12b as well and I have a memory issue.

- OS: Windows 7 Ultimate
- 32 bit or 64 bit: 64 bit
- System memory: 4 gb (Also tried with 8 gb)
- Free system memory (system memory minus other applications): 200-500 mb with 4gb, 1gb with 8gb
- Rate of memory leak: Ramps up to 80-90% total memory utilization upon startup. I cannot see what's using the memory in Task Manager/Resource Monitor. They all say FlexRAID is using ~200 megabytes, and that's the highest using process, yet all of the RAM is shown to be in use on the usage bar. HOWEVER, if I do not enable Pool Auto-Start and manually start the Array, the memory usage does not shoot up! I may roll back to 12a as well since it seems this is a 12b problem. Please update on whether you're able to track this down!

Offline karimkanji

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #7 on: February 09, 2014, 01:10:43 pm »
Rolling back to 12a has fixed the leak for me. Flexraid's been running all day with a fairly constant 40mb memory reservation.

Also, got to the bottom of the high CPU utilisation: Flexraid uses a steady 40% of my CPU (Dual Core 2.2GHz AMD Turion II in an HP N54L) when Tonido Personal Cloud server is also running which is a shame because it's a useful piece of software  :( Now I wish I could find a way to get Transmission to write to the Flexraid array without getting file access denied errors  :(

Offline RyC

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #8 on: February 09, 2014, 02:30:50 pm »
Updates with 12b have taken significantly longer than with 12a, and I always wake up to 90% memory utilization. I've also rolled back to 12a, I'll see if the issues come up.

Offline habe

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #9 on: February 10, 2014, 07:08:47 pm »
- OS? - Win & Home Pro; fresh install
- 32 bit or 64 bit? - 64bit
- System memory - 8GB
- Free system memory (system memory minus other applications) - Hardware Reserve - 337
                                                                                                                        In use - 1392
                                                                                                                      Modified - 60
                                                                                                                      Standby - 1370
                                                                                                                           Free - 5029

- Rate of memory leak:
I first noticed my issue when my pool did not complete its first validate. I've restarted the system and re-run Validate multiple times and always the same. Validate causes all free memory to be displaced by mainly Standby memory and a lower amount of Modified memory. As validate approaches 18% completion point, the Modified memory starts creeping up. Around 38% - 40% Validate completion, Modified takes up all available memory other than Hardware Reserve & in-use (which always remains about the same) and I get the Low Memory Error.

To try and pinpoint the memory leak, I've shutdown all running programs until I've run Validate with no processes running other than Team Viewer which is how I get into my server. I've shutdown AVAST (modified to exclude FlexRaid and the pool drive), XFAST LAN & XFAST USB so, don't think it's Team Viewer causing the leak as it is always running and I never have any other problems.

habe

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #10 on: February 10, 2014, 08:59:46 pm »
@habe
The only thing that will be useful is a screenshot or screencast of the FlexRAID process and its memory usage.

Offline RyC

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #11 on: February 11, 2014, 02:36:13 am »
It seems this may be an issue with 12a as well, I'm not really sure...This is run on 12a.

Randomly, Physical Memory utilization will climb rapidly to ~90%. It seems the used memory is located in "Kernel Memory - Paged". No process, including the FlexRAID process is taking up nearly enough RAM. However, doing any IO on the FlexRAID pool will drop Physical Memory utilization to "accounted" levels. I've included some pictures...I have no idea what's going on. Should I be concerned about memory utilization shooting up randomly? I can't find whatever is doing it, but again, IO on the FlexRAID pool causes the memory to be released.

High Physical Memory Utilization (seems to be in Kernel Memory - Paged)


View All Processes showing no process is using nearly enough memory to account for 3.5 GB


After dragging a random file from the FlexRAID pool to the Desktop, Physical Memory utilization has dropped dramatically


No change in process memory utilization


Once again, is this something to worry about?

Thanks for your help.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #12 on: February 11, 2014, 07:49:31 am »
1. As you can see it is not the FlexRAID.exe process using that memory.

2. You could have another component leaking memory. Usually, your drivers might be leaking memory from FlexRAID activities.

3. Open resource monitor and take a screenshot so we have more details on memory utilization (private, commit, etc).

Offline habe

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #13 on: February 11, 2014, 08:56:47 am »
@habe
The only thing that will be useful is a screenshot or screencast of the FlexRAID process and its memory usage.

OK, I will run a validate tonight and post some screen shots of resource manager as it progresses.

habe

EDIT on 2/12:

I have friend who has identical, part-for-part, server build as mine and he was able to perform validate last night with no issues. I plan on scrubbing my OS drive and doing a complete system re-install sometime this week at which time I will report back.

have
« Last Edit: February 12, 2014, 10:09:52 am by habe »

Offline RyC

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #14 on: February 11, 2014, 11:49:18 am »
1. As you can see it is not the FlexRAID.exe process using that memory.

2. You could have another component leaking memory. Usually, your drivers might be leaking memory from FlexRAID activities.

3. Open resource monitor and take a screenshot so we have more details on memory utilization (private, commit, etc).

Yeah, I know it's not FlexRAID but nothing will show me what's using all the memory other than "Kernel Paged". But it seems that only FlexRAID IO will cause the memory to free up again. I've attached screenshots of Resource Monitor with high memory utilization, still nothing adds up to 3.5 gb.

These are large sized images so i'll link them:

Commit: http://puu.sh/6SeBi.png

Working Set: http://puu.sh/6SeCj.png

Shareable: http://puu.sh/6SeCp.png

Private: http://puu.sh/6SeCx.png

Thanks for your help, I really want this solved.