Author Topic: Users with out of memory error, please report here.  (Read 10881 times)

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #15 on: February 11, 2014, 12:39:29 pm »
Any change in Kernel memory is mostly the doing of drivers.
Read this: http://superuser.com/questions/533135/98-100-memory-use-no-process-uses-more-than-25mb

They have steps to identify which component is using the RAM.


Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #16 on: February 11, 2014, 12:47:59 pm »
Update 2014-02-11:
I have spent weeks doing to serious memory testing on RAID-F only to find out that none of my environments exhibit the issue reported by the few affect users.
It is clear that this issue is system specific and only affect certain hardware and software configurations.

What I need now is access to the system of those affected for some analysis.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #17 on: February 11, 2014, 02:46:10 pm »
@RyC
Unfortunately, I am only interested in those systems where the FlexRAID.exe process is using the RAM.
FlexRAID can stress other components and cause them to leak, but that's just not something for me to chase.
All I can do is give you pointers.

Offline RyC

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #18 on: February 11, 2014, 03:33:04 pm »
@RyC
Unfortunately, I am only interested in those systems where the FlexRAID.exe process is using the RAM.
FlexRAID can stress other components and cause them to leak, but that's just not something for me to chase.
All I can do is give you pointers.

No problem, I figured as much. I found the problem, tag MmSt, which seems to be the file cacher (very broadly). Still not sure why it shoots up randomly, I've checked the Event Log to see if anything is happening, but nothing stands out. It seems if something else needs the RAM/page pool space, MmSt will get out of there. Is this OK for normal operation?

Thanks again for your help
« Last Edit: February 11, 2014, 03:34:41 pm by RyC »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #19 on: February 11, 2014, 04:09:13 pm »
@RyC
Check your PM.

Offline habe

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #20 on: February 16, 2014, 11:33:42 am »
Brahim.

I seem to have solved my memory problem. It does not appear that any programs or OS processes were involved and instead, it was apparently caused by my motherboard bios settings and/or chipset behavior.

I was resolved to scrub my OS drive and do a complete re-install after trying everything I could think of to plug the memory issue. My Flexraid system mobo is an AsRock Z87 Extreme 4 and has 8 SATA III slots ( 6 Intel and 2 Asmedia SATA ports). I decided to check some things one last time and noticed that I was getting no HD SMART data under Flexraid "Drive Testing, Info and S.M.A.R.T (Health)". After doing some research, I read on the internet that the drive type has to be set to RAID instead of AHCI in the bios or some applications will not receive SMART data on this mobo. After making the change (I also moved the OS SSD off of an Intel port and onto an Asmedia port because the OS wouldn't load under a RAID bios setting. note: the Asmedia ports use their own settings...), I immediately was able to see SMART data in Flexraid. I started a Validate, watched it go past 50% while watching System Monitor for memory usage and the "Modified Memory" never climbed at all. Validate complete perfectly.

What changing that bios setting so that I could see SMART data in Flexraid has to do with a memory issue is beyond me but, problem solved. Perhaps this is an issue with the mobo, chipset or Intel SATA controller.... IDK

habe
« Last Edit: February 16, 2014, 11:36:38 am by habe »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #21 on: February 16, 2014, 08:04:57 pm »
@habe
Thanks for the report.
Strangely, this is falling in line with all resolutions so far.

Another resolution I got through PM by RyC:
Quote
Brahim,

Thanks for your help. I finally nailed the culprit application (Ceton My Media Center). No memory leaks since I uninstalled.

Also, the FlexRAID process crashed when running the nightly Update with the replacement nbio.dll. I changed it to the original and it's not crashing now.

Update tasks under 12b still took considerably longer for the same amount of data than under 12a though, maybe we can take that back to the forum.

Thanks again for your help!

Offline WesM63

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #22 on: February 19, 2014, 07:23:31 pm »
I am suffering from this issue. I upgraded from u12 to u12b over the weekend and have been experiencing "java.lang.OutOfMemoryError: Thread creation error: 8"

- OS? Windows 8.1 Pro
- 32 bit or 64 bit? 64bit
- System memory 24GB
- Free system memory (system memory minus other applications) 21.6GB
- Rate of memory leak ~10-12hrs

FlexRaid.exe is currently using 443MB and fluctuates a few MB while writing to/from the FlexRaid volume. The only thing I did this weekend was  Install windows updates, update flexraid to u12b, install paid flex license and update Plex Media Center. Nothing else has been installed or removed from the system.
« Last Edit: February 19, 2014, 07:25:21 pm by WesM63 »

Offline scottbilas

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #23 on: February 21, 2014, 08:43:41 pm »
I am suffering from this issue, and it is making me panic, as I worry about the integrity of my data through repeated lockups and restarts of flexraid.exe.

I have a 100% reproducible out-of-memory issue that takes about 7 hours to make flexraid.exe unresponsive. My only solution at this point is to restart the service and start the storage pool again. This was happening on 11, and I recently upgraded to 12b with no improvement (possibly worse, actually).

Attached is a graph of what I'm seeing. It shows virtual memory usage over time for the flexraid.exe process. I had restarted the FlexRaid service roughly ten minutes before the start of the graph. As you can see, memory usage is reasonable for a little while, suddenly spikes to 2 gigs, then continues growing at a fairly steady pace until it flatlines at max usage. This comes from a perfmon log I took of all flexraid.exe process stats recorded every 5 seconds.

Stats you have asked for:
  • OS is Windows 7 Ultimate, 64-bit
  • System memory 8GB
  • Typical free system memory 4GB
Some others reporting this issue appear have had problems with their OS or hardware. That is not the case for me. This system is a stock HP ProLiant N54L. No modded BIOS, no overclocking, no janky hardware. FlexRaid was running quite well until February 8 of this year, when I saw my first out-of-memory error in the logs.

I also noticed in my logs that there are many, many other exceptions that have been recorded, including some strange characters in error messages around February 8 when this problem first started occurring. Related?

Brahim: I will PM you a link to a file containing the FlexRaid logs and a log of the flexraid.exe process stats.

Offline scottbilas

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #24 on: February 22, 2014, 12:11:57 pm »
I've done some experimenting and have some more information to help narrow down a repro.

So I suspected the problem may have been triggered by my backup software (CrashPlan) which is overwhelming FlexRaid with requests. Each restart of FlexRaid would cause CrashPlan to rescan the drive, which I believe involves both full directory searches and hashing of file data. I disabled CrashPlan and FlexRaid's memory usage stays flat.

This led me to do a new test, still with CrashPlan disabled and no other disk activity going on.

  * Start up several dir /s's of my Raid drive running in parallel
  * Route one of the dir /s's through md5sum so that I'm hashing every file on the Raid in series

This got it to out-of-memory in 6 hours. It also had the same profile as the CrashPlan scenario, where it quickly spikes up to 2GB memory usage, then stays on a steady climb until it runs out.

Possibly relevant stats on my Raid drive:

  * DRU1 = 1.82TB, DRU2 = 931GB, PPU1 = 1.82TB set up in Cruise Control [Snapshot RAID] Tx engine on NTFS
  * Nearly all unchanging data, typically adding 1-2% new files per week (rarely deleting)
  * ~400K files in ~30K folders, about 1.75TB total
  * Some of the dir trees are deep - 10% of the files have >= 120 character paths

Attached a new graph of vmem usage.

Offline Lediur

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #25 on: February 22, 2014, 04:53:36 pm »
I've done some experimenting and have some more information to help narrow down a repro.

So I suspected the problem may have been triggered by my backup software (CrashPlan) which is overwhelming FlexRaid with requests. Each restart of FlexRaid would cause CrashPlan to rescan the drive, which I believe involves both full directory searches and hashing of file data. I disabled CrashPlan and FlexRaid's memory usage stays flat.

This led me to do a new test, still with CrashPlan disabled and no other disk activity going on.

  * Start up several dir /s's of my Raid drive running in parallel
  * Route one of the dir /s's through md5sum so that I'm hashing every file on the Raid in series

This got it to out-of-memory in 6 hours. It also had the same profile as the CrashPlan scenario, where it quickly spikes up to 2GB memory usage, then stays on a steady climb until it runs out.

Possibly relevant stats on my Raid drive:

  * DRU1 = 1.82TB, DRU2 = 931GB, PPU1 = 1.82TB set up in Cruise Control [Snapshot RAID] Tx engine on NTFS
  * Nearly all unchanging data, typically adding 1-2% new files per week (rarely deleting)
  * ~400K files in ~30K folders, about 1.75TB total
  * Some of the dir trees are deep - 10% of the files have >= 120 character paths

Attached a new graph of vmem usage.

I'm also encountering a similar out of memory error, and I'm using CrashPlan as well.

I have four DRUs with 2 TB each, and one PPU with 3 TB with Tx Snapshot RAID in Cruise Control.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #26 on: February 25, 2014, 10:18:18 pm »
Okay guys, here is a patch: http://download.openegg.org/release/FlexRAID/Windows/2.0/w_class1_0.dl

1. Download
2. Stop the FlexRAID service
3. Backup the following file: <install directory>\ext\w_class1_0.dl
4. Copy downloaded file and override: <install directory>\ext\w_class1_0.dl
5. Test and report back.

Thanks.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #27 on: February 27, 2014, 02:56:40 pm »
Anyone on the patch?

Offline pmcelreavy

  • Newbie
  • *
  • Posts: 21
  • Karma: +0/-0
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #28 on: February 27, 2014, 03:45:43 pm »
I wanted to try it but I did not find the w_class1_0.dl file anywhere.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Users with out of memory error, please report here.
« Reply #29 on: February 27, 2014, 04:21:53 pm »
I wanted to try it but I did not find the w_class1_0.dl file anywhere.
If you are running RAID-F, it is in the path as specified above. It is likely hidden. So, enable the viewing of hidden and system files to see it.