Author Topic: java.lang.OutOfMemoryError  (Read 3489 times)

Offline blatus

  • Newbie
  • *
  • Posts: 20
  • Karma: +0/-0
    • View Profile
java.lang.OutOfMemoryError
« on: February 04, 2013, 06:33:50 am »
Hi all,

I'm having a quite severe problem here. FlexRAID is using up copious amounts of RAM (around 3GB) without giving it up and then after a while it crashes. I have logs set to trace and as I was looking over them I see that the logs are filled with smart notices like below. Sooner or later they all end with an out of memory error and FR just crashes.

Any idea what this could be? More information is on the bug tracker but I didn't get any response there so I thought I'd cross-post here.
http://bug.flexraid.com/view.php?id=287

Let me know if I can supply more information. This is starting to get annoying to restart FR every morning and hoping it'll survive the day...

Best,
blatus

[2013-02-04 02:36:35,958] TRACE: Executing: smart
[2013-02-04 02:36:36,532] TRACE:    => {"success": true, "status": null, "commandMessages": [], "serverMessages": null, "infos": [{"id": 1, "name": "Raw_Read_Error_Rate", "flag": "0x002f", "value": "200", "worst": "200", "threshold": "051", "type": "Pre-fail", "updated": "Always", "whenFailed": "-", "rawValue": "0"}, {"id": 3, "name": "Spin_Up_Time", "flag": "0x0027", "value": "175", "worst": "169", "threshold": "021", "type": "Pre-fail", "updated": "Always", "whenFailed": "-", "rawValue": "6241"}, {"id": 4, "name": "Start_Stop_Count", "flag": "0x0032", "value": "100", "worst": "100", "threshold": "000", "type": "Old_age", "updated": "Always", "whenFailed": "-", "rawValue": "152"}, {"id": 5, "name": "Reallocated_Sector_Ct", "flag": "0x0033", "value": "200", "worst": "200", "threshold": "140", "type": "Pre-fail", "updated": "Always", "whenFailed": "-", "rawValue": "0"}, {"id": 7, "name": "Seek_Error_Rate", "flag": "0x002e", "value": "200", "worst": "200", "threshold": "000", "type": "Old_age", "updated": "Always", "whenFailed": "-", "rawValue": "0"}, {"id": 9, "name": "Power_On_Hours", "flag": "0x0032", "value": "096", "worst": "096", "threshold": "000", "type": "Old_age", "updated": "Always", "whenFailed": "-", "rawValue": "3536"}, {"id": 10, "name": "Spin_Retry_Count", "flag": "0x0032", "value": "100", "worst": "100", "threshold": "000", "type": "Old_age", "updated": "Always", "whenFailed": "-", "rawValue": "0"}, {"id": 11, "name": "Calibration_Retry_Count", "flag": "0x0032", "value": "100", "worst": "253", "threshold": "000", "type": "Old_age", "updated": "Always", "whenFailed": "-", "rawValue": "0"}, {"id": 12, "name": "Power_Cycle_Count", "flag": "0x0032", "value": "100", "worst": "100", "threshold": "000", "type": "Old_age", "updated": "Always", "whenFailed": "-", "rawValue": "53"}, {"id": 192, "name": "Power-Off_Retract_Count", "flag": "0x0032", "value": "200", "worst": "200", "threshold": "000", "type": "Old_age", "updated": "Always", "whenFailed": "-", "rawValue": "52"}, {"id": 193, "name": "Load_Cycle_Count", "flag": "0x0032", "value": "198", "worst": "198", "threshold": "000", "type": "Old_age", "updated": "Always", "whenFailed": "-", "rawValue": "8713"}, {"id": 194, "name": "Temperature_Celsius", "flag": "0x0022", "value": "115", "worst": "097", "threshold": "000", "type": "Old_age", "updated": "Always", "whenFailed": "-", "rawValue": "35"}, {"id": 196, "name": "Reallocated_Event_Count", "flag": "0x0032", "value": "200", "worst": "200", "threshold": "000", "type": "Old_age", "updated": "Always", "whenFailed": "-", "rawValue": "0"}, {"id": 197, "name": "Current_Pending_Sector", "flag": "0x0032", "value": "200", "worst": "200", "threshold": "000", "type": "Old_age", "updated": "Always", "whenFailed": "-", "rawValue": "0"}, {"id": 198, "name": "Offline_Uncorrectable", "flag": "0x0030", "value": "200", "worst": "200", "threshold": "000", "type": "Old_age", "updated": "Offline", "whenFailed": "-", "rawValue": "0"}, {"id": 199, "name": "UDMA_CRC_Error_Count", "flag": "0x0032", "value": "200", "worst": "200", "threshold": "000", "type": "Old_age", "updated": "Always", "whenFailed": "-", "rawValue": "0"}, {"id": 200, "name": "Multi_Zone_Error_Rate", "flag": "0x0008", "value": "200", "worst": "200", "threshold": "000", "type": "Old_age", "updated": "Offline", "whenFailed": "-", "rawValue": "0"}]}
[2013-02-04 03:19:24,915] ERROR:
java.lang.OutOfMemoryError
   at java.lang.Void.<unknown>(Unknown Source)
   at java.lang.Void.<unknown>(Unknown Source)
   at java.lang.Void.<unknown>(Unknown Source)
   at java.lang.Void.<unknown>(Unknown Source)
[2013-02-04 03:21:15,935] ERROR:
java.lang.OutOfMemoryError
   at java.lang.Void.<unknown>(Unknown Source)
   at java.lang.Void.<unknown>(Unknown Source)
   at java.lang.Void.<unknown>(Unknown Source)
   at java.lang.Void.<unknown>(Unknown Source)
[2013-02-04 03:22:26,400] ERROR:
java.lang.OutOfMemoryError
   at java.lang.Void.<unknown>(Unknown Source)
   at java.lang.Void.<unknown>(Unknown Source)
   at java.lang.Void.<unknown>(Unknown Source)
   at java.lang.Void.<unknown>(Unknown Source)

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: java.lang.OutOfMemoryError
« Reply #1 on: February 04, 2013, 12:00:26 pm »
A few things:
- Post the actually memory usage of FlexRAID (take a screenshot of your task manager showing the usage)
- Clarify whether this is happening when it is idle
- Make sure nothing is interfering with FlexRAID activities (configure your AV to leave FlexRAID alone)

Offline blatus

  • Newbie
  • *
  • Posts: 20
  • Karma: +0/-0
    • View Profile
Re: java.lang.OutOfMemoryError
« Reply #2 on: February 04, 2013, 12:33:57 pm »
Here's a few screen shots. I uploaded them to the bugtracker.

RAM usage increases when I play video files from the pool or move some data. RAM is not being freed up afterwards. After an UPDATE it goes through the roof.
I use MSE but have added flexraid.exe to the ignored process list.

Like I said on the tracker, I have lots of logs where FR crashed, however I do not want to upload them publicly. Should I rar them up with a pass and PM you that?

Cheers,
blatus

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: java.lang.OutOfMemoryError
« Reply #3 on: February 04, 2013, 12:45:06 pm »
I would test with all AV and whatnot disabled.

Offline blatus

  • Newbie
  • *
  • Posts: 20
  • Karma: +0/-0
    • View Profile
Re: java.lang.OutOfMemoryError
« Reply #4 on: February 05, 2013, 12:31:50 am »
Left PC on overnight. No updates were scheduled, AV was disabled and no videos were played/streamed. RAM is now at 2.47GB.

As I said on the tracker. This did not happen at first. It started popping up one day out of the blue. Do you think it could be fixed by re-installing / re-creating the array and whatnot? I was hesitant to do that as I thought maybe there's an issue here that needs fixing. But maybe some odd thing just screwed up something? I am at a complete loss in any case.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: java.lang.OutOfMemoryError
« Reply #5 on: February 05, 2013, 10:05:31 am »
Yeah, I would try a clean reinstall (reboot + delete installation folder).

Offline blatus

  • Newbie
  • *
  • Posts: 20
  • Karma: +0/-0
    • View Profile
Re: java.lang.OutOfMemoryError
« Reply #6 on: February 05, 2013, 10:41:33 am »
I will have to try that then. Can I Keep the parity data? Or should I wipe the PPU clean and recalculate?

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: java.lang.OutOfMemoryError
« Reply #7 on: February 05, 2013, 11:59:48 am »
I will have to try that then. Can I Keep the parity data? Or should I wipe the PPU clean and recalculate?

Data and parity will stay untouched and you won't need to recompute parity.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: java.lang.OutOfMemoryError
« Reply #8 on: February 06, 2013, 01:40:42 pm »
Also, check this previous bug report and the resolution: http://bug.flexraid.com/view.php?id=257

Offline blatus

  • Newbie
  • *
  • Posts: 20
  • Karma: +0/-0
    • View Profile
Re: java.lang.OutOfMemoryError
« Reply #9 on: February 07, 2013, 04:25:53 am »
I get an access denied when trying to open that link.

Also, I re-installed FR. Uninstall -> remove /openegg folder in program files. Restarted. FR installed pool driver. Restarted again. However, the problem persists. I am currently at 3GB RAM usage (after roughly 9h). AV is still off.
This is again after the reinstall yesterday, same crash as before:
[2013-02-04 20:41:31,821] INFO : Successful UPDATE operation!
[2013-02-04 20:41:32,284] INFO : [update] completed at: Mon Feb 04 20:41:32 CET 2013
[2013-02-05 20:00:00,046] INFO : Trigger update.Media[update][Every Day at: 20:00:00
[Starting = null/null/null null:null:null]
[Ending = null/null/null null:null:null]] fired job Media.Media[update] at:  20:00:00 02/05/2013
[2013-02-05 20:00:00,062] TRACE: Executing: upload
[2013-02-05 20:00:00,324] TRACE:    => {"success": true, "status": {"referenceCode": "4", "task": "update", "status": "STATUS_STARTED", "progress": null, "report": null, "processes": 0, "processesRequested": 0}, "commandMessages": [{"messageCode": "successTaskInitiated", "messageData": ["update", "4"]}], "serverMessages": null}
[2013-02-05 20:00:00,337] INFO : Executing update...
[2013-02-05 20:00:00,342] INFO : [update] started at: Tue Feb 05 20:00:00 CET 2013
[2013-02-05 20:00:40,546] INFO : Using RAID engine Tx
[2013-02-05 20:00:41,805] TRACE: Exclusion match - Skipping C:\mounts\LR1\$RECYCLE.BIN...
[2013-02-05 20:00:43,618] TRACE: Exclusion match - Skipping C:\mounts\LR1\System Volume Information...
[2013-02-05 20:00:44,548] ERROR: [update] error: OutOfMemoryError[Thread creation error: 8]
java.lang.OutOfMemoryError: Thread creation error: 8
   at java.lang.Void.<unknown>(Unknown Source)
   at java.lang.Void.<unknown>(Unknown Source)
   at java.lang.Void.<unknown>(Unknown Source)
   at java.lang.Void.<unknown>(Unknown Source)
[2013-02-05 20:00:44,549] INFO : [update] completed at: Tue Feb 05 20:00:44 CET 2013

This is copy/pasted as is. What makes me worry is the timestamp added. It jumps from 20:41 to 20:00.

I sent you the link to the logs + password via PM.

Thanks for looking into this!

Best,
blatus

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: java.lang.OutOfMemoryError
« Reply #10 on: February 07, 2013, 09:42:24 am »
Try again on the link.
Did not get the password.

Offline blatus

  • Newbie
  • *
  • Posts: 20
  • Karma: +0/-0
    • View Profile
Re: java.lang.OutOfMemoryError
« Reply #11 on: February 07, 2013, 11:07:55 am »
Guess you wouldn't, seeing as I didn't put it in the PM...  :-[

Oh yea, I disabled the SMART monitoring in between, thus the hourly SMART notices in the logs have disappeared.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: java.lang.OutOfMemoryError
« Reply #12 on: February 07, 2013, 11:27:41 am »
Did you check the bug report link again?
Please read it and try the resolution there.

Offline blatus

  • Newbie
  • *
  • Posts: 20
  • Karma: +0/-0
    • View Profile
Re: java.lang.OutOfMemoryError
« Reply #13 on: February 07, 2013, 12:09:48 pm »
That bug report sounds familiar. Also XMBC + sab/etc and the same silent growing RAM usage.
However, the he solution there is to reinstall the OS. Quite frankly this is not really something I want to do. I have too much set up on this pc as is at the moment.
Any other ideas? Is there perhaps a way to check what gets loaded into FRs RAM usage?

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: java.lang.OutOfMemoryError
« Reply #14 on: February 07, 2013, 12:47:37 pm »
Try to isolate things.

1. Run the service without the pool running and see if that changes things.
2. Run the pool without the any RAID task being executed and see if that changes things.

Basically, let's try to isolate where the leak might be coming from.