Author Topic: Out of memory on linux with 8GB RAM during reconcile  (Read 3689 times)

Offline birkinshawc

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Out of memory on linux with 8GB RAM during reconcile
« on: July 22, 2013, 06:12:08 pm »
Reconcile job says it has failed in the GUI: "Error .RunTime exception". Looking at linux top I see the FlexRAID process using 85% of the memory in my server, which has 8GB.

Will send the full logs privately. End of the FlexRAID.log is below:

Code: [Select]
[2013-07-22 21:36:58,822] INFO : Adding /root/FlexRAID-Managed-Pool/class1_0/{5972901b-23de-4986-ae48-d59de036a93f}/TV/BBC LIFE/BBC.Life.s01e08.Creatures.of.the.Deep.2009.HDTV.720p.x264.AC3.mkv size=0
[2013-07-22 21:36:58,822] INFO : Adding /root/FlexRAID-Managed-Pool/class1_0/{5972901b-23de-4986-ae48-d59de036a93f}/TV/BBC LIFE/BBC.Life.s01e09.Plants.2009.HDTV.720p.x264.AC3.mkv size=0
[2013-07-22 21:41:40,846] ERROR:
java.lang.RuntimeException
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
Caused by: java.lang.OutOfMemoryError
... 4 more

I do have quite a lot of files on the system, as I use it to store Crashplan backups and Windows 8 file history backups.


Thanks,

Chris

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Out of memory on linux with 8GB RAM during reconcile
« Reply #1 on: July 22, 2013, 08:28:47 pm »
Chances are, you have a high fragmentation level.
See this post: http://forum.flexraid.com/index.php/topic,1099.msg8937.html#msg8937
You might need to set it as high as 1MB if you have mostly large files.

FlexRAID is a 32 bit process and limited to how much memory it can use.

Offline birkinshawc

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: Out of memory on linux with 8GB RAM during reconcile
« Reply #2 on: July 24, 2013, 04:15:48 pm »
Thanks for the fast response.

I set it to 512KB on the second attempt which seems to have worked.

However I now get the following error:

Code: [Select]
[2013-7-24 21:52:40.149933][3777014592]Mounting using /media/pool1
[2013-7-24 21:52:40.149940][3777014592]Using program name FlexRAIDFS
[2013-7-24 21:52:40.149947][3777014592]Granting storage pool access to all other users...
[2013-7-24 21:52:40.149953][3777014592]Running in single threaded mode...
[2013-7-24 21:52:40.149959][3777014592]Enabling large writes...
[2013-7-24 21:52:40.150049][3777014592]Calling main
[2013-7-24 21:52:40.161553][3777014592][on_init]: major=7, minor=19, want=50, capable=4091, async_read=1, max_readahead=131072, max_write=131072
[2013-7-24 21:58:5.236339][3777014592] [on_open]: errorMsg=Too many open files for /Crashplan Backups/513532437772369932/cpbf0000000000005880348/cpbdf -> /root/FlexRAID-Managed-Pool/class1_0/{247b2a8d-f158-4b81-bd75-7e4b247833b1}/Crashplan Backups/513532437772369932/cpbf0000000000005880348/cpbdf
[2013-7-24 21:58:5.237674][3777014592] [on_open]: errorMsg=Too many open files for /Crashplan Backups/513532437772369932/cp.properties -> /root/FlexRAID-Managed-Pool/class1_0/{247b2a8d-f158-4b81-bd75-7e4b247833b1}/Crashplan Backups/513532437772369932/cp.properties
[2013-7-24 21:58:5.238892][3777014592] [on_opendir]: errorMsg=Too many open files for /Crashplan Backups/513532437772369932 -> /root/FlexRAID-Managed-Pool/class1_0/{61901cee-ed24-424f-b511-445c2f621fff}/Crashplan Backups/513532437772369932

Offline birkinshawc

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: Out of memory on linux with 8GB RAM during reconcile
« Reply #3 on: July 24, 2013, 04:18:54 pm »
After this latest crash I got the message to reconcile, so I restored the latest backup and started the reconcile. But this failed:

Code: [Select]
[2013-07-24 22:12:46,974] WARN : RAID error detected. Checking for pending operations before exiting...
[2013-07-24 22:12:46,974] INFO : Unmounting View...
[2013-07-24 22:12:47,012] INFO : Unmounting View complete...
[2013-07-24 22:12:47,012] INFO : Closing Registry...
[2013-07-24 22:12:47,023] INFO : Closing Registry complete...
[2013-07-24 22:12:55,734] INFO : Executing copy-live-dbs...
[2013-07-24 22:12:55,735] INFO : [copy-live-dbs] started at: Wed Jul 24 22:12:55 BST 2013
[2013-07-24 22:12:55,840] INFO : [copy-live-dbs] completed at: Wed Jul 24 22:12:55 BST 2013
[2013-07-24 22:15:14,122] ERROR: Need for reconciliation detected! Please run the reconcile task...
[2013-07-24 22:15:14,214] WARN : Zero live size for /root/FlexRAID-Managed-Pool/class1_0/{61901cee-ed24-424f-b511-445c2f621fff}/Crashplan Backups/513541299162316817/cptl
[2013-07-24 22:15:14,243] ERROR: Invalid state for live info with offset but size=0 - name=/root/FlexRAID-Managed-Pool/class1_0/{61901cee-ed24-424f-b511-445c2f621fff}/Crashplan Backups/513541299162316817/cptl - hash=-1515190026 - handle=-448779144 - offset=67392 - size=0
java.lang.RuntimeException: Invalid state for live info with offset but size=0 - name=/root/FlexRAID-Managed-Pool/class1_0/{61901cee-ed24-424f-b511-445c2f621fff}/Crashplan Backups/513541299162316817/cptl - hash=-1515190026 - handle=-448779144 - offset=67392 - size=0
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Out of memory on linux with 8GB RAM during reconcile
« Reply #4 on: July 26, 2013, 06:49:06 am »
1. Google for "linux too many open files". You should be able to find suggested settings that will help.

2. Reconciling is part of the life of RT-RAID. Always restore the backup prior to the last or even the one before that before you reconcile.
The last backup is usually out of sync.

3. You might need to bump it up to 1MB.

How many backup jobs are you running on the pool?
If many in parallel, this might be the issue as RT RAID is not well suited for that as it will incur a high fragmentation level.

Offline birkinshawc

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: Out of memory on linux with 8GB RAM during reconcile
« Reply #5 on: July 26, 2013, 07:26:32 am »
My system wide setting for max open files seems quite large:

fs.file-max = 393076

I then checked out the max open files setting for the FlexRAID process:

Soft Limit            1024
Hard limit             4096

Seems quite low. What do you advise to set these to? How can I set this higher for the FlexRAID process?


Regarding loading/use case... I have Crashplan installed locally on the server which is using the pool to store backups. I am also backing up a Windows 8 machine over SMB. My pool currently contains about 50% media files and 50% backup data. I had considered 2 separate pools... this is still an option. I am coming over from unRAID where all this lived in one RT pool, so tried to clone that functionality in FlexRAID.

Do you think I should switch to snapshot for this particular application?

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Out of memory on linux with 8GB RAM during reconcile
« Reply #6 on: July 26, 2013, 09:27:39 am »
Try Google'ing on "linux increase file descriptors per process" to find what are the suggested settings for your distro.

I think Transparent RAID will be the best option for you although it is currently in RC release mode and the Linux release is even further out.
Else, give Snapshot RAID a trial. When you do, set "Differential Check" property to true, and update the "Differential Check Match" property with the file extensions of Crashplan.

RT-RAID is experimental and does not fit all realtime RAID scenarios. Processes that create a lot of fragmentation (like Crashplan) might not be good candidate for RT-RAID depending on the usage.
I have used Crashplan just fine on RT-RAID, but I think my data set was relatively small.

Ultimately, RT-RAID will be phased out in favor of tRAID.
With Transparent RAID around, there is just no compelling reason to continue the development of RT-RAID even though it was an ingenious feat.