Author Topic: FlexRAID crashing when writing files via NFS or NFS4  (Read 4254 times)

Offline brdhse1

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
FlexRAID crashing when writing files via NFS or NFS4
« on: April 06, 2013, 12:49:25 pm »
FlexRAID crashes when I try to copy files to my RT pool via NFS or NFS4.  I'm trying to copy a 800MB file, and FR crashes before the copy finishes.  I can copy the same file without error via SMB or RCP.

I attached TRACE logs from the crash.  Let me know if I can provide any other details.

Thanks!

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: FlexRAID crashing when writing files via NFS or NFS4
« Reply #1 on: April 06, 2013, 03:31:39 pm »
Hum... I wonder if copying via NFS is failing to pre-allocate the file.
Did you get a chance to monitor the memory usage when it happened?

Offline brdhse1

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: FlexRAID crashing when writing files via NFS or NFS4
« Reply #2 on: April 06, 2013, 03:50:46 pm »
Monitor RAM?  I just ran the test again and I didn't see RAM usage change during the copy.  Is there any other way to know if it is pre-allocating the file?

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: FlexRAID crashing when writing files via NFS or NFS4
« Reply #3 on: April 06, 2013, 04:31:56 pm »
Download the CMD client and run the following command: live class1_0 fragmentation-report

Report back.

Offline brdhse1

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: FlexRAID crashing when writing files via NFS or NFS4
« Reply #4 on: April 06, 2013, 05:19:19 pm »
Well, this doesn't look promising.

This is the exact command I ran:
$ sudo ./FlexRAIDCMD live class1_0 fragmentation-report

Did I do that correctly?

Silly question... Which JRE are you expecting?  This is all running on a clean install of Ubuntu.  I didn't explicitly install any JRE yet.

See attached log.


Offline brdhse1

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: FlexRAID crashing when writing files via NFS or NFS4
« Reply #6 on: April 06, 2013, 05:42:48 pm »
See attached.


Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: FlexRAID crashing when writing files via NFS or NFS4
« Reply #7 on: April 06, 2013, 07:46:42 pm »
What are you showing me?  ???

I don't see anything about you correctly running the requested operation.

Offline brdhse1

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: FlexRAID crashing when writing files via NFS or NFS4
« Reply #8 on: April 07, 2013, 12:21:08 am »
I read the link you sent, which was a rather long post about scripting the CMD client.  From that post, I gathered that this is the command you wanted me to run:

Code: [Select]
sudo ./FlexRAIDCMD localhost - - live class1_0 fragmentation-report

It appeared that all of the stdout from that command was also going into the log files that I attached.  Just to make sure, this is the stdout I got from the above command:

Code: [Select]
{"success": true, "status": {"referenceCode": "2", "task": "fragmentation-report", "status": "STATUS_STARTED", "progress": null, "report": null, "processes": 0, "processesRequested": 0}, "commandMessages": [{"messageCode": "successTaskInitiated", "messageData": ["fragmentation-report", "2"]}], "serverMessages": null}
=> The [<[fragmentation-report]>] task has successfully initiated... Process number: 2


If I didn't run the correct command or if I didn't attach the correct output, please tell me which command you want to run and where I can find the correct output. 

Any help is much appreciated.

Thanks.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: FlexRAID crashing when writing files via NFS or NFS4
« Reply #9 on: April 07, 2013, 11:19:15 am »
Looks correct.
Now, you have to wait for it to finish.

You can query for the status using the "reference" value.

The command to run is: status <ID>
In your case that would have been: status 2

Offline Stokkes

  • Newbie
  • *
  • Posts: 48
  • Karma: +0/-0
    • View Profile
Re: FlexRAID crashing when writing files via NFS or NFS4
« Reply #10 on: April 08, 2013, 04:51:41 pm »
I'm gonna throw this in this thread, hope it's not off-topic.

I decided to move my Plex Media Server to a Mac OSX VM for a few reasons (I needed silverlight and flash). I exposed my FlexRAID pool which is mounted in /mnt/raid (Snapshot RAID, not RT) as both NFS and NFS v4. After a certain period of time, the storage pool stops responding and I believe FlexRAID is crashing although the process is still running.

My pool is mounted in /mnt/raid and when I experience this behaviour, two things happen:

1. Trying to browse /mnt/raid in a CLI hangs the linux CLI
OR
2. I get an error "Transport endpoint is not connected"

I never had FlexRAID crash before (not I am not doing any writing, just a lot of scanning of files). This has only happened since I moved my Plex server and exposed the storage pool via NFS (it's always been accessible via SMB without any problems).

The funny thing (or maybe not) is that I have verbose debug logging on and there is NOTHING in FlexRAID.log or FlexRAID.sys.log that would indicate a bad write/read or anything indicating a crash.

I have to kill the FlexRAID process, kill the nfs-kernel-server service and restart everything. On the odd occasion, I've had to restart the linux box (Ubuntu 12.04) completely.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: FlexRAID crashing when writing files via NFS or NFS4
« Reply #11 on: April 08, 2013, 05:05:59 pm »
I have not looked at NFS4, but I would say you should revert to a version "less bleeding edge".
NFS brings it own set of problems.

Offline Stokkes

  • Newbie
  • *
  • Posts: 48
  • Karma: +0/-0
    • View Profile
Re: FlexRAID crashing when writing files via NFS or NFS4
« Reply #12 on: April 08, 2013, 05:14:51 pm »
Ya I know NFSv4 is a different beast. Let's just say that doing a mount --bind on the storage pool, well, FlexRAID does NOT like that.

I reverted to NFS3 for the exports:
Code: [Select]
/mnt/raid 10.0.1.0/255.255.255.0(rw,nohide,sync,insecure,root_squash,no_subtree_check,fsid=0)

But it will crash periodically, usually within 24-48 hours. Sometimes sooner. I have used NFS in the past with other solutions, both personally and professionally. It's a great file sharing protocol for linux systems (and even Mac systems). Mac NFS is faster than Mac SMB.. AFP would be preferred but I'm not a fan of Netatalk.


Offline brdhse1

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: FlexRAID crashing when writing files via NFS or NFS4
« Reply #13 on: April 21, 2013, 04:46:05 pm »
Quote
But it will crash periodically, usually within 24-48 hours.

Are you saying that it's still crashing for you with NFSv3?  I changed from RT to Snapshot because I had other stability issues with RT.  At first NFSv4 was working fine with Snapshot, but now I'm staring to see issues.  I'd switch to NFSv3, but if I'm still going to have problems, I'm not sure what to do.

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: FlexRAID crashing when writing files via NFS or NFS4
« Reply #14 on: April 21, 2013, 07:46:04 pm »
Are you saying that it's still crashing for you with NFSv3?  I changed from RT to Snapshot because I had other stability issues with RT.  At first NFSv4 was working fine with Snapshot, but now I'm staring to see issues.  I'd switch to NFSv3, but if I'm still going to have problems, I'm not sure what to do.
If you are using NFSv4 with your issue posted about SVN, you certainly want to revert to a different version.