Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - djvj

Pages: [1] 2 3 4
1
General Discussion / Re: Issues upgrading to latest tRaid
« on: September 23, 2015, 09:10:39 pm »
Reverted to last release of Host and Client and has been stable for about an hour now. New version crashed after 5-15 minutes...

That's also with the new drive I added.

2
General Discussion / Re: Issues upgrading to latest tRaid
« on: September 23, 2015, 08:18:08 pm »
Don't know if it matters, but the broker never crashes until I have the storage pool enabled. Before enabling it the system works fine.

3
General Discussion / Re: Issues upgrading to latest tRaid
« on: September 23, 2015, 06:32:38 pm »
Well it was stable before I upgraded and added a drive. Which is why I removed the drive to take that out of the equation. Only variable was now the tRaid upgrade.

I also find it odd that if my system was so unstable, why isn't anything else crashing and generating crash dumps? Only thing in that folder at that time is NZFSB crashes. See attached.

I can try downgrading again and getting the older version working with a new db.


4
General Discussion / Re: Issues upgrading to latest tRaid
« on: September 23, 2015, 05:36:59 pm »
Ok thanks for the folder, they were there! These are all separate crashes.

http://pastebin.com/qQRkkP1j
http://pastebin.com/qsKsPXcw
http://pastebin.com/XjniWfwR
http://pastebin.com/MpzrSYuT

The above were all 8kb files, but the one attached had 5 files in it so it's attached as a zip. Hope this helps.

5
General Discussion / Re: Issues upgrading to latest tRaid
« on: September 23, 2015, 12:05:34 am »
Ok that makes perfect sense about hte issue with downgrading then. I don't have a backup of the database though, so I tried upgrading again to get you the info you need.

So after upgrading, about 5 minutes in, the broker crashed and pool was removed.

This is the info from the event viewer but it did not provide a crash dump.

Error
Code: [Select]
Log Name:      System
Source:        Service Control Manager
Date:          9/23/2015 12:53:28 AM
Event ID:      7034
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Storinator
Description:
The NZFS Broker Service service terminated unexpectedly.  It has done this 1 time(s).
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="49152">7034</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2015-09-23T04:53:28.495711000Z" />
    <EventRecordID>59281</EventRecordID>
    <Correlation />
    <Execution ProcessID="764" ThreadID="2388" />
    <Channel>System</Channel>
    <Computer>Storinator</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">NZFS Broker Service</Data>
    <Data Name="param2">1</Data>
    <Binary>4E005A004600530042000000</Binary>
  </EventData>
</Event>

Warning:
Code: [Select]
Log Name:      System
Source:        disk
Date:          9/23/2015 12:53:28 AM
Event ID:      157
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      Storinator
Description:
Disk 35 has been surprise removed.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="disk" />
    <EventID Qualifiers="32772">157</EventID>
    <Level>3</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-09-23T04:53:28.496712900Z" />
    <EventRecordID>59282</EventRecordID>
    <Channel>System</Channel>
    <Computer>Storinator</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\Harddisk35\DR35</Data>
    <Data>35</Data>
    <Binary>0000000002003000000000009D000480000000000000000000000000000000000000000000000000</Binary>
  </EventData>
</Event>

The host log on TRACE has no info regarding the crash, just data on the files on the drives.

I still have procdump running on the broker service, which closes out right when the broker crashes, it doesn't give me a log though. I'm not sure if any data from it would be useful for you but so far nothing out of the ordinary I've noticed.

This is the web client log:
http://pastebin.com/8xSJAHJD

I just noticed some application errors about 5 minutes before the above entries, mostly about dlls and HDSentinal monitoring I use for drive temps. For testing, I just disabled the HDSentinal and started the pool again to see if by chance that may have caused it. And...still crashed, so that had nothing to do with it.

Another warning log on 2nd crash:
Code: [Select]
Log Name:      System
Source:        disk
Date:          9/23/2015 1:20:41 AM
Event ID:      51
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      Storinator
Description:
An error was detected on device \Device\Harddisk35\DR36 during a paging operation.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="disk" />
    <EventID Qualifiers="32772">51</EventID>
    <Level>3</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-09-23T05:20:41.407240900Z" />
    <EventRecordID>59299</EventRecordID>
    <Channel>System</Channel>
    <Computer>Storinator</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\Harddisk35\DR36</Data>
    <Binary>030080000100000000000000330004802D010000130000C0000000000000000000EABBF3EE280000C30D020000000000FFFFFFFF0100000058000084020000000020101240032040000200003C0000003022851900E0FFFF280B4E0700E0FFFF0000000000000000904A201A00E0FFFF1050671A00E0FFFFFFFFFFFF000000008800000000147779DDF5000000010000F00002000000000B000000003A0000000000000000000000</Binary>
  </EventData>
</Event>

6
General Discussion / Issues upgrading to latest tRaid
« on: September 22, 2015, 05:00:38 pm »
So my space was starting to run low and I decided to add another drive to the pool. At the same time, the new tRaid version came out and figured I would add the drive and upgrade the tRaid software as well. My current setup has been going strong for at least 8 months or so now.

Adding the drive went well, along with upgrading to the latest tRaid. The next day, I noticed the storage pool was not showing up in windows explorer. I checked windows events and it said the disk #36 was suddenly removed. Drive #36 is the storage pool, not a physical drive. No other errors are reported.

When trying to restart the pool, I got errors in the client. The errors were because the broker service stopped running. Checking windows services and the broker was stopped. Simply restarting it works and refreshing the host list in the web client allowed me to connect back with my host and could restart the pool and it showed back up in windows explorer. Thing is, now within 5-15 minutes the pool disappears again and the broker service shows it stopped.

Being that I made 2 changes on the last maintenance, I decided to mark the new drive as failed in the web client and see if the same behavior would occur with the disk not inserted. After removing the drive and on a fresh reboot, the same exact behavior occurred. So the only change now was the upgrade to the new tRaid software version. Host log had no info in it and client logs only showed info when I tried connecting to the host before refreshing the host list. Host log was not on TRACE at the time though, just INFO.

Now I decided to revert back to the old version from 11/2014 to make sure I can get the stability back, then try adding only the drive and going from there. Problem is I reverted fine, but the hosts explorer no longer shows anything. Upon logging in, I get a couple errors in the client that you can see in the attached screenshot. When right clicking to refresh hosts, the menu shows nothing and has no options. I've treid chrome and IE, same result. The dashboard however shows my drives and host fine, but hosts explorer isn't detecting it...

Host Log:
http://pastebin.com/wScJpB3X

Web Client log:
http://pastebin.com/AaDSGgTT

7
He said he tried disabling it.

8
General Discussion / Re: Failure Verify Sync & no error in log to reflect
« on: January 24, 2015, 02:44:29 am »
So in the client log, I do get this, but that's all:
Code: [Select]
[2015-01-21 05:00:00,033] INFO  executeAction(363) - Starting tRAID
[2015-01-21 05:00:00,041] INFO  executeAction(363) - Processing range from 800.0GB to 1.781TB...
[2015-01-21 21:09:22,117] WARN  send(42) -
java.lang.NullPointerException
at java.net.URLEncoder.encode(URLEncoder.class)
at com.techventus.server.voice.Voice.sendSMS(Voice.java:1080)
at com.tchegbe.lib.gwt.server.notification.SMSAlert.execute(SMSAlert.java:107)
at com.tchegbe.nzfs.ui.server.op.NZFSNotificationOperations.send(NZFSNotificationOperations.java:42)
at com.tchegbe.nzfs.ui.server.op.NZFSNotificationOperations.sendNotification(NZFSNotificationOperations.java:42)
at com.tchegbe.nzfs.ui.server.util.StatusTracker.run(StatusTracker.java:107)
at java.lang.Thread.run(Thread.class)
[2015-01-22 05:00:00,010] INFO  executeAction(363) - Starting tRAID
[2015-01-22 05:00:00,020] INFO  executeAction(363) - Processing range from 1.781TB to 2.781TB...
[2015-01-22 11:47:27,484] WARN  send(42) -
java.lang.NullPointerException
at java.net.URLEncoder.encode(URLEncoder.class)
at com.techventus.server.voice.Voice.sendSMS(Voice.java:1080)
at com.tchegbe.lib.gwt.server.notification.SMSAlert.execute(SMSAlert.java:107)
at com.tchegbe.nzfs.ui.server.op.NZFSNotificationOperations.send(NZFSNotificationOperations.java:42)
at com.tchegbe.nzfs.ui.server.op.NZFSNotificationOperations.sendNotification(NZFSNotificationOperations.java:42)
at com.tchegbe.nzfs.ui.server.util.StatusTracker.run(StatusTracker.java:107)
at java.lang.Thread.run(Thread.class)
[2015-01-22 21:32:53,154] INFO  executeAction(363) - Starting tRAID
[2015-01-22 21:32:53,157] INFO  executeAction(363) - Processing range from 2.781TB to 3.638TB...

Getting a failure when only blocks were updated is VERY misleading to the end user. Regardless, that email needs more info, and nowhere is it stating blocks were updated and I might need to check my settings. How am I supposed to possibly know this with just the above info given to me?

So right now my CQ is 64, and my salt is 16, which I believe is default. I'm not quite sure what I should set these to, even after reading the article you posted. But with 16GB ram and 6 free atm, I should have enough to play around. From that Read Update so high, I'm thinking it's what's affecting my write performance. Think I will try doubling the values for now and see what happens. Are these settings that will apply on the fly with OS caching on or do they always need a reboot?

I should show you the logs I give to my users, they are very descriptive to where to look and what to change to fix errors in my apps. Helpful and descriptive logging goes a long way man.

9
General Discussion / Re: Failure Verify Sync & no error in log to reflect
« on: January 23, 2015, 06:31:16 am »
So since making Verify Sync do 1024GB at a time, I have had 3 scheduled events and 3 failures in this task. The log reports nothing and nothing is standing out in windows events either. Where do I start figuring out why this is failing?

10
General Discussion / Failure Verify Sync & no error in log to reflect
« on: January 22, 2015, 03:58:25 am »
Got an email this morning that I had a failure syncing.

The email contains no info on what triggered the error though. Can you please send off the info that triggered the error too?

I don't really know what triggered the error, which is a big problem. Nothing is standing out saying, this caused your error. Also the notification at the top of the client should have some more functionality, like clicking to bring you to a page with what caused the failure, or hovering over it to show the cause of the errors. Bringing errors to the front will go a long way to ease of use.

My logging level on the environment variable FLEXRAID_NZFS_LOG_LEVEL is set to ERROR, yet not one thing appeared in the log. This is the last entry:


Code: [Select]
[2015-01-21 04:01:08.899137][1716]
[2015-01-21 04:01:08.899139][1716]
[2015-01-21 04:01:08.899139][1716]=======================================================================
[2015-01-21 04:01:08.899139][1716]|||||||||||||||||||||   NZFS Service Start ||||||||||||||||||||||||||||
[2015-01-21 04:01:08.899139][1716]=======================================================================

11
General Discussion / Re: Missing space after deleting files
« on: January 22, 2015, 03:43:50 am »
Ok disabled it for the tRaid array as a whole. Does it need to be done for each disk mounted, or is just the entire array fine?

12
General Discussion / Null error when scheduling job
« on: January 21, 2015, 04:00:35 am »
Brahim, might have a bug here, or just to fill this box with a proper reason for the error. I'm simply trying to add the job shown and I get this error.

Once I added a start and end date, it took. I didn't have to add dates on my verify sync raid job, so why should I on this on? Regardless, error should be more informative than that.

13
General Discussion / Re: Missing space after deleting files
« on: January 21, 2015, 03:13:26 am »
Brahim, attached is what I found on the 1st two drives I mounted. Why is flex not deleting properly and sending data to these recycle bins that I cannot access w/o going offline? I can't have this keep happening every time I delete files.

I have about 20 drives, and every one I mount has old deleted data on it in those same folders and says the recycle bin is corrupted. So basically when I delete stuff, I'm not getting the space back! I have a Sync run every sunday and are successful, but not sure that has anything to do with removing deleted files cause that's not working if it does.

My Scheduled Range Operation is set to 100GB, and if it runs once a week, I'm not really getting much done with ~40TB. I just upped it, but not sure this matters for the deletion problem.

14
General Discussion / Re: Swap drive stuck in executing
« on: January 18, 2015, 09:31:51 pm »
You can usually find them in the root of your C drive unless you changed the paths to them.

15
General Discussion / Re: Missing space after deleting files
« on: January 18, 2015, 09:28:02 pm »
So now this gets worse, my media scraper is reporting that there is no space left on disk. When that is not true as the array shows 628GB left. That's besides the 2TB I'm somehow missing.

The other app is dumping the error:
System.IO.IOException: There is not enough space on the disk.

Why is space being reported wrong and/or missing to windows?? Something screwy is going on here.

Pages: [1] 2 3 4