Author Topic: Snapshot update failed  (Read 5869 times)

Offline derelict

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Snapshot update failed
« on: January 04, 2012, 03:16:10 am »
Hi

I get this errormessage in my logs:
Code: [Select]
[2012-01-04 02:00:00,596] INFO : Trigger update.SnapshotRaid[update][0 0 14,2 * * ?] fired job SnapshotRaid.SnapshotRaid[update] at:  02:00:00 01/04/2012
[2012-01-04 02:00:00,600] INFO : Executing update...
[2012-01-04 02:00:00,601] INFO : [update] started at: Wed Jan 04 02:00:00 CET 2012
[2012-01-04 02:00:12,530] INFO : Using RAID engine Tx
[2012-01-04 02:00:43,501] INFO : Total process size = 18726518784
[2012-01-04 02:01:09,264] WARN : C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_multimedia\_video\_Movies\Contagion (2011)\Contagion.mkv no longer exist! Skipping...
[2012-01-04 02:01:09,264] ERROR: Slippery file encountered... forced to abort... C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_multimedia\_video\_Movies\Contagion (2011)\Contagion.mkv
java.io.IOException: Slippery file encountered... forced to abort... C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_multimedia\_video\_Movies\Contagion (2011)\Contagion.mkv
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
[2012-01-04 02:01:09,396] INFO : [update] completed at: Wed Jan 04 02:01:09 CET 2012
[2012-01-04 02:01:09,498] WARN : Memory leak detected on native buffers! Leaks count = 1 out of 377
[2012-01-04 02:01:09,860] INFO : Sending email alert - message=[SnapshotRaid][UPDATE] FAILURE
[2012-01-04 02:01:10,728] INFO : [SnapshotRaid][UPDATE] FAILURE - emailed...
[2012-01-04 02:01:10,959] INFO : Trigger update.SnapshotRaid[update][0 0 14,2 * * ?] completed firing job SnapshotRaid.SnapshotRaid[update] at  02:01:10 01/04/2012 with resulting trigger instruction code: DO NOTHING

That file has recently been added.... but was fine at the time of the update! And it still exists and is still fine !?
« Last Edit: January 04, 2012, 03:17:52 am by derelict »

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Snapshot update failed
« Reply #1 on: January 04, 2012, 10:11:14 am »
The update will run again.
You can always trigger a manual update by going to the command center.

I can't tell you why there was an error on that file.

Offline derelict

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Snapshot update failed
« Reply #2 on: January 04, 2012, 10:19:41 am »
ok... it indeed successfully ran again at 14:00....

But thanks for the input.... so this is not a "bug" ?

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Snapshot update failed
« Reply #3 on: January 04, 2012, 10:43:19 am »
It is not a bug.
Another program could have had the file locked, or one of your disk timed out, etc.


Offline derelict

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Snapshot update failed
« Reply #4 on: January 10, 2012, 04:54:38 am »
OK... i run a validate job every friday morning... and a verify every monday morning....

no i've got for the validate job:

Code: [Select]
[2012-01-06 07:00:00,656] INFO : [validate] started at: Fri Jan 06 07:00:00 CET 2012
[2012-01-06 07:00:03,478] INFO : Using RAID engine Tx
[2012-01-06 10:28:00,833] WARN : Changed: C:\FlexRAID-Managed-Pool\class1_0\{99b7e86b-7fa8-424e-bd3b-5ab0ee6b83c4}\_data\_xbmc_sync\_logs\PC003.txt
[2012-01-06 10:33:45,112] WARN : Changed: C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\_logs\PC001.txt
[2012-01-06 10:33:45,112] WARN : Changed: C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\_logs\PC002.txt
[2012-01-06 10:33:45,112] WARN : Changed: C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\_logs\PC006.txt
[2012-01-06 10:33:45,759] WARN : Changed: C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\_logs\PC01.txt
[2012-01-06 10:47:28,180] WARN : Changed: C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\_logs\PC003.txt
[2012-01-06 10:47:28,314] WARN : Changed: C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\_logs\PC03.txt
[2012-01-06 10:47:30,454] WARN : Changed: C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\PC006\Eventghost_Config_2\presence.log.txt
[2012-01-06 10:47:30,455] WARN : Changed: C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_xbmc_sync\_logs\TN-PC03.txt
[2012-01-06 10:47:30,456] WARN : Changed: C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_xbmc_sync\_logs\USERDATA-PC03.txt
[2012-01-06 10:47:30,469] WARN : Changed: C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_xbmc_sync\_logs\TN-PC01.txt
[2012-01-06 10:47:30,470] WARN : Changed: C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_xbmc_sync\_logs\USERDATA-PC01.txt
[2012-01-06 11:05:19,939] ERROR: ?????HANDLE_VALUE error=32 for path=\\?\C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\eventghost_backup.cmd
java.io.IOException: ?????HANDLE_VALUE error=32 for path=\\?\C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\eventghost_backup.cmd
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)
at java.lang.Void.<unknown>(Unknown Source)

and for the verify job:

Code: [Select]
[2012-01-09 07:00:12,477] INFO : Starting verify at: 0 - Number of bytes: 989962698752
[2012-01-09 11:11:53,499] ERROR: Verify failure: bit mismatch!
[2012-01-09 11:11:53,500] ERROR: At byte: 950480417487...
[2012-01-09 11:11:53,500] ERROR: NOTE: C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_xbmc_sync\_logs\TN-PC01.txt was open at the time of the failure
[2012-01-09 11:11:53,500] ERROR: WARN: modified file => C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_xbmc_sync\_logs\TN-PC01.txt
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\PCS006\Eventghost_Config_2\v1_prod.xml
[2012-01-09 11:11:53,500] ERROR: WARN: modified file => C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_xbmc_sync\_logs\USERDATA-PC03.txt
[2012-01-09 11:11:53,500] ERROR: WARN: modified file => C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_xbmc_sync\_logs\TN-PC03.txt
[2012-01-09 11:11:53,500] ERROR: WARN: modified file => C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\PCS006\Eventghost_Config_2\presence.log.txt
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_xbmc_sync\_logs\USERDATA-PCDEVEL.txt
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_xbmc_sync\_logs\TN-PCDEVEL.txt
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\PCDEVEL\Eventghost_Config_1\prod_v2.xml
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\_logs\PCDEVEL.txt
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\_masterscript.cmd
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\PC01\Eventghost_Plugins_2\XBMCEventReceiver\__init__.pyc
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\PC01\Eventghost_Plugins_2\XBMCEventReceiver\__init__.py
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{ba674485-982a-4e5d-8888-81800b416674}\_data\_Station_Sync\_logs\PCDEVEL.txt
[2012-01-09 11:11:53,500] ERROR: NOTE: C:\FlexRAID-Managed-Pool\class1_0\{99b7e86b-7fa8-424e-bd3b-5ab0ee6b83c4}\_multimedia\_video\_Videos\AHS 2010\PICT0003.MP4 was open at the time of the failure
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{99b7e86b-7fa8-424e-bd3b-5ab0ee6b83c4}\_multimedia\_video\_Videos\AHS 2010\PICT0003.MP4
[2012-01-09 11:11:53,500] ERROR: NOTE:  was open at the time of the failure
[2012-01-09 11:11:53,500] ERROR: At:
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\WS-PC\Eventghost_Plugins_2\Snarl\pySnarl.py
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\PC03\Eventghost_Plugins_2\Snarl\pySnarl.py
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\PC02\Eventghost_Plugins_2\Snarl\pySnarl.py
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\WS-PC\Eventghost_Plugins_2\Snarl\pySnarl.pyc
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\PC03\Eventghost_Plugins_2\Snarl\pySnarl.pyc
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\PC02\Eventghost_Plugins_2\Snarl\pySnarl.pyc
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\PCS003\Eventghost_Config_1\_archive\2012-01-09___00-00-00__.v1_prod.xml
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\WS-PC\Eventghost_Plugins_2\Snarl\__init__.pyc
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\PC03\Eventghost_Plugins_2\Snarl\__init__.pyc
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\PC02\Eventghost_Plugins_2\Snarl\__init__.pyc
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\PCS002\Eventghost_Config_3\_archive\2012-01-08___15-40-00__.PS3handling.xml
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\PCS002\Eventghost_Config_3\_archive\2012-01-09___00-00-00__.PS3handling.xml
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\PCS002\Eventghost_Config_3\_archive\2012-01-08___15-55-00__.PS3handling.xml
[2012-01-09 11:11:53,500] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\WS-PC\Eventghost_Plugins_2\Snarl\__init__.py
[2012-01-09 11:11:53,501] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\PC03\Eventghost_Plugins_2\Snarl\__init__.py
[2012-01-09 11:11:53,501] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\PC02\Eventghost_Plugins_2\Snarl\__init__.py
[2012-01-09 11:11:53,501] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\PCS006\Eventghost_Config_2\_archive\2012-01-09___00-00-00__.v1_prod.xml
[2012-01-09 11:11:53,501] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e80-314f-11e1-844c-005056c00008}\_data\_Station_Sync\PCS006\Eventghost_Config_2\_archive\2012-01-08___15-10-00__.v1_prod.xml
[2012-01-09 11:11:53,501] ERROR: NOTE:  was open at the time of the failure
[2012-01-09 11:11:53,501] ERROR: At:
[2012-01-09 11:11:53,501] ERROR: NOTE: C:\FlexRAID-Managed-Pool\class1_0\{551e1e79-314f-11e1-844c-005056c00008}\000886.flxr was open at the time of the failure
[2012-01-09 11:11:53,501] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e79-314f-11e1-844c-005056c00008}\000886.flxr
[2012-01-09 11:11:53,501] ERROR: INFO: healthy file => C:\FlexRAID-Managed-Pool\class1_0\{551e1e79-314f-11e1-844c-005056c00008}\000885.flxr
[2012-01-09 11:11:53,501] ERROR: ***Failure details for parity: 1***
[2012-01-09 11:11:53,501] ERROR: Buffer size: 1048576 - Number of failed bytes in buffer: 31
[2012-01-09 11:11:53,501] ERROR: First failure at position: 950480417487 - Last failure at position: 950480424577
[2012-01-09 11:11:53,507] INFO : Verify failure: operation aborted!

Don't get me wrong.... :-) ... but what is the point in validating and verifying the snapshot raid on a regular basis... if it's fails most of the time ?

Secondly... sometimes if i change a file (lets say a batch script) through a mapped network drive on a flexraid snapshot volume using notepad.... after saving the file the content of the file is NOT changed and is still the same as before i started to edit that file! what could be possible reasons for that ?

also sickbeard processing (mapped network drive) is NOT working (massive unpacking and moving of files).

Please give me some insights ;-)

dere

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Snapshot update failed
« Reply #5 on: January 10, 2012, 07:53:34 am »
OK... i run a validate job every friday morning... and a verify every monday morning....
....

Don't get me wrong.... :-) ... but what is the point in validating and verifying the snapshot raid on a regular basis... if it's fails most of the time ?

Secondly... sometimes if i change a file (lets say a batch script) through a mapped network drive on a flexraid snapshot volume using notepad.... after saving the file the content of the file is NOT changed and is still the same as before i started to edit that file! what could be possible reasons for that ?

also sickbeard processing (mapped network drive) is NOT working (massive unpacking and moving of files).

Please give me some insights ;-)

dere

1. In Snapshot RAID, you must always sync the RAID before validating or verifying. And you should do that during the time when nothing else will be modifying something in the RAID. No point otherwise.
I run Validate like once a month and Verify rarely on my production box. See the wiki for the difference between the two.
If Validate takes too long, you can validate sub-set of files by playing with the ValidateSet property in the UI.

2. All other issues, file bug report.

Offline derelict

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Snapshot update failed
« Reply #6 on: January 10, 2012, 08:51:22 am »
thanks for the info.... i didn't knew, that a sync is "required" before the validation/verification..... i'll give it a try then!

I'll create a bug report, as soon as this happens again.

Thanks a lot!

Offline adoucette

  • Jr. Member
  • **
  • Posts: 83
  • Karma: +1/-0
    • View Profile
Re: Snapshot update failed
« Reply #7 on: January 22, 2012, 06:12:49 pm »
1. In Snapshot RAID, you must always sync the RAID before validating or verifying. And you should do that during the time when nothing else will be modifying something in the RAID. No point otherwise.
What's the best way to make sure that nothing else will be changing files on the pool at that time? Is there something that can be done through the FR scheduler to prevent writing/changing files on the pool before an update validate or verify and then switch it back to allowing writing/changing files on the pool once the update/validate/verify process completes?
LIAN LI PC-Q25B Case (5 hot-swap, 3 fixed) | ASUS E35M1-I Motherboard (6 SATA III ports on a Mini-ITX) | AMD Hudson M1 | WHS 2011 --- 1W sleep | 20W idle | 50W r/w

Offline adoucette

  • Jr. Member
  • **
  • Posts: 83
  • Karma: +1/-0
    • View Profile
Re: Snapshot update failed
« Reply #8 on: February 22, 2012, 08:55:07 pm »
BUMP
Is there a preferred way to take the FR volume offline during the Update or Verify process (just so that no changes can be made to files or folders)?
Right now I'm using Windows task scheduler to run a bat file with "wmic path win32_networkadapter where index=7 call disable" to disable the NIC on the server running FlexRaid. Is there another (or better) way to take the FR offline during Update and Verify?
THanks,
Ari
LIAN LI PC-Q25B Case (5 hot-swap, 3 fixed) | ASUS E35M1-I Motherboard (6 SATA III ports on a Mini-ITX) | AMD Hudson M1 | WHS 2011 --- 1W sleep | 20W idle | 50W r/w

Offline waffles

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: Snapshot update failed
« Reply #9 on: January 15, 2014, 11:59:18 am »
1. In Snapshot RAID, you must always sync the RAID before validating or verifying. And you should do that during the time when nothing else will be modifying something in the RAID. No point otherwise.
I run Validate like once a month and Verify rarely on my production box. See the wiki for the difference between the two.
If Validate takes too long, you can validate sub-set of files by playing with the ValidateSet property in the UI.
....

I found this thread because I keep getting this error message "HANDLE_VALUE error=32" during scheduled Validation runs.

Brahim, I did not find a setting for 'sync', so I wonder if you actually mean 'UPDATE', correct?

Re: 'always sync before validating'. From what I understand from above, validating does not include a sync (update?) step. Therefore it appears that I would have to initiate them separately. Also, the validation step should be done right after syncing to avoid that data has changed since. 
I want to run an automated schedule, so how do I schedule the second / validating step, if I do not know how long the first/syncing step takes?

(all that said, I think I must be missing something... requiring 2 steps for a single process (validation) would be very cumbersome and I would expect that Brahim has defined a process that combines these 2 steps into one.)

I am looking for your guidance.

Thanks!