Author Topic: Java IO Exceptions - Updates but no validataion  (Read 1402 times)

Offline ThreeIsTwoTwoIsOneOneIs..

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
  • Small 4 Drive Pool, for now
    • View Profile
Java IO Exceptions - Updates but no validataion
« on: December 09, 2015, 08:07:16 am »
Hi,

Not 100% sure this is 2.1 related, but it seems likely.

Firstly my schedules got wiped, so my array didnt sync for over two weeks before I spotted it. Rescheduled it when I noticed and it started running them, grand.

14-30NOV 2015 No actions run

30th Update = Success
1st   Update = Success

Since that I have had 0 Updates and Verifys that passed. I normally have a certain % of Validates/Verifys that fail due to them taking so long and something else needing to spin back up in the meantime. But never none of them passing.

Troubleshooting begins:
Code: [Select]
Bad Files:
[2015-12-09 12:13:39,750] INFO : [dates-check] started at: Wed Dec 09 12:13:39 GMT 2015
[2015-12-09 12:14:26,291] WARN : File with invalid date: V:\Backups\....\Guitar.jpg => Tue Dec 31 23:00:00 GMT 2047
[2015-12-09 12:15:09,949] WARN : File with invalid date: V:\Backups\....\MEMSTICK.IND => Tue Dec 31 23:00:00 GMT 2097
[2015-12-09 12:15:09,952] WARN : File with invalid date: V:\Backups\....\MSTK_PRO.IND => Tue Dec 31 23:00:00 GMT 2097
DELETE 2, 3. EDIT 1.
Scan again
Code: [Select]
[2015-12-09 12:22:48,145] INFO : [dates-check] started at: Wed Dec 09 12:22:48 GMT 2015
[2015-12-09 12:28:02,859] INFO : [dates-check] completed at: Wed Dec 09 12:28:02 GMT 2015
DATES CLEAN
Update run
Quick Validate.... The validation process is reporting 0 new, 6 changed, 0 deleted, and 0 corrupted files!

Code: [Select]
[2015-12-09 12:34:23,217] WARN : Changed: D:2:0\Data\....\build\web\gfv3ee6.dpf
[2015-12-09 12:34:23,217] WARN : Changed: D:2:0\Data\....\web\WEB-INF\classes\.netbeans_automatic_build
[2015-12-09 12:34:23,217] WARN : Changed: D:2:0\Data\.....\web\WEB-INF\classes\.netbeans_update_resources
[2015-12-09 12:34:28,088] WARN : Changed: D:0:0\Data\...\LOCK
[2015-12-09 12:34:28,088] WARN : Changed: D:0:0\Data\....\LOCK
[2015-12-09 12:35:06,591] WARN : Changed: D:0:0\Data\.....\.lock

Delete locks, remove error files from the array
Update run
Quick Validate... [QUICK-VALIDATE] SUCCESS - emailed...
Validate:
Code: [Select]
[2015-12-09 12:53:06,619] ERROR: Corrupted: xxxxx
Repeating 10x, files ARE KNOWN GOOD

java.io.IOException: リ
at com.tchegbe.lib.common.io.NativeIO.open(Native Method)
at com.tchegbe.lib.common.io.NativeIO.<unknown>(Unknown Source)
at com.tchegbe.lib.common.io.b.d.<unknown>(Unknown Source)
at com.tchegbe.lib.common.io.b.e.<unknown>(Unknown Source)
at com.tchegbe.lib.common.io.d.c.a(Unknown Source)
at com.tchegbe.lib.common.io.F.a(Unknown Source)
at com.tchegbe.lib.common.io.F.a(Unknown Source)
at com.tchegbe.lib.io.a.e.a.k.run(Unknown Source)
at com.tchegbe.lib.common.io.e.m.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
RE RUN =  EXACT SAME.
DISABLE AV
RE RUN = SAME
Reboot again
Remove camera uploads
Run update
Run validate
Hundreds of "corrupt" files
Check dates again
Code: [Select]
[2015-12-09 13:40:00,863] INFO : [dates-check] started at: Wed Dec 09 13:40:00 GMT 2015
[2015-12-09 13:45:16,042] INFO : [dates-check] completed at: Wed Dec 09 13:45:16 GMT 2015
Validate?
Code: [Select]
[2015-12-09 13:49:38,221] ERROR: Corrupted: C:\FlexRAID-Managed-Pool\class1_0\{e2334ccc-910b-44ff-b418-50c7c7a269ee}\....\IMG_7364.JPG
[2015-12-09 13:49:43,795] ERROR: M
java.io.IOException: M
at com.tchegbe.lib.common.io.NativeIO.open(Native Method)
at com.tchegbe.lib.common.io.NativeIO.<unknown>(Unknown Source)
at com.tchegbe.lib.common.io.b.d.<unknown>(Unknown Source)
at com.tchegbe.lib.common.io.b.e.<unknown>(Unknown Source)
at com.tchegbe.lib.common.io.d.c.a(Unknown Source)
at com.tchegbe.lib.common.io.F.a(Unknown Source)
at com.tchegbe.lib.common.io.F.a(Unknown Source)
at com.tchegbe.lib.io.a.e.a.k.run(Unknown Source)
at com.tchegbe.lib.common.io.e.m.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

So where do I go from here? Is it safe to roll back to the previous version?

Thanks,
Three
DS380 - C2750 - WD Reds

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Java IO Exceptions - Updates but no validataion
« Reply #1 on: December 09, 2015, 09:11:34 am »
1. Update your exclusions to exclude more of those files that should not be in a Snapshot array.

2. Set your logs to TRACE if you wish to get more details and whenever you have issues to troubleshoot.

Offline ThreeIsTwoTwoIsOneOneIs..

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
  • Small 4 Drive Pool, for now
    • View Profile
Re: Java IO Exceptions - Updates but no validataion
« Reply #2 on: December 10, 2015, 03:42:55 am »
I've added an exclusion and stopped the array but Validate is still finding corruption within the exclusion path?
DS380 - C2750 - WD Reds

Offline ThreeIsTwoTwoIsOneOneIs..

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
  • Small 4 Drive Pool, for now
    • View Profile
Re: Java IO Exceptions - Updates but no validataion
« Reply #3 on: December 10, 2015, 08:30:10 am »
Code: [Select]
The validation process is reporting 0 new, 0 changed, 0 deleted, and 575 corrupted files!

Name: Validate Process
Start Date: Thu Dec 10 09:41:59 GMT+000 2015
End Date: Thu Dec 10 13:53:21 GMT+000 2015
Duration: 04:11:21
Throughput: 432.88 MB/s

I'm pretty sure the files are good, it'll take a while to pull them over my current connection so I can test, but assuming they are should I go ahead and touch all of them?
DS380 - C2750 - WD Reds

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Java IO Exceptions - Updates but no validataion
« Reply #4 on: December 10, 2015, 09:20:12 am »
If the files are good, then yeah, touch them.

Offline ThreeIsTwoTwoIsOneOneIs..

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
  • Small 4 Drive Pool, for now
    • View Profile
Re: Java IO Exceptions - Updates but no validataion
« Reply #5 on: December 11, 2015, 05:47:53 am »
Touch, update, same files come back corrupt....

EDIT: Could a temporarily incorrect sys time cause parity files to be out of whack?
« Last Edit: December 11, 2015, 07:11:22 am by ThreeIsTwoTwoIsOneOneIs.. »
DS380 - C2750 - WD Reds

Offline Brahim

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8,547
  • Karma: +204/-16
    • View Profile
Re: Java IO Exceptions - Updates but no validataion
« Reply #6 on: December 11, 2015, 10:28:47 am »
Make sure you ran the Update task after touching.

Offline ThreeIsTwoTwoIsOneOneIs..

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
  • Small 4 Drive Pool, for now
    • View Profile
Re: Java IO Exceptions - Updates but no validataion
« Reply #7 on: December 25, 2015, 02:59:46 pm »
Ok, bit of a hiatus between posts, that time of year.


Corruption issues all over the place. Peaked previously with 575 files detected corrupt. I manually checked some, and some with FFMPEG looking for errors, all clean. Copy out of array, update, copy back to array. Corrupt count reduced.

Then did some more, should have been 10-15 files left detected as corrupt (and seemingly false positive). Next validate throws a few hundred again. So I gave up and force sync verify'd two days ago:
Code: [Select]
270500102144
1823131828224 [bytes out of]

Next update task seemed to crash out, it got stuck using 1.7GB of mem allocated. Eventually aborted it.

At this point I think I discovered a "bug", in that an exclude on *\directory and *\directory\ where the latter won't work? Is that right?

In any case, the latest validate:

Code: [Select]
The validation process is reporting 0 new, 0 changed, 0 deleted, and 1803 corrupted files!
I can only conclude theres either a serious software issue or the parity disk is totally knackered. I'm going to shtudown the system and scan the parity drive overnight to clear the parity disk as likely ok, but really I think a downgrade is probably the route I should take.

--

On a separate note, why is it that Flexraid seems to not play nice with operations that write files out rapidly?

Dropbox: Rapid changes cause conflicted copies only on Flex, its the only one of 5 windows clients in the group that does this, doesnt happen on a regular drive
Cryptocurrenty clients: Crash/get stuck after a certain number of blocks
uTorrent: Intensive multifile writes causes a missing file detection vs "Disk overloaded" that should be expected
Plex Server: Creating metadata is very hit and miss where some library art never downloads when the data directory is pointed to flex instead of the OS drive.

I dont quite undsertand why Flex acting as an intermediary would cause this, thought maybe it was the buffer but even at 50MB its still the same.


EDIT: Just to add, running trace through non specific JavaIO errors and made readability an issue
« Last Edit: December 25, 2015, 03:01:40 pm by ThreeIsTwoTwoIsOneOneIs.. »
DS380 - C2750 - WD Reds