Task 11825500

Name	famous_vl3j_799_200_006713109_0
Workunit	6916362
Created	26 Aug 2010, 17:34:23 UTC
Sent	3 Nov 2010, 4:41:04 UTC
Report deadline	2 Feb 2011, 12:08:15 UTC
Received	4 Nov 2010, 0:40:41 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	56705
Run time	16 hours 8 min 52 sec
CPU time	11 hours 45 min 11 sec
Validate state	Invalid
Credit	92.73
Device peak FLOPS	0.85 GFLOPS
Application version	UK Met Office FAMOUS v6.11 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit reWorker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1108, selfPID=1108, iMonCtr=1 07:09:00 (580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 08:25:02 (3192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:26:28 (3100): Can't acquire lockfile (32) - waiting 35s 10:27:07 (3100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:31:11 (2872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:41:12 (272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:24:37 (988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:26:05 (3712): Can't acquire lockfile (32) - waiting 35s BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 14:10:45 (3712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:10:46 (3712): No heartbeat from core client for 30 sec - exiting 14:10:47 (3712): No heartbeat from core client for 30 sec - exiting 14:10:48 (3712): No heartbeat from core client for 30 sec - exiting 14:10:49 (3712): No heartbeat from core client for 30 sec - exiting 14:10:50 (3712): No heartbeat from core client for 30 sec - exiting 14:10:52 (3712): No heartbeat from core client for 30 sec - exiting 14:10:53 (3712): No heartbeat from core client for 30 sec - exiting 14:10:54 (3712): No heartbeat from core client for 30 sec - exiting 14:10:55 (3712): No heartbeat from core client for 30 sec - exiting 14:10:56 (3712): No heartbeat from core client for 30 sec - exiting 14:12:12 (208): Can't acquire lockfile (32) - waiting 35s 14:12:47 (208): Can't acquire lockfile (32) - exiting 14:12:47 (208): Error: The process cannot access the file because it is being used by another process. (0x20) 16:15:59 (2624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:16:00 (2624): No heartbeat from core client for 30 sec - exiting 16:16:01 (2624): No heartbeat from core client for 30 sec - exiting 16:16:02 (2624): No heartbeat from core client for 30 sec - exiting 16:16:03 (2624): No heartbeat from core client for 30 sec - exiting 16:16:04 (2624): No heartbeat from core client for 30 sec - exiting 16:16:06 (2624): No heartbeat from core client for 30 sec - exiting 16:16:07 (2624): No heartbeat from core client for 30 sec - exiting 16:16:08 (2624): No heartbeat from core client for 30 sec - exiting 16:16:09 (2624): No heartbeat from core client for 30 sec - exiting 16:16:10 (2624): No heartbeat from core client for 30 sec - exiting 16:17:26 (3640): Can't acquire lockfile (32) - waiting 35s 16:18:01 (3640): Can't acquire lockfile (32) - exiting 16:18:01 (3640): Error: The process cannot access the file because it is being used by another process. (0x20) 18:17:41 (2636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:19:08 (1216): Can't acquire lockfile (32) - waiting 35s 20:31:05 (1216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 6.11.1 Dump Timestamp : 11/03/10 20:34:35 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 1114 LoadLibraryA( dbghelp.dll ): GetLastError = 1114 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 254, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 34748, QuotaPeakPagedPoolUsage: 34748 QuotaNonPagedPoolUsage: 1912, QuotaPeakNonPagedPoolUsage: 1912 - Virtual Memory Usage - VirtualSize: 873103360, PeakVirtualSize: 873103360 - Pagefile Usage - PagefileUsage: 40194048, PeakPagefileUsage: 52645888 - Working Set Size - WorkingSetSize: 2060288, PeakWorkingSetSize: 2060288, PageFaultCount: 502 * Dump of thread ID 2772 (state: Ready): * - Information - Status: Base Priority: Above Normal, Priority: Above Normal, , Kernel Time: 468750.000000, User Time: 156250.000000, Wait Time: 120107928.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB * Dump of thread ID 148 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 120107928.000000 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=168, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.11.1 Dump Timestamp : 11/03/10 20:34:37 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 1114 LoadLibraryA( dbghelp.dll ): GetLastError = 1114 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 37, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 34748, QuotaPeakPagedPoolUsage: 34748 QuotaNonPagedPoolUsage: 1912, QuotaPeakNonPagedPoolUsage: 1912 - Virtual Memory Usage - VirtualSize: 873103360, PeakVirtualSize: 873103360 - Pagefile Usage - PagefileUsage: 40194048, PeakPagefileUsage: 52645888 - Working Set Size - WorkingSetSize: 2060288, PeakWorkingSetSize: 2060288, PageFaultCount: 502 * Dump of thread ID 1076 (state: Ready): * - Information - Status: Base Priority: Above Normal, Priority: Above Normal, , Kernel Time: 0.000000, User Time: 156250.000000, Wait Time: 120108056.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB * Dump of thread ID 2468 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 120108056.000000 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=168, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.11.1 Dump Timestamp : 11/03/10 20:34:39 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 1114 LoadLibraryA( dbghelp.dll ): GetLastError = 1114 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 37, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 34748, QuotaPeakPagedPoolUsage: 34748 QuotaNonPagedPoolUsage: 1912, QuotaPeakNonPagedPoolUsage: 1912 - Virtual Memory Usage - VirtualSize: 873103360, PeakVirtualSize: 873103360 - Pagefile Usage - PagefileUsage: 40194048, PeakPagefileUsage: 52645888 - Working Set Size - WorkingSetSize: 2060288, PeakWorkingSetSize: 2060288, PageFaultCount: 502 * Dump of thread ID 940 (state: Ready): * - Information - Status: Base Priority: Above Normal, Priority: Above Normal, , Kernel Time: 156250.000000, User Time: 0.000000, Wait Time: 120108200.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB * Dump of thread ID 2944 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 120108200.000000 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=168, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.11.1 Dump Timestamp : 11/03/10 20:34:43 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 1114 LoadLibraryA( dbghelp.dll ): GetLastError = 1114 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 37, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 34748, QuotaPeakPagedPoolUsage: 34748 QuotaNonPagedPoolUsage: 1912, QuotaPeakNonPagedPoolUsage: 1912 - Virtual Memory Usage - VirtualSize: 873103360, PeakVirtualSize: 873103360 - Pagefile Usage - PagefileUsage: 40194048, PeakPagefileUsage: 52645888 - Working Set Size - WorkingSetSize: 2060288, PeakWorkingSetSize: 2060288, PageFaultCount: 502 * Dump of thread ID 3852 (state: Ready): * - Information - Status: Base Priority: Above Normal, Priority: Above Normal, , Kernel Time: 468750.000000, User Time: 0.000000, Wait Time: 120108456.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB * Dump of thread ID 2864 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 120108448.000000 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=168, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.11.1 Dump Timestamp : 11/03/10 20:34:48 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 1114 LoadLibraryA( dbghelp.dll ): GetLastError = 1114 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 37, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 34748, QuotaPeakPagedPoolUsage: 34748 QuotaNonPagedPoolUsage: 1912, QuotaPeakNonPagedPoolUsage: 1912 - Virtual Memory Usage - VirtualSize: 873103360, PeakVirtualSize: 873103360 - Pagefile Usage - PagefileUsage: 40194048, PeakPagefileUsage: 52645888 - Working Set Size - WorkingSetSize: 2060288, PeakWorkingSetSize: 2060288, PageFaultCount: 502 * Dump of thread ID 1996 (state: Ready): * - Information - Status: Base Priority: Above Normal, Priority: Above Normal, , Kernel Time: 0.000000, User Time: 156250.000000, Wait Time: 120108792.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB * Dump of thread ID 3188 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 120108792.000000 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=168, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.11.1 Dump Timestamp : 11/03/10 20:34:52 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 1114 LoadLibraryA( dbghelp.dll ): GetLastError = 1114 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 37, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 34748, QuotaPeakPagedPoolUsage: 34748 QuotaNonPagedPoolUsage: 1912, QuotaPeakNonPagedPoolUsage: 1912 - Virtual Memory Usage - VirtualSize: 873103360, PeakVirtualSize: 873103360 - Pagefile Usage - PagefileUsage: 40194048, PeakPagefileUsage: 52645888 - Working Set Size - WorkingSetSize: 2060288, PeakWorkingSetSize: 2060288, PageFaultCount: 502 * Dump of thread ID 3732 (state: Ready): * - Information - Status: Base Priority: Above Normal, Priority: Above Normal, , Kernel Time: 0.000000, User Time: 156250.000000, Wait Time: 120109016.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB * Dump of thread ID 2072 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 120109016.000000 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=168, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( 20:35:07 (168): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Nov 2010 00:34:21	56705	11825500	famous_vl3j_799_200_006713109_0	28,106	42,329	1.5060
03 Nov 2010 17:28:04	56705	11825500	famous_vl3j_799_200_006713109_0	18,746	28,446	1.5174
03 Nov 2010 11:09:30	56705	11825500	famous_vl3j_799_200_006713109_0	9,386	14,239	1.5170