Task 15827737

Name	hadcm3n_4e5q_1980_40_008390320_0
Workunit	8541179
Created	4 Jun 2013, 3:45:39 UTC
Sent	4 Jun 2013, 21:14:56 UTC
Report deadline	4 Sep 2013, 4:42:07 UTC
Received	23 Sep 2013, 16:00:49 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1060699
Run time	5 days 1 hours 10 min 31 sec
CPU time	4 days 5 hours 35 min 24 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.40 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:00:21 (5436): No heartbeat from core client for 30 sec - exiting 17:00:22 (5436): No heartbeat from core client for 30 sec - exiting 17:00:23 (5436): No heartbeat from core client for 30 sec - exiting 17:00:24 (5436): No heartbeat from core client for 30 sec - exiting 17:00:25 (5436): No heartbeat from core client for 30 sec - exiting 17:00:26 (5436): No heartbeat from core client for 30 sec - exiting 17:00:27 (5436): No heartbeat from core client for 30 sec - exiting 17:00:28 (5436): No heartbeat from core client for 30 sec - exiting 17:00:29 (5436): No heartbeat from core client for 30 sec - exiting 17:00:30 (5436): No heartbeat from core client for 30 sec - exiting 17:00:31 (5436): No heartbeat from core client for 30 sec - exiting 17:00:32 (5436): No heartbeat from core client for 30 sec - exiting 17:00:33 (5436): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7698B727 Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 06/08/13 17:00:34 Install Directory : C:\Program Files (x86)\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : LoadLibraryA( C:\Program Files (x86)\BOINC\\dbghelp.dll ): GetLastError = 1114 LoadLibraryA( dbghelp.dll ): GetLastError = 1114 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 0, Write: 0, Other 0 - I/O Transfers Counters - Read: 0, Write: 0, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0 QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0 - Virtual Memory Usage - VirtualSize: 0, PeakVirtualSize: 0 - Pagefile Usage - PagefileUsage: 0, PeakPagefileUsage: 0 - Working Set Size - WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0 * Dump of thread ID 712 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7698B727 * Dump of thread ID 588 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... CPDN Monitor - No 'heartbeat' from BOINC... 17:01:19 (4196): No heartbeat from core client for 30 sec - exiting 17:01:20 (4196): No heartbeat from core client for 30 sec - exiting 17:01:21 (4196): No heartbeat from core client for 30 sec - exiting 17:01:22 (4196): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7698B727 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 06/08/13 17:01:23 Install Directory : C:\Program Files (x86)\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : Loaded Library : C:\Program Files (x86)\BOINC\\dbghelp.dll Loaded Library : C:\Program Files (x86)\BOINC\\symsrv.dll Loaded Library : C:\Program Files (x86)\BOINC\\srcsrv.dll LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7698B727 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 06/08/13 17:02:03 Install Directory : C:\Program Files (x86)\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : LoadLibraryA( C:\Program Files (x86)\BOINC\\dbghelp.dll ): GetLastError = 1114 LoadLibraryA( dbghelp.dll ): GetLastError = 1114 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 0, Write: 0, Other 0 - I/O Transfers Counters - Read: 0, Write: 0, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0 QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0 - Virtual Memory Usage - VirtualSize: 0, PeakVirtualSize: 0 - Pagefile Usage - PagefileUsage: 0, PeakPagefileUsage: 0 - Working Set Size - WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0 * Dump of thread ID 3856 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7698B727 * Dump of thread ID 5604 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4684, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 21:01:13 (5544): No heartbeat from core client for 30 sec - exiting 21:01:14 (5544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:33:51 (1296): No heartbeat from core client for 30 sec - exiting 18:33:52 (1296): No heartbeat from core client for 30 sec - exiting 18:33:53 (1296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:24:49 (4196): No heartbeat from core client for 30 sec - exiting 21:24:50 (4196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:43:01 (4156): No heartbeat from core client for 30 sec - exiting 14:43:02 (4156): No heartbeat from core client for 30 sec - exiting 14:43:03 (4156): No heartbeat from core client for 30 sec - exiting 14:43:04 (4156): No heartbeat from core client for 30 sec - exiting 14:43:05 (4156): No heartbeat from core client for 30 sec - exiting 14:43:06 (4156): No heartbeat from core client for 30 sec - exiting 14:43:07 (4156): No heartbeat from core client for 30 sec - exiting 14:43:08 (4156): No heartbeat from core client for 30 sec - exiting 14:43:09 (4156): No heartbeat from core client for 30 sec - exiting 14:43:10 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:44:13 (8984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Sep 2013 16:04:51	1060699	15827737	hadcm3n_4e5q_1980_40_008390320_0	259,200	365,723	1.4110
12 Sep 2013 21:38:26	1060699	15827737	hadcm3n_4e5q_1980_40_008390320_0	233,280	329,021	1.4104
12 Sep 2013 09:14:36	1060699	15827737	hadcm3n_4e5q_1980_40_008390320_0	207,360	293,253	1.4142
07 Sep 2013 13:45:18	1060699	15827737	hadcm3n_4e5q_1980_40_008390320_0	181,440	256,475	1.4136
25 Aug 2013 23:36:10	1060699	15827737	hadcm3n_4e5q_1980_40_008390320_0	155,520	219,900	1.4140
03 Jul 2013 10:08:28	1060699	15827737	hadcm3n_4e5q_1980_40_008390320_0	129,600	183,345	1.4147
02 Jul 2013 10:59:33	1060699	15827737	hadcm3n_4e5q_1980_40_008390320_0	103,680	146,331	1.4114
28 Jun 2013 12:10:01	1060699	15827737	hadcm3n_4e5q_1980_40_008390320_0	77,760	109,165	1.4039
18 Jun 2013 16:41:14	1060699	15827737	hadcm3n_4e5q_1980_40_008390320_0	51,840	72,170	1.3922
08 Jun 2013 10:09:13	1060699	15827737	hadcm3n_4e5q_1980_40_008390320_0	25,920	35,940	1.3866