Task 13748923

Name	hadcm3n_yisc_1940_40_007614335_1
Workunit	7792465
Created	8 Dec 2011, 0:48:23 UTC
Sent	8 Dec 2011, 20:10:12 UTC
Report deadline	9 Mar 2012, 3:37:23 UTC
Received	14 Jan 2012, 5:17:17 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1105901
Run time	18 days 16 hours 35 min 16 sec
CPU time	11 days 21 hours 6 min 25 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.12 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 13:50:42 (3536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:14:54 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:17:08 (2628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:17:45 (5376): No heartbeat from core client for 30 sec - exiting 12:17:46 (5376): No heartbeat from core client for 30 sec - exiting 12:17:47 (5376): No heartbeat from core client for 30 sec - exiting 12:17:48 (5376): No heartbeat from core client for 30 sec - exiting 12:17:49 (5376): No heartbeat from core client for 30 sec - exiting 12:17:50 (5376): No heartbeat from core client for 30 sec - exiting 12:17:51 (5376): No heartbeat from core client for 30 sec - exiting 12:17:52 (5376): No heartbeat from core client for 30 sec - exiting 12:17:53 (5376): No heartbeat from core client for 30 sec - exiting 12:17:54 (5376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:26:28 (5516): No heartbeat from core client for 30 sec - exiting 14:26:29 (5516): No heartbeat from core client for 30 sec - exiting 14:26:30 (5516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:27:21 (5584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:28:09 (5708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:34:36 (5924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:44:03 (5912): No heartbeat from core client for 30 sec - exiting 16:44:04 (5912): No heartbeat from core client for 30 sec - exiting 16:44:05 (5912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5852, iMonCtr=1 Model crash detected, will try to restart... 13:17:48 (5956): No heartbeat from core client for 30 sec - exiting 13:17:49 (5956): No heartbeat from core client for 30 sec - exiting 13:17:50 (5956): No heartbeat from core client for 30 sec - exiting 13:17:51 (5956): No heartbeat from core client for 30 sec - exiting 13:17:52 (5956): No heartbeat from core client for 30 sec - exiting 13:17:53 (5956): No heartbeat from core client for 30 sec - exiting 13:17:54 (5956): No heartbeat from core client for 30 sec - exiting 13:17:55 (5956): No heartbeat from core client for 30 sec - exiting 13:17:56 (5956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:18:33 (4240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5720, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4604, iMonCtr=1 Model crash detected, will try to restart... 13:44:22 (5748): No heartbeat from core client for 30 sec - exiting 13:44:23 (5748): No heartbeat from core client for 30 sec - exiting 13:44:24 (5748): No heartbeat from core client for 30 sec - exiting 13:44:25 (5748): No heartbeat from core client for 30 sec - exiting 13:44:27 (5748): No heartbeat from core client for 30 sec - exiting 13:44:28 (5748): No heartbeat from core client for 30 sec - exiting 13:44:29 (5748): No heartbeat from core client for 30 sec - exiting 13:44:30 (5748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:19:09 (4804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6016, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6016, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6016, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6016, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6016, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6016, iMonCtr=1 Model crash detected, will try to restart... 17:22:44 (6016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:46:58 (3984): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 17:48:49 (7784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:48:51 (7784): No heartbeat from core client for 30 sec - exiting 17:49:49 (8492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:50:45 (9192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:50:47 (9192): No heartbeat from core client for 30 sec - exiting 20:40:11 (12612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:41:28 (15332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15308, iMonCtr=1 Model crash detected, will try to restart... 13:52:23 (5240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6356, iMonCtr=1 Model crash detected, will try to restart... CSuspended CPDN Monitor - Suspend request from BOINC... 10:58:43 (5940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 17:58:39 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:40 (5428): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5736, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5032, iMonCtr=1 Model crash detected, will try to restart... CCPDN Monitor - Quit request from BOINC... 09:58:34 (5948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=164, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:06:45 (4568): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:59:58 (7328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77A03232 read attempt to address 0x40DE152F Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_yisc_1940_40_007614335/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Jan 2012 05:21:56	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	518,400	1,026,382	1.9799
12 Jan 2012 06:17:32	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	492,480	969,979	1.9696
11 Jan 2012 05:03:06	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	466,560	910,343	1.9512
10 Jan 2012 05:17:35	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	440,640	853,153	1.9362
09 Jan 2012 04:02:10	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	414,720	793,625	1.9136
08 Jan 2012 03:23:49	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	388,800	734,453	1.8890
04 Jan 2012 08:41:11	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	362,880	677,915	1.8682
02 Jan 2012 05:19:05	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	336,960	620,941	1.8428
01 Jan 2012 02:16:02	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	311,040	564,235	1.8140
28 Dec 2011 04:13:50	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	285,120	507,690	1.7806
26 Dec 2011 06:29:44	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	259,200	451,600	1.7423
25 Dec 2011 12:54:45	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	233,280	395,619	1.6959
24 Dec 2011 17:50:31	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	207,360	340,738	1.6432
23 Dec 2011 20:27:09	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	181,440	283,632	1.5632
22 Dec 2011 22:38:06	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	155,520	226,464	1.4562
20 Dec 2011 03:56:42	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	129,600	170,264	1.3138
18 Dec 2011 23:45:35	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	103,680	198,173	1.9114
17 Dec 2011 21:09:47	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	77,760	169,098	2.1746
13 Dec 2011 04:34:28	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	51,840	112,431	2.1688
10 Dec 2011 06:20:33	1105901	13748923	hadcm3n_yisc_1940_40_007614335_1	25,920	56,002	2.1606