Task 15827111

Name	hadcm3n_3h2j_1980_40_008389879_0
Workunit	8540738
Created	4 Jun 2013, 1:47:12 UTC
Sent	5 Jun 2013, 0:29:33 UTC
Report deadline	4 Sep 2013, 7:56:44 UTC
Received	14 Aug 2013, 19:21:14 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1173689
Run time	7 days 1 hours 29 min 22 sec
CPU time	6 days 15 hours 50 min 3 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.84 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> 23:07:32 (2080): No heartbeat from core client for 30 sec - exiting 23:07:33 (2080): No heartbeat from core client for 30 sec - exiting 23:07:34 (2080): No heartbeat from core client for 30 sec - exiting 23:07:35 (2080): No heartbeat from core client for 30 sec - exiting 23:07:36 (2080): No heartbeat from core client for 30 sec - exiting 23:07:37 (2080): No heartbeat from core client for 30 sec - exiting 23:07:38 (2080): No heartbeat from core client for 30 sec - exiting 23:07:39 (2080): No heartbeat from core client for 30 sec - exiting 23:07:40 (2080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:10:17 (6004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:37:40 (5460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:32:15 (5340): No heartbeat from core client for 30 sec - exiting 10:32:16 (5340): No heartbeat from core client for 30 sec - exiting 10:32:17 (5340): No heartbeat from core client for 30 sec - exiting 10:32:18 (5340): No heartbeat from core client for 30 sec - exiting 10:32:19 (5340): No heartbeat from core client for 30 sec - exiting 10:32:20 (5340): No heartbeat from core client for 30 sec - exiting 10:32:21 (5340): No heartbeat from core client for 30 sec - exiting 10:32:22 (5340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:49:16 (7440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9756, iMonCtr=1 Model crash detected, will try to restart... 09:39:50 (7896): No heartbeat from core client for 30 sec - exiting 09:39:51 (7896): No heartbeat from core client for 30 sec - exiting 09:39:52 (7896): No heartbeat from core client for 30 sec - exiting 09:39:53 (7896): No heartbeat from core client for 30 sec - exiting 09:39:54 (7896): No heartbeat from core client for 30 sec - exiting 09:39:55 (7896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:28:31 (4672): No heartbeat from core client for 30 sec - exiting 08:28:32 (4672): No heartbeat from core client for 30 sec - exiting 08:28:33 (4672): No heartbeat from core client for 30 sec - exiting 08:28:34 (4672): No heartbeat from core client for 30 sec - exiting 08:28:35 (4672): No heartbeat from core client for 30 sec - exiting 08:28:36 (4672): No heartbeat from core client for 30 sec - exiting 08:28:37 (4672): No heartbeat from core client for 30 sec - exiting 08:28:38 (4672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:31:47 (5288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:25:36 (4836): No heartbeat from core client for 30 sec - exiting 20:25:37 (4836): No heartbeat from core client for 30 sec - exiting 20:25:38 (4836): No heartbeat from core client for 30 sec - exiting 20:25:39 (4836): No heartbeat from core client for 30 sec - exiting 20:25:40 (4836): No heartbeat from core client for 30 sec - exiting 20:25:41 (4836): No heartbeat from core client for 30 sec - exiting 20:25:42 (4836): No heartbeat from core client for 30 sec - exiting 20:25:43 (4836): No heartbeat from core client for 30 sec - exiting 20:25:44 (4836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:17:54 (4012): No heartbeat from core client for 30 sec - exiting 22:17:55 (4012): No heartbeat from core client for 30 sec - exiting 22:17:56 (4012): No heartbeat from core client for 30 sec - exiting 22:17:57 (4012): No heartbeat from core client for 30 sec - exiting 22:17:58 (4012): No heartbeat from core client for 30 sec - exiting 22:17:59 (4012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:16:32 (5336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:39:14 (1644): No heartbeat from core client for 30 sec - exiting 11:39:16 (1644): No heartbeat from core client for 30 sec - exiting 11:39:17 (1644): No heartbeat from core client for 30 sec - exiting 11:39:18 (1644): No heartbeat from core client for 30 sec - exiting 11:39:19 (1644): No heartbeat from core client for 30 sec - exiting 11:39:20 (1644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:42:06 (5108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:26:49 (4612): No heartbeat from core client for 30 sec - exiting 19:26:50 (4612): No heartbeat from core client for 30 sec - exiting 19:26:51 (4612): No heartbeat from core client for 30 sec - exiting 19:26:52 (4612): No heartbeat from core client for 30 sec - exiting 19:26:53 (4612): No heartbeat from core client for 30 sec - exiting 19:26:54 (4612): No heartbeat from core client for 30 sec - exiting 19:26:55 (4612): No heartbeat from core client for 30 sec - exiting 19:26:56 (4612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:24:10 (5616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:26:53 (6924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5976, iMonCtr=1 Model crash detected, will try to restart... 08:32:33 (5324): No heartbeat from core client for 30 sec - exiting 08:32:34 (5324): No heartbeat from core client for 30 sec - exiting 08:32:35 (5324): No heartbeat from core client for 30 sec - exiting 08:32:36 (5324): No heartbeat from core client for 30 sec - exiting 08:32:37 (5324): No heartbeat from core client for 30 sec - exiting 08:32:38 (5324): No heartbeat from core client for 30 sec - exiting 08:32:39 (5324): No heartbeat from core client for 30 sec - exiting 08:32:40 (5324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1360, iMonCtr=1 Model crash detected, will try to restart... 20:39:14 (6096): No heartbeat from core client for 30 sec - exiting 20:39:15 (6096): No heartbeat from core client for 30 sec - exiting 20:39:16 (6096): No heartbeat from core client for 30 sec - exiting 20:39:17 (6096): No heartbeat from core client for 30 sec - exiting 20:39:18 (6096): No heartbeat from core client for 30 sec - exiting 20:39:19 (6096): No heartbeat from core client for 30 sec - exiting 20:39:20 (6096): No heartbeat from core client for 30 sec - exiting 20:39:21 (6096): No heartbeat from core client for 30 sec - exiting 20:39:22 (6096): No heartbeat from core client for 30 sec - exiting 20:39:23 (6096): No heartbeat from core client for 30 sec - exiting 20:39:24 (6096): No heartbeat from core client for 30 sec - exiting 20:39:25 (6096): No heartbeat from core client for 30 sec - exiting 20:39:26 (6096): No heartbeat from core client for 30 sec - exiting 20:39:27 (6096): No heartbeat from core client for 30 sec - exiting 20:39:28 (6096): No heartbeat from core client for 30 sec - exiting 20:39:29 (6096): No heartbeat from core client for 30 sec - exiting 20:39:30 (6096): No heartbeat from core client for 30 sec - exiting 20:39:31 (6096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:44:23 (5396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:49:12 (5916): No heartbeat from core client for 30 sec - exiting 21:49:13 (5916): No heartbeat from core client for 30 sec - exiting 21:49:14 (5916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2720, iMonCtr=1 Model crash detected, will try to restart... 19:27:10 (4740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:57:17 (5352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:57:18 (5352): No heartbeat from core client for 30 sec - exiting 21:57:57 (7556): No heartbeat from core client for 30 sec - exiting 21:57:58 (7556): No heartbeat from core client for 30 sec - exiting 21:57:59 (7556): No heartbeat from core client for 30 sec - exiting 21:58:00 (7556): No heartbeat from core client for 30 sec - exiting 21:58:01 (7556): No heartbeat from core client for 30 sec - exiting 21:58:02 (7556): No heartbeat from core client for 30 sec - exiting 21:58:03 (7556): No heartbeat from core client for 30 sec - exiting 21:58:04 (7556): No heartbeat from core client for 30 sec - exiting 21:58:05 (7556): No heartbeat from core client for 30 sec - exiting 21:58:06 (7556): No heartbeat from core client for 30 sec - exiting 21:58:07 (7556): No heartbeat from core client for 30 sec - exiting 21:58:08 (7556): No heartbeat from core client for 30 sec - exiting 21:58:09 (7556): No heartbeat from core client for 30 sec - exiting 21:58:10 (7556): No heartbeat from core client for 30 sec - exiting 21:58:11 (7556): No heartbeat from core client for 30 sec - exiting 21:58:12 (7556): No heartbeat from core client for 30 sec - exiting 21:58:13 (7556): No heartbeat from core client for 30 sec - exiting 21:58:14 (7556): No heartbeat from core client for 30 sec - exiting 21:58:15 (7556): No heartbeat from core client for 30 sec - exiting 21:58:16 (7556): No heartbeat from core client for 30 sec - exiting 21:58:17 (7556): No heartbeat from core client for 30 sec - exiting 21:58:18 (7556): No heartbeat from core client for 30 sec - exiting 21:58:19 (7556): No heartbeat from core client for 30 sec - exiting 21:58:20 (7556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:58:21 (7556): No heartbeat from core client for 30 sec - exiting 21:58:22 (7556): No heartbeat from core client for 30 sec - exiting 21:58:23 (7556): No heartbeat from core client for 30 sec - exiting 21:58:24 (7556): No heartbeat from core client for 30 sec - exiting 21:58:25 (7556): No heartbeat from core client for 30 sec - exiting 21:58:26 (7556): No heartbeat from core client for 30 sec - exiting 21:58:27 (7556): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 23:48:15 (5824): No heartbeat from core client for 30 sec - exiting 23:48:16 (5824): No heartbeat from core client for 30 sec - exiting 23:48:17 (5824): No heartbeat from core client for 30 sec - exiting 23:48:18 (5824): No heartbeat from core client for 30 sec - exiting 23:48:19 (5824): No heartbeat from core client for 30 sec - exiting 23:48:20 (5824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:21:57 (6100): No heartbeat from core client for 30 sec - exiting 23:21:58 (6100): No heartbeat from core client for 30 sec - exiting 23:21:59 (6100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:18:08 (4404): No heartbeat from core client for 30 sec - exiting 22:18:09 (4404): No heartbeat from core client for 30 sec - exiting 22:18:10 (4404): No heartbeat from core client for 30 sec - exiting 22:18:11 (4404): No heartbeat from core client for 30 sec - exiting 22:18:12 (4404): No heartbeat from core client for 30 sec - exiting 22:18:13 (4404): No heartbeat from core client for 30 sec - exiting 22:18:14 (4404): No heartbeat from core client for 30 sec - exiting 22:18:15 (4404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:33:31 (5460): No heartbeat from core client for 30 sec - exiting 23:33:32 (5460): No heartbeat from core client for 30 sec - exiting 23:33:33 (5460): No heartbeat from core client for 30 sec - exiting 23:33:34 (5460): No heartbeat from core client for 30 sec - exiting 23:33:35 (5460): No heartbeat from core client for 30 sec - exiting 23:33:36 (5460): No heartbeat from core client for 30 sec - exiting 23:33:37 (5460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... C17:58:32 (6400): No heartbeat from core client for 30 sec - exiting 17:58:35 (6400): No heartbeat from core client for 30 sec - exiting 17:58:36 (6400): No heartbeat from core client for 30 sec - exiting 17:58:37 (6400): No heartbeat from core client for 30 sec - exiting 17:58:38 (6400): No heartbeat from core client for 30 sec - exiting 17:58:39 (6400): No heartbeat from core client for 30 sec - exiting 17:58:40 (6400): No heartbeat from core client for 30 sec - exiting 17:58:41 (6400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x778771A8 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... 10:10:13 (5700): No heartbeat from core client for 30 sec - exiting 10:10:14 (5700): No heartbeat from core client for 30 sec - exiting 10:10:15 (5700): No heartbeat from core client for 30 sec - exiting 10:10:16 (5700): No heartbeat from core client for 30 sec - exiting 10:10:17 (5700): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77B77373 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... 14:37:24 (1668): No heartbeat from core client for 30 sec - exiting 14:37:25 (1668): No heartbeat from core client for 30 sec - exiting 14:37:27 (1668): No heartbeat from core client for 30 sec - exiting 14:37:28 (1668): No heartbeat from core client for 30 sec - exiting 14:37:29 (1668): No heartbeat from core client for 30 sec - exiting 14:37:30 (1668): No heartbeat from core client for 30 sec - exiting 14:37:31 (1668): No heartbeat from core client for 30 sec - exiting 14:37:32 (1668): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77E87373 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3h2j_1980_40_008389879/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Aug 2013 19:24:10	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	518,400	573,629	1.1065
23 Jul 2013 19:16:29	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	492,480	545,788	1.1082
12 Jul 2013 01:11:58	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	466,560	517,322	1.1088
08 Jul 2013 03:36:13	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	440,640	488,600	1.1088
02 Jul 2013 12:05:05	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	414,720	460,104	1.1094
02 Jul 2013 11:00:05	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	388,800	431,424	1.1096
02 Jul 2013 10:29:44	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	362,880	402,815	1.1101
27 Jun 2013 03:31:44	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	336,960	374,570	1.1116
25 Jun 2013 00:37:10	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	311,040	346,158	1.1129
20 Jun 2013 23:58:38	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	285,120	317,466	1.1134
19 Jun 2013 21:06:28	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	259,200	288,905	1.1146
18 Jun 2013 02:10:54	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	233,280	260,512	1.1167
16 Jun 2013 20:58:51	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	207,360	232,004	1.1188
16 Jun 2013 01:22:40	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	181,440	203,355	1.1208
15 Jun 2013 15:57:56	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	155,520	174,595	1.1227
13 Jun 2013 00:50:47	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	129,600	145,727	1.1244
10 Jun 2013 02:50:20	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	103,680	116,967	1.1282
09 Jun 2013 19:03:37	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	77,760	87,917	1.1306
08 Jun 2013 23:30:59	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	51,840	58,672	1.1318
08 Jun 2013 04:11:45	1173689	15827111	hadcm3n_3h2j_1980_40_008389879_0	25,920	29,455	1.1364