Task 16160899

Name	hadcm3n_833q_1980_40_008461961_3
Workunit	8612817
Created	25 Dec 2013, 9:28:05 UTC
Sent	25 Dec 2013, 9:28:10 UTC
Report deadline	26 Mar 2014, 16:55:21 UTC
Received	11 Mar 2014, 1:40:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1307977
Run time	34 days 9 hours 58 min 27 sec
CPU time	25 days 12 hours 42 min 6 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.42 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.39</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> 09:55:38 (5620): No heartbeat from core client for 30 sec - exiting 09:55:39 (5620): No heartbeat from core client for 30 sec - exiting 09:55:40 (5620): No heartbeat from core client for 30 sec - exiting 09:55:42 (5620): No heartbeat from core client for 30 sec - exiting 09:55:43 (5620): No heartbeat from core client for 30 sec - exiting 09:55:44 (5620): No heartbeat from core client for 30 sec - exiting 09:55:45 (5620): No heartbeat from core client for 30 sec - exiting 09:55:46 (5620): No heartbeat from core client for 30 sec - exiting 09:55:47 (5620): No heartbeat from core client for 30 sec - exiting 09:55:48 (5620): No heartbeat from core client for 30 sec - exiting 09:55:49 (5620): No heartbeat from core client for 30 sec - exiting 09:55:51 (5620): No heartbeat from core client for 30 sec - exiting 09:55:52 (5620): No heartbeat from core client for 30 sec - exiting 09:55:53 (5620): No heartbeat from core client for 30 sec - exiting 09:55:54 (5620): No heartbeat from core client for 30 sec - exiting 09:55:55 (5620): No heartbeat from core client for 30 sec - exiting 09:55:56 (5620): No heartbeat from core client for 30 sec - exiting 09:55:57 (5620): No heartbeat from core client for 30 sec - exiting 09:55:58 (5620): No heartbeat from core client for 30 sec - exiting 09:55:59 (5620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:57:38 (3668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:17:44 (5520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:19:42 (3568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:40:31 (4052): No heartbeat from core client for 30 sec - exiting 10:40:33 (4052): No heartbeat from core client for 30 sec - exiting 10:40:34 (4052): No heartbeat from core client for 30 sec - exiting 10:40:35 (4052): No heartbeat from core client for 30 sec - exiting 10:40:36 (4052): No heartbeat from core client for 30 sec - exiting 10:40:37 (4052): No heartbeat from core client for 30 sec - exiting 10:40:38 (4052): No heartbeat from core client for 30 sec - exiting 10:40:39 (4052): No heartbeat from core client for 30 sec - exiting 10:40:40 (4052): No heartbeat from core client for 30 sec - exiting 10:40:41 (4052): No heartbeat from core client for 30 sec - exiting 10:40:42 (4052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:59:50 (4320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:29:46 (2508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:24:13 (5956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:41:08 (5804): No heartbeat from core client for 30 sec - exiting 08:41:09 (5804): No heartbeat from core client for 30 sec - exiting 08:41:10 (5804): No heartbeat from core client for 30 sec - exiting 08:41:12 (5804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:41:13 (5804): No heartbeat from core client for 30 sec - exiting 08:43:56 (5652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:01:02 (5632): No heartbeat from core client for 30 sec - exiting 13:01:03 (5632): No heartbeat from core client for 30 sec - exiting 13:01:04 (5632): No heartbeat from core client for 30 sec - exiting 13:01:05 (5632): No heartbeat from core client for 30 sec - exiting 13:01:06 (5632): No heartbeat from core client for 30 sec - exiting 13:01:07 (5632): No heartbeat from core client for 30 sec - exiting 13:01:09 (5632): No heartbeat from core client for 30 sec - exiting 13:01:10 (5632): No heartbeat from core client for 30 sec - exiting 13:01:11 (5632): No heartbeat from core client for 30 sec - exiting 13:01:12 (5632): No heartbeat from core client for 30 sec - exiting 13:01:13 (5632): No heartbeat from core client for 30 sec - exiting 13:01:14 (5632): No heartbeat from core client for 30 sec - exiting 13:01:15 (5632): No heartbeat from core client for 30 sec - exiting 13:01:16 (5632): No heartbeat from core client for 30 sec - exiting 13:01:17 (5632): No heartbeat from core client for 30 sec - exiting 13:01:18 (5632): No heartbeat from core client for 30 sec - exiting 13:01:19 (5632): No heartbeat from core client for 30 sec - exiting 13:01:21 (5632): No heartbeat from core client for 30 sec - exiting 13:01:22 (5632): No heartbeat from core client for 30 sec - exiting 13:01:23 (5632): No heartbeat from core client for 30 sec - exiting 13:01:24 (5632): No heartbeat from core client for 30 sec - exiting 13:01:25 (5632): No heartbeat from core client for 30 sec - exiting 13:01:26 (5632): No heartbeat from core client for 30 sec - exiting 13:01:27 (5632): No heartbeat from core client for 30 sec - exiting 13:01:28 (5632): No heartbeat from core client for 30 sec - exiting 13:01:29 (5632): No heartbeat from core client for 30 sec - exiting 13:01:30 (5632): No heartbeat from core client for 30 sec - exiting 13:01:31 (5632): No heartbeat from core client for 30 sec - exiting 13:01:33 (5632): No heartbeat from core client for 30 sec - exiting 13:01:34 (5632): No heartbeat from core client for 30 sec - exiting 13:01:35 (5632): No heartbeat from core client for 30 sec - exiting 13:01:36 (5632): No heartbeat from core client for 30 sec - exiting 13:01:37 (5632): No heartbeat from core client for 30 sec - exiting 13:01:38 (5632): No heartbeat from core client for 30 sec - exiting 13:01:39 (5632): No heartbeat from core client for 30 sec - exiting 13:01:40 (5632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:54:10 (5516): No heartbeat from core client for 30 sec - exiting 13:54:11 (5516): No heartbeat from core client for 30 sec - exiting 13:54:12 (5516): No heartbeat from core client for 30 sec - exiting 13:54:13 (5516): No heartbeat from core client for 30 sec - exiting 13:54:14 (5516): No heartbeat from core client for 30 sec - exiting 13:54:15 (5516): No heartbeat from core client for 30 sec - exiting 13:54:16 (5516): No heartbeat from core client for 30 sec - exiting 13:54:17 (5516): No heartbeat from core client for 30 sec - exiting 13:54:19 (5516): No heartbeat from core client for 30 sec - exiting 13:54:20 (5516): No heartbeat from core client for 30 sec - exiting 13:54:21 (5516): No heartbeat from core client for 30 sec - exiting 13:54:22 (5516): No heartbeat from core client for 30 sec - exiting 13:54:23 (5516): No heartbeat from core client for 30 sec - exiting 13:54:24 (5516): No heartbeat from core client for 30 sec - exiting 13:54:25 (5516): No heartbeat from core client for 30 sec - exiting 13:54:26 (5516): No heartbeat from core client for 30 sec - exiting 13:54:27 (5516): No heartbeat from core client for 30 sec - exiting 13:54:28 (5516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:16:36 (5748): No heartbeat from core client for 30 sec - exiting 19:16:37 (5748): No heartbeat from core client for 30 sec - exiting 19:16:38 (5748): No heartbeat from core client for 30 sec - exiting 19:16:39 (5748): No heartbeat from core client for 30 sec - exiting 19:16:40 (5748): No heartbeat from core client for 30 sec - exiting 19:16:41 (5748): No heartbeat from core client for 30 sec - exiting 19:16:42 (5748): No heartbeat from core client for 30 sec - exiting 19:16:43 (5748): No heartbeat from core client for 30 sec - exiting 19:16:44 (5748): No heartbeat from core client for 30 sec - exiting 19:16:46 (5748): No heartbeat from core client for 30 sec - exiting 19:16:47 (5748): No heartbeat from core client for 30 sec - exiting 19:16:48 (5748): No heartbeat from core client for 30 sec - exiting 19:16:49 (5748): No heartbeat from core client for 30 sec - exiting 19:16:50 (5748): No heartbeat from core client for 30 sec - exiting 19:16:51 (5748): No heartbeat from core client for 30 sec - exiting 19:16:52 (5748): No heartbeat from core client for 30 sec - exiting 19:16:53 (5748): No heartbeat from core client for 30 sec - exiting 19:16:54 (5748): No heartbeat from core client for 30 sec - exiting 19:16:55 (5748): No heartbeat from core client for 30 sec - exiting 19:16:56 (5748): No heartbeat from core client for 30 sec - exiting 19:16:58 (5748): No heartbeat from core client for 30 sec - exiting 19:16:59 (5748): No heartbeat from core client for 30 sec - exiting 19:17:00 (5748): No heartbeat from core client for 30 sec - exiting 19:17:01 (5748): No heartbeat from core client for 30 sec - exiting 19:17:02 (5748): No heartbeat from core client for 30 sec - exiting 19:17:03 (5748): No heartbeat from core client for 30 sec - exiting 19:17:04 (5748): No heartbeat from core client for 30 sec - exiting 19:17:05 (5748): No heartbeat from core client for 30 sec - exiting 19:17:06 (5748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:17:07 (5748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17892, iMonCtr=1 Model crash detected, will try to restart... 19:28:49 (5400): No heartbeat from core client for 30 sec - exiting 19:28:50 (5400): No heartbeat from core client for 30 sec - exiting 19:28:51 (5400): No heartbeat from core client for 30 sec - exiting 19:28:52 (5400): No heartbeat from core client for 30 sec - exiting 19:28:53 (5400): No heartbeat from core client for 30 sec - exiting 19:28:54 (5400): No heartbeat from core client for 30 sec - exiting 19:28:56 (5400): No heartbeat from core client for 30 sec - exiting 19:28:57 (5400): No heartbeat from core client for 30 sec - exiting 19:28:58 (5400): No heartbeat from core client for 30 sec - exiting 19:28:59 (5400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:30:39 (6128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:23:09 (5956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3240, iMonCtr=1 Model crash detected, will try to restart... 11:42:31 (1484): No heartbeat from core client for 30 sec - exiting 11:42:32 (1484): No heartbeat from core client for 30 sec - exiting 11:42:33 (1484): No heartbeat from core client for 30 sec - exiting 11:42:34 (1484): No heartbeat from core client for 30 sec - exiting 11:42:35 (1484): No heartbeat from core client for 30 sec - exiting 11:42:36 (1484): No heartbeat from core client for 30 sec - exiting 11:42:37 (1484): No heartbeat from core client for 30 sec - exiting 11:42:39 (1484): No heartbeat from core client for 30 sec - exiting 11:42:40 (1484): No heartbeat from core client for 30 sec - exiting 11:42:41 (1484): No heartbeat from core client for 30 sec - exiting 11:42:42 (1484): No heartbeat from core client for 30 sec - exiting 11:42:43 (1484): No heartbeat from core client for 30 sec - exiting 11:42:44 (1484): No heartbeat from core client for 30 sec - exiting 11:42:45 (1484): No heartbeat from core client for 30 sec - exiting 11:42:46 (1484): No heartbeat from core client for 30 sec - exiting 11:42:47 (1484): No heartbeat from core client for 30 sec - exiting 11:42:48 (1484): No heartbeat from core client for 30 sec - exiting 11:42:49 (1484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4200, iMonCtr=1 Model crash detected, will try to restart... 12:00:42 (5288): No heartbeat from core client for 30 sec - exiting 12:00:43 (5288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77582AB1 read attempt to address 0x40BD201D Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77582AB1 read attempt to address 0x40BD201D Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_833q_1980_40_008461961/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Mar 2014 20:35:52	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	777,600	2,196,059	2.8241
22 Feb 2014 20:44:22	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	751,680	2,093,927	2.7857
20 Feb 2014 05:31:16	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	725,760	1,987,441	2.7384
18 Feb 2014 12:01:34	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	699,840	1,887,716	2.6974
15 Feb 2014 13:22:27	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	673,920	1,807,324	2.6818
12 Feb 2014 10:31:50	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	648,000	1,708,899	2.6372
09 Feb 2014 14:01:48	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	622,080	1,598,181	2.5691
08 Feb 2014 02:34:31	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	596,160	1,490,924	2.5009
05 Feb 2014 11:06:20	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	570,240	1,387,320	2.4329
31 Jan 2014 05:17:44	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	544,320	1,282,615	2.3564
27 Jan 2014 07:12:19	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	518,400	1,175,794	2.2681
25 Jan 2014 19:54:27	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	492,480	1,068,264	2.1692
24 Jan 2014 13:17:53	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	466,560	977,160	2.0944
21 Jan 2014 19:10:27	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	440,640	882,165	2.0020
14 Jan 2014 12:44:13	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	414,720	800,078	1.9292
07 Jan 2014 15:23:28	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	388,800	724,135	1.8625
06 Jan 2014 23:25:27	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	362,880	674,212	1.8579
06 Jan 2014 09:08:48	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	336,960	624,188	1.8524
05 Jan 2014 18:49:23	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	311,040	574,128	1.8458
05 Jan 2014 02:03:30	1307977	16160899	hadcm3n_833q_1980_40_008461961_3	285,120	526,916	1.8480