Task 15937176

Name	hadcm3n_zcqq_1960_40_008366395_3
Workunit	8517254
Created	23 Aug 2013, 13:38:34 UTC
Sent	23 Aug 2013, 13:46:43 UTC
Report deadline	22 Nov 2013, 21:13:54 UTC
Received	19 Sep 2013, 16:42:03 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1251442
Run time	12 days 4 hours 55 min 33 sec
CPU time	8 days 20 hours 37 min 32 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.28 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6520, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 18:22:37 (6224): No heartbeat from core client for 30 sec - exiting 18:22:38 (6224): No heartbeat from core client for 30 sec - exiting 18:22:39 (6224): No heartbeat from core client for 30 sec - exiting 18:22:40 (6224): No heartbeat from core client for 30 sec - exiting 18:22:41 (6224): No heartbeat from core client for 30 sec - exiting 18:22:42 (6224): No heartbeat from core client for 30 sec - exiting 18:22:43 (6224): No heartbeat from core client for 30 sec - exiting 18:22:44 (6224): No heartbeat from core client for 30 sec - exiting 18:22:45 (6224): No heartbeat from core client for 30 sec - exiting 18:22:47 (6224): No heartbeat from core client for 30 sec - exiting 18:22:48 (6224): No heartbeat from core client for 30 sec - exiting 18:22:49 (6224): No heartbeat from core client for 30 sec - exiting 18:22:50 (6224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5936, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5428, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5428, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5428, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5728, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:45:27 (6456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6736, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6212, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5864, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2704, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4572, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7580, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6964, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 13:02:23 (6256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:18:14 (4968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:36:47 (1868): No heartbeat from core client for 30 sec - exiting 09:36:48 (1868): No heartbeat from core client for 30 sec - exiting 09:36:49 (1868): No heartbeat from core client for 30 sec - exiting 09:36:50 (1868): No heartbeat from core client for 30 sec - exiting 09:36:51 (1868): No heartbeat from core client for 30 sec - exiting 09:36:53 (1868): No heartbeat from core client for 30 sec - exiting 09:36:54 (1868): No heartbeat from core client for 30 sec - exiting 09:36:55 (1868): No heartbeat from core client for 30 sec - exiting 09:36:56 (1868): No heartbeat from core client for 30 sec - exiting 09:36:57 (1868): No heartbeat from core client for 30 sec - exiting 09:36:58 (1868): No heartbeat from core client for 30 sec - exiting 09:36:59 (1868): No heartbeat from core client for 30 sec - exiting 09:37:00 (1868): No heartbeat from core client for 30 sec - exiting 09:37:01 (1868): No heartbeat from core client for 30 sec - exiting 09:37:03 (1868): No heartbeat from core client for 30 sec - exiting 09:37:04 (1868): No heartbeat from core client for 30 sec - exiting 09:37:05 (1868): No heartbeat from core client for 30 sec - exiting 09:37:06 (1868): No heartbeat from core client for 30 sec - exiting 09:37:07 (1868): No heartbeat from core client for 30 sec - exiting 09:37:08 (1868): No heartbeat from core client for 30 sec - exiting 09:37:09 (1868): No heartbeat from core client for 30 sec - exiting 09:37:10 (1868): No heartbeat from core client for 30 sec - exiting 09:37:11 (1868): No heartbeat from core client for 30 sec - exiting 09:37:12 (1868): No heartbeat from core client for 30 sec - exiting 09:37:13 (1868): No heartbeat from core client for 30 sec - exiting 09:37:15 (1868): No heartbeat from core client for 30 sec - exiting 09:37:16 (1868): No heartbeat from core client for 30 sec - exiting 09:37:17 (1868): No heartbeat from core client for 30 sec - exiting 09:37:18 (1868): No heartbeat from core client for 30 sec - exiting 09:37:19 (1868): No heartbeat from core client for 30 sec - exiting 09:37:20 (1868): No heartbeat from core client for 30 sec - exiting 09:37:21 (1868): No heartbeat from core client for 30 sec - exiting 09:37:22 (1868): No heartbeat from core client for 30 sec - exiting 09:37:23 (1868): No heartbeat from core client for 30 sec - exiting 09:37:24 (1868): No heartbeat from core client for 30 sec - exiting 09:37:25 (1868): No heartbeat from core client for 30 sec - exiting 09:37:27 (1868): No heartbeat from core client for 30 sec - exiting 09:37:28 (1868): No heartbeat from core client for 30 sec - exiting 09:37:29 (1868): No heartbeat from core client for 30 sec - exiting 09:37:30 (1868): No heartbeat from core client for 30 sec - exiting 09:37:31 (1868): No heartbeat from core client for 30 sec - exiting 09:37:32 (1868): No heartbeat from core client for 30 sec - exiting 09:37:33 (1868): No heartbeat from core client for 30 sec - exiting 09:37:34 (1868): No heartbeat from core client for 30 sec - exiting 09:37:35 (1868): No heartbeat from core client for 30 sec - exiting 09:37:36 (1868): No heartbeat from core client for 30 sec - exiting 09:37:37 (1868): No heartbeat from core client for 30 sec - exiting 09:37:39 (1868): No heartbeat from core client for 30 sec - exiting 09:37:40 (1868): No heartbeat from core client for 30 sec - exiting 09:37:41 (1868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7436, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 11:06:13 (7024): No heartbeat from core client for 30 sec - exiting 11:06:14 (7024): No heartbeat from core client for 30 sec - exiting 11:06:15 (7024): No heartbeat from core client for 30 sec - exiting 11:06:16 (7024): No heartbeat from core client for 30 sec - exiting 11:06:17 (7024): No heartbeat from core client for 30 sec - exiting 11:06:18 (7024): No heartbeat from core client for 30 sec - exiting 11:06:19 (7024): No heartbeat from core client for 30 sec - exiting 11:06:20 (7024): No heartbeat from core client for 30 sec - exiting 11:06:21 (7024): No heartbeat from core client for 30 sec - exiting 11:06:23 (7024): No heartbeat from core client for 30 sec - exiting 11:06:24 (7024): No heartbeat from core client for 30 sec - exiting 11:06:25 (7024): No heartbeat from core client for 30 sec - exiting 11:06:26 (7024): No heartbeat from core client for 30 sec - exiting 11:06:27 (7024): No heartbeat from core client for 30 sec - exiting 11:06:28 (7024): No heartbeat from core client for 30 sec - exiting 11:06:29 (7024): No heartbeat from core client for 30 sec - exiting 11:06:30 (7024): No heartbeat from core client for 30 sec - exiting 11:06:31 (7024): No heartbeat from core client for 30 sec - exiting 11:06:32 (7024): No heartbeat from core client for 30 sec - exiting 11:06:33 (7024): No heartbeat from core client for 30 sec - exiting 11:06:35 (7024): No heartbeat from core client for 30 sec - exiting 11:06:36 (7024): No heartbeat from core client for 30 sec - exiting 11:06:37 (7024): No heartbeat from core client for 30 sec - exiting 11:06:38 (7024): No heartbeat from core client for 30 sec - exiting 11:06:39 (7024): No heartbeat from core client for 30 sec - exiting 11:06:40 (7024): No heartbeat from core client for 30 sec - exiting 11:06:41 (7024): No heartbeat from core client for 30 sec - exiting 11:06:42 (7024): No heartbeat from core client for 30 sec - exiting 11:06:43 (7024): No heartbeat from core client for 30 sec - exiting 11:06:44 (7024): No heartbeat from core client for 30 sec - exiting 11:06:45 (7024): No heartbeat from core client for 30 sec - exiting 11:06:47 (7024): No heartbeat from core client for 30 sec - exiting 11:06:48 (7024): No heartbeat from core client for 30 sec - exiting 11:06:49 (7024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2828, iMonCtr=1 Model crash detected, will try to restart... 09:32:26 (2184): No heartbeat from core client for 30 sec - exiting 09:32:27 (2184): No heartbeat from core client for 30 sec - exiting 09:32:28 (2184): No heartbeat from core client for 30 sec - exiting 09:32:29 (2184): No heartbeat from core client for 30 sec - exiting 09:32:30 (2184): No heartbeat from core client for 30 sec - exiting 09:32:32 (2184): No heartbeat from core client for 30 sec - exiting 09:32:33 (2184): No heartbeat from core client for 30 sec - exiting 09:32:34 (2184): No heartbeat from core client for 30 sec - exiting 09:32:35 (2184): No heartbeat from core client for 30 sec - exiting 09:32:36 (2184): No heartbeat from core client for 30 sec - exiting 09:32:37 (2184): No heartbeat from core client for 30 sec - exiting 09:32:38 (2184): No heartbeat from core client for 30 sec - exiting 09:32:39 (2184): No heartbeat from core client for 30 sec - exiting 09:32:40 (2184): No heartbeat from core client for 30 sec - exiting 09:32:41 (2184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x774B3AC3 read attempt to address 0x40A983C5 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x774B3AC3 read attempt to address 0x40A983C5 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zcqq_1960_40_008366395/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Sep 2013 15:29:28	1251442	15937176	hadcm3n_zcqq_1960_40_008366395_3	259,200	761,867	2.9393
15 Sep 2013 17:41:37	1251442	15937176	hadcm3n_zcqq_1960_40_008366395_3	233,280	683,918	2.9317
13 Sep 2013 01:29:25	1251442	15937176	hadcm3n_zcqq_1960_40_008366395_3	207,360	600,357	2.8952
09 Sep 2013 21:49:41	1251442	15937176	hadcm3n_zcqq_1960_40_008366395_3	181,440	522,832	2.8816
07 Sep 2013 16:56:18	1251442	15937176	hadcm3n_zcqq_1960_40_008366395_3	155,520	441,347	2.8379
05 Sep 2013 02:56:47	1251442	15937176	hadcm3n_zcqq_1960_40_008366395_3	129,600	366,517	2.8281
02 Sep 2013 13:06:01	1251442	15937176	hadcm3n_zcqq_1960_40_008366395_3	103,680	292,565	2.8218
30 Aug 2013 21:03:42	1251442	15937176	hadcm3n_zcqq_1960_40_008366395_3	77,760	211,836	2.7242
28 Aug 2013 03:11:30	1251442	15937176	hadcm3n_zcqq_1960_40_008366395_3	51,840	147,340	2.8422
26 Aug 2013 02:17:19	1251442	15937176	hadcm3n_zcqq_1960_40_008366395_3	25,920	75,774	2.9234