Task 15914576

Name	hadcm3n_3ga7_2020_40_008361972_1
Workunit	8512831
Created	14 Aug 2013, 11:39:05 UTC
Sent	14 Aug 2013, 18:18:20 UTC
Report deadline	14 Nov 2013, 1:45:31 UTC
Received	27 Sep 2013, 14:55:13 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1179495
Run time	26 days 1 hours 31 min 2 sec
CPU time	19 days 9 hours 58 min 32 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.75 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3556, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3556, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3556, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3392, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3392, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3284, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3876, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3876, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3820, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3548, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3548, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3196, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3916, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3116, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3116, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3116, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4356, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4356, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3080, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3392, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:25:21 (3936): No heartbeat from core client for 30 sec - exiting 18:25:22 (3936): No heartbeat from core client for 30 sec - exiting 18:25:23 (3936): No heartbeat from core client for 30 sec - exiting 18:25:24 (3936): No heartbeat from core client for 30 sec - exiting 18:25:25 (3936): No heartbeat from core client for 30 sec - exiting 18:25:26 (3936): No heartbeat from core client for 30 sec - exiting 18:25:27 (3936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3352, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3820, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3764, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3764, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3764, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3920, iMonCtr=1 Model crash detected, will try to restart... 17:57:57 (3988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3340, iMonCtr=1 Model crash detected, will try to restart... 16:43:42 (3640): No heartbeat from core client for 30 sec - exiting 16:43:43 (3640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5296, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3392, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3392, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3388, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3388, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:34:41 (4860): No heartbeat from core client for 30 sec - exiting 16:34:43 (4860): No heartbeat from core client for 30 sec - exiting 16:34:44 (4860): No heartbeat from core client for 30 sec - exiting 16:34:45 (4860): No heartbeat from core client for 30 sec - exiting 16:34:46 (4860): No heartbeat from core client for 30 sec - exiting 16:34:47 (4860): No heartbeat from core client for 30 sec - exiting 16:34:48 (4860): No heartbeat from core client for 30 sec - exiting 16:34:49 (4860): No heartbeat from core client for 30 sec - exiting 16:34:50 (4860): No heartbeat from core client for 30 sec - exiting 16:34:51 (4860): No heartbeat from core client for 30 sec - exiting 16:34:52 (4860): No heartbeat from core client for 30 sec - exiting 16:34:53 (4860): No heartbeat from core client for 30 sec - exiting 16:34:54 (4860): No heartbeat from core client for 30 sec - exiting 16:34:55 (4860): No heartbeat from core client for 30 sec - exiting 16:34:56 (4860): No heartbeat from core client for 30 sec - exiting 16:34:57 (4860): No heartbeat from core client for 30 sec - exiting 16:34:58 (4860): No heartbeat from core client for 30 sec - exiting 16:34:59 (4860): No heartbeat from core client for 30 sec - exiting 16:35:00 (4860): No heartbeat from core client for 30 sec - exiting 16:35:01 (4860): No heartbeat from core client for 30 sec - exiting 16:35:02 (4860): No heartbeat from core client for 30 sec - exiting 16:35:03 (4860): No heartbeat from core client for 30 sec - exiting 16:35:04 (4860): No heartbeat from core client for 30 sec - exiting 16:35:05 (4860): No heartbeat from core client for 30 sec - exiting 16:35:06 (4860): No heartbeat from core client for 30 sec - exiting 16:35:07 (4860): No heartbeat from core client for 30 sec - exiting 16:35:08 (4860): No heartbeat from core client for 30 sec - exiting 16:35:09 (4860): No heartbeat from core client for 30 sec - exiting 16:35:10 (4860): No heartbeat from core client for 30 sec - exiting 16:35:11 (4860): No heartbeat from core client for 30 sec - exiting 16:35:12 (4860): No heartbeat from core client for 30 sec - exiting 16:35:13 (4860): No heartbeat from core client for 30 sec - exiting 16:35:14 (4860): No heartbeat from core client for 30 sec - exiting 16:35:15 (4860): No heartbeat from core client for 30 sec - exiting 16:35:16 (4860): No heartbeat from core client for 30 sec - exiting 16:35:17 (4860): No heartbeat from core client for 30 sec - exiting 16:35:18 (4860): No heartbeat from core client for 30 sec - exiting 16:35:19 (4860): No heartbeat from core client for 30 sec - exiting 16:35:20 (4860): No heartbeat from core client for 30 sec - exiting 16:35:21 (4860): No heartbeat from core client for 30 sec - exiting 16:35:22 (4860): No heartbeat from core client for 30 sec - exiting 16:35:23 (4860): No heartbeat from core client for 30 sec - exiting 16:35:24 (4860): No heartbeat from core client for 30 sec - exiting 16:35:25 (4860): No heartbeat from core client for 30 sec - exiting 16:35:26 (4860): No heartbeat from core client for 30 sec - exiting 16:35:27 (4860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x777D7383 read attempt to address 0x40D681A0 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77453AC3 read attempt to address 0x40D681A8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_3ga7_2020_40_008361972/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Sep 2013 14:26:43	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	518,400	1,620,349	3.1257
24 Sep 2013 05:07:27	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	492,480	1,537,490	3.1219
21 Sep 2013 23:12:25	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	466,560	1,454,652	3.1178
20 Sep 2013 02:57:10	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	440,640	1,372,331	3.1144
18 Sep 2013 02:59:33	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	414,720	1,291,662	3.1145
18 Sep 2013 02:59:33	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	414,720	1,291,662	3.1145
15 Sep 2013 23:33:11	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	388,800	1,213,404	3.1209
14 Sep 2013 05:41:47	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	362,880	1,134,691	3.1269
11 Sep 2013 23:55:55	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	336,960	1,055,684	3.1330
09 Sep 2013 00:51:20	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	311,040	977,316	3.1421
07 Sep 2013 10:49:31	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	285,120	895,009	3.1391
05 Sep 2013 05:32:20	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	259,200	810,423	3.1266
02 Sep 2013 03:22:02	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	233,280	729,720	3.1281
31 Aug 2013 09:35:05	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	207,360	649,513	3.1323
29 Aug 2013 02:42:51	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	181,440	566,511	3.1223
26 Aug 2013 19:06:34	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	155,520	487,731	3.1361
25 Aug 2013 01:16:41	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	129,600	404,269	3.1194
23 Aug 2013 13:07:12	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	103,680	321,395	3.0999
20 Aug 2013 18:11:02	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	77,760	242,022	3.1124
18 Aug 2013 11:30:26	1179495	15914576	hadcm3n_3ga7_2020_40_008361972_1	51,840	160,941	3.1046