Task 15447000

Name	hadcm3n_z928_1880_40_008248473_0
Workunit	8403597
Created	21 Nov 2012, 14:18:11 UTC
Sent	21 Nov 2012, 14:18:22 UTC
Report deadline	20 Feb 2013, 21:45:33 UTC
Received	21 Feb 2013, 5:46:05 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1190093
Run time	20 days 20 hours 0 min 8 sec
CPU time	19 days 18 hours 46 min 26 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.41 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 06:33:39 (3764): No heartbeat from core client for 30 sec - exiting 06:33:40 (3764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:15:07 (3664): No heartbeat from core client for 30 sec - exiting 11:15:08 (3664): No heartbeat from core client for 30 sec - exiting 11:15:10 (3664): No heartbeat from core client for 30 sec - exiting 11:15:11 (3664): No heartbeat from core client for 30 sec - exiting 11:15:12 (3664): No heartbeat from core client for 30 sec - exiting 11:15:13 (3664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:03:46 (3888): No heartbeat from core client for 30 sec - exiting 17:03:47 (3888): No heartbeat from core client for 30 sec - exiting 17:03:49 (3888): No heartbeat from core client for 30 sec - exiting 17:03:50 (3888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/z928ko.pj81c10 Error converting file to netcdf: dataout/z928ko.pi81c10 Error converting file to netcdf: dataout/z928ko.pf81c10 Error converting file to netcdf: dataout/z928ka.ph81c10 Error converting file to netcdf: dataout/z928ka.pg81c10 Error converting file to netcdf: dataout/z928ka.pe81c10 Error converting file to netcdf: dataout/z928ka.pd81c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:46:19 (3800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:08:57 (3528): No heartbeat from core client for 30 sec - exiting 15:08:58 (3528): No heartbeat from core client for 30 sec - exiting 15:08:59 (3528): No heartbeat from core client for 30 sec - exiting 15:09:00 (3528): No heartbeat from core client for 30 sec - exiting 15:09:01 (3528): No heartbeat from core client for 30 sec - exiting 15:09:03 (3528): No heartbeat from core client for 30 sec - exiting 15:09:04 (3528): No heartbeat from core client for 30 sec - exiting 15:09:05 (3528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:34:07 (3312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:39:19 (3140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:39:21 (3140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 06:38:04 (3280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:39:37 (3336): No heartbeat from core client for 30 sec - exiting 06:39:39 (3336): No heartbeat from core client for 30 sec - exiting 06:39:40 (3336): No heartbeat from core client for 30 sec - exiting 06:39:41 (3336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:38:48 (3904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:43:20 (3480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:42:55 (3884): No heartbeat from core client for 30 sec - exiting 06:42:56 (3884): No heartbeat from core client for 30 sec - exiting 06:42:57 (3884): No heartbeat from core client for 30 sec - exiting 06:42:58 (3884): No heartbeat from core client for 30 sec - exiting 06:42:59 (3884): No heartbeat from core client for 30 sec - exiting 06:43:00 (3884): No heartbeat from core client for 30 sec - exiting 06:43:01 (3884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:10:53 (3516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:41:41 (4020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:10:10 (3748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:19:01 (3488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:43:17 (3532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=984, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3924, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 06:42:08 (3208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:42:00 (2664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:15:58 (4088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:22:56 (3800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2044, iMonCtr=1 Model crash detected, will try to restart... 21:45:22 (3508): No heartbeat from core client for 30 sec - exiting 21:45:23 (3508): No heartbeat from core client for 30 sec - exiting 21:45:24 (3508): No heartbeat from core client for 30 sec - exiting 21:45:26 (3508): No heartbeat from core client for 30 sec - exiting 21:45:27 (3508): No heartbeat from core client for 30 sec - exiting 21:45:28 (3508): No heartbeat from core client for 30 sec - exiting 21:45:29 (3508): No heartbeat from core client for 30 sec - exiting 21:45:30 (3508): No heartbeat from core client for 30 sec - exiting 21:45:31 (3508): No heartbeat from core client for 30 sec - exiting 21:45:32 (3508): No heartbeat from core client for 30 sec - exiting 21:45:33 (3508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3908, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3820, iMonCtr=1 Model crash detected, will try to restart... 12:49:07 (3864): No heartbeat from core client for 30 sec - exiting 12:49:09 (3864): No heartbeat from core client for 30 sec - exiting 12:49:10 (3864): No heartbeat from core client for 30 sec - exiting 12:49:11 (3864): No heartbeat from core client for 30 sec - exiting 12:49:12 (3864): No heartbeat from core client for 30 sec - exiting 12:49:13 (3864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3584, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76F03AB3 read attempt to address 0x40B14A53 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77453AB3 read attempt to address 0x40B14A53 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_z928_1880_40_008248473/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Feb 2013 22:47:21	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	1,036,800	1,708,562	1.6479
19 Feb 2013 23:10:49	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	1,010,880	1,666,886	1.6489
18 Feb 2013 22:40:53	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	984,960	1,622,823	1.6476
17 Feb 2013 21:20:04	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	959,040	1,577,971	1.6454
16 Feb 2013 19:10:54	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	933,120	1,534,763	1.6448
15 Feb 2013 18:22:02	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	907,200	1,490,588	1.6431
14 Feb 2013 15:10:34	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	881,280	1,447,512	1.6425
13 Feb 2013 14:25:26	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	855,360	1,405,471	1.6431
11 Feb 2013 17:36:26	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	829,440	1,362,892	1.6431
09 Feb 2013 20:01:27	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	803,520	1,319,983	1.6428
08 Feb 2013 22:08:50	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	777,600	1,275,682	1.6405
07 Feb 2013 11:28:52	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	751,680	1,232,299	1.6394
06 Feb 2013 22:49:26	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	725,760	1,189,338	1.6387
05 Feb 2013 10:47:18	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	699,840	1,145,703	1.6371
03 Feb 2013 19:46:45	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	673,920	1,102,862	1.6365
01 Feb 2013 23:23:24	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	648,000	1,059,579	1.6352
31 Jan 2013 22:29:26	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	622,080	1,017,027	1.6349
29 Jan 2013 22:26:48	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	596,160	974,802	1.6351
21 Jan 2013 18:11:31	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	570,240	933,249	1.6366
20 Jan 2013 00:38:05	1190093	15447000	hadcm3n_z928_1880_40_008248473_0	544,320	891,230	1.6373