Task 13327186

Name	hadcm3n_t4ek_1980_40_007436643_1
Workunit	7634146
Created	1 Sep 2011, 20:35:40 UTC
Sent	1 Sep 2011, 21:59:52 UTC
Report deadline	2 Dec 2011, 5:27:03 UTC
Received	12 Dec 2011, 21:11:34 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1088606
Run time	24 days 21 hours 10 min 53 sec
CPU time	24 days 6 hours 22 min 55 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.41 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:48:46 (5124): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:23:03 (6448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:54:47 (1300): No heartbeat from core client for 30 sec - exiting 14:54:48 (1300): No heartbeat from core client for 30 sec - exiting 14:54:49 (1300): No heartbeat from core client for 30 sec - exiting 14:54:50 (1300): No heartbeat from core client for 30 sec - exiting 14:54:51 (1300): No heartbeat from core client for 30 sec - exiting 14:54:52 (1300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:54:53 (1300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... ControllCPDN Monitor - Quit request from BOINC... 10:51:26 (5012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:00:52 (3360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:17:31 (5048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:23:54 (4700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t4ekko.pjj2c10 Error converting file to netcdf: dataout/t4ekko.pij2c10 Error converting file to netcdf: dataout/t4ekko.pfj2c10 Error converting file to netcdf: dataout/t4ekka.phj2c10 Error converting file to netcdf: dataout/t4ekka.pgj2c10 Error converting file to netcdf: dataout/t4ekka.pej2c10 Error converting file to netcdf: dataout/t4ekka.pdj2c10 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4520, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 19:17:41 (1376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:42:59 (1032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5980, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1200, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4224, iMonCtr=1 Model crash detected, will try to restart... 12:42:37 (4396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4148, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4412, iMonCtr=1 Model crash detected, will try to restart... 08:56:28 (4848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4808, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4328, iMonCtr=1 Model crash detected, will try to restart... 10:56:56 (4588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4140, iMonCtr=1 Model crash detected, will try to restart... 08:14:51 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t4ekko.pjk7c10 Error converting file to netcdf: dataout/t4ekko.pik7c10 Error converting file to netcdf: dataout/t4ekko.pfk7c10 Error converting file to netcdf: dataout/t4ekka.phk7c10 Error converting file to netcdf: dataout/t4ekka.pgk7c10 Error converting file to netcdf: dataout/t4ekka.pek7c10 Error converting file to netcdf: dataout/t4ekka.pdk7c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3716, iMonCtr=1 Model crash detected, will try to restart... 08:17:20 (4312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4052, iMonCtr=1 Model crash detected, will try to restart... 08:13:18 (4544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:13:38 (4500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:16:14 (4748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3612, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3444, iMonCtr=1 Model crash detected, will try to restart... 08:13:17 (4552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:31:35 (4144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:49:10 (5084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:17:39 (4972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:33:59 (4660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:51:59 (3680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:17:36 (4940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Dec 2011 20:14:42	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	1,036,800	2,096,565	2.0221
08 Dec 2011 22:14:57	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	1,010,880	2,043,789	2.0218
08 Dec 2011 07:02:41	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	984,960	1,991,038	2.0214
07 Dec 2011 15:19:05	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	959,040	1,937,883	2.0206
05 Dec 2011 17:28:59	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	933,120	1,885,082	2.0202
01 Dec 2011 20:43:31	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	907,200	1,832,639	2.0201
01 Dec 2011 04:54:26	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	881,280	1,778,782	2.0184
29 Nov 2011 22:18:14	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	855,360	1,726,318	2.0182
28 Nov 2011 16:47:40	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	829,440	1,674,025	2.0183
22 Nov 2011 18:21:30	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	803,520	1,621,646	2.0182
18 Nov 2011 20:49:52	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	777,600	1,569,330	2.0182
16 Nov 2011 21:04:49	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	751,680	1,516,748	2.0178
15 Nov 2011 17:39:07	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	725,760	1,464,200	2.0175
15 Nov 2011 17:39:06	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	699,840	1,411,749	2.0172
09 Nov 2011 19:31:59	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	673,920	1,358,786	2.0162
07 Nov 2011 20:53:59	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	648,000	1,306,499	2.0162
03 Nov 2011 21:24:22	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	622,080	1,253,979	2.0158
02 Nov 2011 14:38:30	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	596,160	1,201,604	2.0156
31 Oct 2011 18:09:51	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	570,240	1,149,643	2.0161
31 Oct 2011 16:38:57	1088606	13327186	hadcm3n_t4ek_1980_40_007436643_1	544,320	1,097,202	2.0157