Task 13656219

Name	hadcm3n_u4bh_1980_40_007545074_2
Workunit	7742306
Created	23 Nov 2011, 12:46:46 UTC
Sent	23 Nov 2011, 12:54:27 UTC
Report deadline	22 Feb 2012, 20:21:38 UTC
Received	12 Dec 2011, 22:37:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1181178
Run time	1 days 18 hours 50 min 18 sec
CPU time	1 days 11 hours 49 min 58 sec
Validate state	Invalid
Credit	622.08
Device peak FLOPS	2.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> Il dispositivo non riconosce il comando. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 18:11:03 (2092): No heartbeat from core client for 30 sec - exiting 18:11:05 (2092): No heartbeat from core client for 30 sec - exiting 18:11:06 (2092): No heartbeat from core client for 30 sec - exiting 18:11:07 (2092): No heartbeat from core client for 30 sec - exiting 18:11:08 (2092): No heartbeat from core client for 30 sec - exiting 18:11:09 (2092): No heartbeat from core client for 30 sec - exiting 18:11:10 (2092): No heartbeat from core client for 30 sec - exiting 18:11:11 (2092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:27:59 (1920): No heartbeat from core client for 30 sec - exiting 23:28:00 (1920): No heartbeat from core client for 30 sec - exiting 23:28:01 (1920): No heartbeat from core client for 30 sec - exiting 23:28:02 (1920): No heartbeat from core client for 30 sec - exiting 23:28:03 (1920): No heartbeat from core client for 30 sec - exiting 23:28:04 (1920): No heartbeat from core client for 30 sec - exiting 23:28:05 (1920): No heartbeat from core client for 30 sec - exiting 23:28:06 (1920): No heartbeat from core client for 30 sec - exiting 23:28:07 (1920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:15:32 (5824): No heartbeat from core client for 30 sec - exiting 14:15:33 (5824): No heartbeat from core client for 30 sec - exiting 14:15:34 (5824): No heartbeat from core client for 30 sec - exiting 14:15:35 (5824): No heartbeat from core client for 30 sec - exiting 14:15:36 (5824): No heartbeat from core client for 30 sec - exiting 14:15:37 (5824): No heartbeat from core client for 30 sec - exiting 14:15:38 (5824): No heartbeat from core client for 30 sec - exiting 14:15:40 (5824): No heartbeat from core client for 30 sec - exiting 14:15:41 (5824): No heartbeat from core client for 30 sec - exiting 14:15:42 (5824): No heartbeat from core client for 30 sec - exiting 14:15:43 (5824): No heartbeat from core client for 30 sec - exiting 14:15:44 (5824): No heartbeat from core client for 30 sec - exiting 14:15:45 (5824): No heartbeat from core client for 30 sec - exiting 14:15:46 (5824): No heartbeat from core client for 30 sec - exiting 14:15:47 (5824): No heartbeat from core client for 30 sec - exiting 14:15:48 (5824): No heartbeat from core client for 30 sec - exiting 14:15:49 (5824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:05:46 (2840): No heartbeat from core client for 30 sec - exiting 13:05:47 (2840): No heartbeat from core client for 30 sec - exiting 13:05:48 (2840): No heartbeat from core client for 30 sec - exiting 13:05:49 (2840): No heartbeat from core client for 30 sec - exiting 13:05:50 (2840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:02:44 (524): No heartbeat from core client for 30 sec - exiting 15:02:45 (524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2392, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/u4bhko.pji1c10 Error converting file to netcdf: dataout/u4bhko.pii1c10 Error converting file to netcdf: dataout/u4bhko.pfi1c10 Error converting file to netcdf: dataout/u4bhka.phi1c10 Error converting file to netcdf: dataout/u4bhka.pgi1c10 Error converting file to netcdf: dataout/u4bhka.pei1c10 Error converting file to netcdf: dataout/u4bhka.pdi1c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6132, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5080, iMonCtr=1 Model crash detected, will try to restart... 19:26:39 (2524): No heartbeat from core client for 30 sec - exiting 19:26:40 (2524): No heartbeat from core client for 30 sec - exiting 19:26:42 (2524): No heartbeat from core client for 30 sec - exiting 19:26:43 (2524): No heartbeat from core client for 30 sec - exiting 19:26:44 (2524): No heartbeat from core client for 30 sec - exiting 19:26:45 (2524): No heartbeat from core client for 30 sec - exiting 19:26:46 (2524): No heartbeat from core client for 30 sec - exiting 19:26:47 (2524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:30:37 (2908): No heartbeat from core client for 30 sec - exiting 23:30:38 (2908): No heartbeat from core client for 30 sec - exiting 23:30:39 (2908): No heartbeat from core client for 30 sec - exiting 23:30:40 (2908): No heartbeat from core client for 30 sec - exiting 23:30:41 (2908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:31:11 (1764): No heartbeat from core client for 30 sec - exiting 13:31:12 (1764): No heartbeat from core client for 30 sec - exiting 13:31:13 (1764): No heartbeat from core client for 30 sec - exiting 13:31:14 (1764): No heartbeat from core client for 30 sec - exiting 13:31:15 (1764): No heartbeat from core client for 30 sec - exiting 13:31:16 (1764): No heartbeat from core client for 30 sec - exiting 13:31:17 (1764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:55:52 (4812): No heartbeat from core client for 30 sec - exiting 16:55:54 (4812): No heartbeat from core client for 30 sec - exiting 16:55:55 (4812): No heartbeat from core client for 30 sec - exiting 16:55:56 (4812): No heartbeat from core client for 30 sec - exiting 16:55:57 (4812): No heartbeat from core client for 30 sec - exiting 16:55:58 (4812): No heartbeat from core client for 30 sec - exiting 16:55:59 (4812): No heartbeat from core client for 30 sec - exiting 16:56:00 (4812): No heartbeat from core client for 30 sec - exiting 16:56:01 (4812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:08:05 (6776): No heartbeat from core client for 30 sec - exiting 14:08:07 (6776): No heartbeat from core client for 30 sec - exiting 14:08:08 (6776): No heartbeat from core client for 30 sec - exiting 14:08:10 (6776): No heartbeat from core client for 30 sec - exiting 14:08:11 (6776): No heartbeat from core client for 30 sec - exiting 14:08:12 (6776): No heartbeat from core client for 30 sec - exiting 14:08:13 (6776): No heartbeat from core client for 30 sec - exiting 14:08:14 (6776): No heartbeat from core client for 30 sec - exiting 14:08:15 (6776): No heartbeat from core client for 30 sec - exiting 14:08:16 (6776): No heartbeat from core client for 30 sec - exiting 14:08:17 (6776): No heartbeat from core client for 30 sec - exiting 14:08:18 (6776): No heartbeat from core client for 30 sec - exiting 14:08:19 (6776): No heartbeat from core client for 30 sec - exiting 14:08:20 (6776): No heartbeat from core client for 30 sec - exiting 14:08:22 (6776): No heartbeat from core client for 30 sec - exiting 14:08:23 (6776): No heartbeat from core client for 30 sec - exiting 14:08:24 (6776): No heartbeat from core client for 30 sec - exiting 14:08:25 (6776): No heartbeat from core client for 30 sec - exiting 14:08:26 (6776): No heartbeat from core client for 30 sec - exiting 14:08:27 (6776): No heartbeat from core client for 30 sec - exiting 14:08:28 (6776): No heartbeat from core client for 30 sec - exiting 14:08:29 (6776): No heartbeat from core client for 30 sec - exiting 14:08:30 (6776): No heartbeat from core client for 30 sec - exiting 14:08:31 (6776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:08:32 (6776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:14:28 (1116): No heartbeat from core client for 30 sec - exiting 23:14:29 (1116): No heartbeat from core client for 30 sec - exiting 23:14:30 (1116): No heartbeat from core client for 30 sec - exiting 23:14:31 (1116): No heartbeat from core client for 30 sec - exiting 23:14:32 (1116): No heartbeat from core client for 30 sec - exiting 23:14:34 (1116): No heartbeat from core client for 30 sec - exiting 23:14:35 (1116): No heartbeat from core client for 30 sec - exiting 23:14:36 (1116): No heartbeat from core client for 30 sec - exiting 23:14:37 (1116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:48:47 (5288): No heartbeat from core client for 30 sec - exiting 18:48:48 (5288): No heartbeat from core client for 30 sec - exiting 18:48:49 (5288): No heartbeat from core client for 30 sec - exiting 18:48:51 (5288): No heartbeat from core client for 30 sec - exiting 18:48:52 (5288): No heartbeat from core client for 30 sec - exiting 18:48:53 (5288): No heartbeat from core client for 30 sec - exiting 18:48:54 (5288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Dec 2011 15:09:02	1181178	13656219	hadcm3n_u4bh_1980_40_007545074_2	51,840	118,662	2.2890
29 Nov 2011 12:06:22	1181178	13656219	hadcm3n_u4bh_1980_40_007545074_2	25,920	58,247	2.2472