Task 13352114

Name	hadcm3n_t72j_1940_40_007444164_2
Workunit	7641667
Created	9 Sep 2011, 7:17:20 UTC
Sent	9 Sep 2011, 7:17:32 UTC
Report deadline	9 Dec 2011, 14:44:43 UTC
Received	16 Sep 2011, 17:40:40 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1066561
Run time	2 days 11 hours 35 min 57 sec
CPU time	2 days 11 hours 35 min 57 sec
Validate state	Invalid
Credit	1,555.20
Device peak FLOPS	2.49 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.2.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 13:59:05 (2064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:29:50 (7824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:28:40 (3828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:27:37 (6864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:33:34 (6276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:10:45 (4500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:09:32 (6236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:08:29 (4312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:48:53 (2332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:55:35 (4720): No heartbeat from core client for 30 sec - exiting 13:55:36 (4720): No heartbeat from core client for 30 sec - exiting 13:55:37 (4720): No heartbeat from core client for 30 sec - exiting 13:55:38 (4720): No heartbeat from core client for 30 sec - exiting 13:55:39 (4720): No heartbeat from core client for 30 sec - exiting 13:55:40 (4720): No heartbeat from core client for 30 sec - exiting 13:55:41 (4720): No heartbeat from core client for 30 sec - exiting 13:55:42 (4720): No heartbeat from core client for 30 sec - exiting 13:55:43 (4720): No heartbeat from core client for 30 sec - exiting 13:55:44 (4720): No heartbeat from core client for 30 sec - exiting 13:55:46 (4720): No heartbeat from core client for 30 sec - exiting 13:55:47 (4720): No heartbeat from core client for 30 sec - exiting 13:55:48 (4720): No heartbeat from core client for 30 sec - exiting 13:55:49 (4720): No heartbeat from core client for 30 sec - exiting 13:55:50 (4720): No heartbeat from core client for 30 sec - exiting 13:55:51 (4720): No heartbeat from core client for 30 sec - exiting 13:55:52 (4720): No heartbeat from core client for 30 sec - exiting 13:55:53 (4720): No heartbeat from core client for 30 sec - exiting 13:55:54 (4720): No heartbeat from core client for 30 sec - exiting 13:55:56 (4720): No heartbeat from core client for 30 sec - exiting 13:55:57 (4720): No heartbeat from core client for 30 sec - exiting 13:55:58 (4720): No heartbeat from core client for 30 sec - exiting 13:55:59 (4720): No heartbeat from core client for 30 sec - exiting 13:56:00 (4720): No heartbeat from core client for 30 sec - exiting 13:56:01 (4720): No heartbeat from core client for 30 sec - exiting 13:56:02 (4720): No heartbeat from core client for 30 sec - exiting 13:56:03 (4720): No heartbeat from core client for 30 sec - exiting 13:56:04 (4720): No heartbeat from core client for 30 sec - exiting 13:56:05 (4720): No heartbeat from core client for 30 sec - exiting 13:56:06 (4720): No heartbeat from core client for 30 sec - exiting 13:56:08 (4720): No heartbeat from core client for 30 sec - exiting 13:56:09 (4720): No heartbeat from core client for 30 sec - exiting 13:56:10 (4720): No heartbeat from core client for 30 sec - exiting 13:56:11 (4720): No heartbeat from core client for 30 sec - exiting 13:56:12 (4720): No heartbeat from core client for 30 sec - exiting 13:56:13 (4720): No heartbeat from core client for 30 sec - exiting 13:56:14 (4720): No heartbeat from core client for 30 sec - exiting 13:56:15 (4720): No heartbeat from core client for 30 sec - exiting 13:56:16 (4720): No heartbeat from core client for 30 sec - exiting 13:56:17 (4720): No heartbeat from core client for 30 sec - exiting 13:56:19 (4720): No heartbeat from core client for 30 sec - exiting 13:56:20 (4720): No heartbeat from core client for 30 sec - exiting 13:56:21 (4720): No heartbeat from core client for 30 sec - exiting 13:56:22 (4720): No heartbeat from core client for 30 sec - exiting 13:56:23 (4720): No heartbeat from core client for 30 sec - exiting 13:56:24 (4720): No heartbeat from core client for 30 sec - exiting 13:56:25 (4720): No heartbeat from core client for 30 sec - exiting 13:56:26 (4720): No heartbeat from core client for 30 sec - exiting 13:56:27 (4720): No heartbeat from core client for 30 sec - exiting 13:56:28 (4720): No heartbeat from core client for 30 sec - exiting 13:56:29 (4720): No heartbeat from core client for 30 sec - exiting 13:56:31 (4720): No heartbeat from core client for 30 sec - exiting 13:56:32 (4720): No heartbeat from core client for 30 sec - exiting 13:56:33 (4720): No heartbeat from core client for 30 sec - exiting 13:56:34 (4720): No heartbeat from core client for 30 sec - exiting 13:56:35 (4720): No heartbeat from core client for 30 sec - exiting 13:56:36 (4720): No heartbeat from core client for 30 sec - exiting 13:56:37 (4720): No heartbeat from core client for 30 sec - exiting 13:56:38 (4720): No heartbeat from core client for 30 sec - exiting 13:56:39 (4720): No heartbeat from core client for 30 sec - exiting 13:56:40 (4720): No heartbeat from core client for 30 sec - exiting 13:56:41 (4720): No heartbeat from core client for 30 sec - exiting 13:56:43 (4720): No heartbeat from core client for 30 sec - exiting 13:56:44 (4720): No heartbeat from core client for 30 sec - exiting 13:56:45 (4720): No heartbeat from core client for 30 sec - exiting 13:56:46 (4720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t72jko.pje5c10 Error converting file to netcdf: dataout/t72jko.pie5c10 Error converting file to netcdf: dataout/t72jko.pfe5c10 Error converting file to netcdf: dataout/t72jka.phe5c10 Error converting file to netcdf: dataout/t72jka.pge5c10 Error converting file to netcdf: dataout/t72jka.pee5c10 Error converting file to netcdf: dataout/t72jka.pde5c10 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Sep 2011 12:14:02	1066561	13352114	hadcm3n_t72j_1940_40_007444164_2	129,600	214,754	1.6571
14 Sep 2011 07:45:23	1066561	13352114	hadcm3n_t72j_1940_40_007444164_2	103,680	171,766	1.6567
12 Sep 2011 20:29:30	1066561	13352114	hadcm3n_t72j_1940_40_007444164_2	77,760	128,389	1.6511
11 Sep 2011 15:16:21	1066561	13352114	hadcm3n_t72j_1940_40_007444164_2	51,840	85,759	1.6543
10 Sep 2011 10:49:01	1066561	13352114	hadcm3n_t72j_1940_40_007444164_2	25,920	43,276	1.6696