Task 16048901

Name	hadcm3n_u1qe_2020_40_008336230_2
Workunit	8487091
Created	29 Sep 2013, 2:58:19 UTC
Sent	29 Sep 2013, 3:03:06 UTC
Report deadline	29 Dec 2013, 10:30:17 UTC
Received	6 Oct 2013, 21:37:57 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1291356
Run time	7 days 7 hours 37 min 21 sec
CPU time	7 days 6 hours 25 min 42 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.10 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 16:43:52 (28762): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:45:49 (32439): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:47:51 (32446): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:49:53 (32454): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:51:51 (32469): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:54:02 (32481): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:56:24 (32488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:58:26 (32498): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:52 (32510): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:03:14 (32518): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:05:36 (32526): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:07:48 (32534): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:10:05 (32542): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:12:16 (32557): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:14:33 (32564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:16:50 (32571): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:18:47 (32580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:20:43 (32595): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:23:15 (32606): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:25:02 (32613): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:27:13 (32620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:29:15 (32628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:31:27 (32640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:34:04 (32649): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:36:16 (32657): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:38:23 (32665): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:40:59 (32677): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:43:11 (32687): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:45:13 (32694): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:47:49 (32701): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:49:56 (32711): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:52:08 (32725): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:54:00 (32733): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:56:27 (32740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:58 (32748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:01:00 (32760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:02:52 (302): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:04:59 (309): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:20 (316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:09:57 (324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:11:44 (339): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:46 (347): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:15:52 (357): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:18:14 (364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:20:46 (377): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:23:23 (388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:19 (396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:28:02 (432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:29:53 (505): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:32:36 (535): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:23 (546): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:36:45 (554): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:38:42 (563): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:59 (595): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:43:11 (678): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:45:40 (721): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:48:22 (754): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:50:28 (815): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:52:25 (824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:54:22 (834): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:56:20 (844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:58:42 (852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:01:10 (868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:03:12 (877): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:05:13 (888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:07:26 (900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:09:47 (911): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:12:00 (930): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:14:38 (938): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:16:40 (947): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:18:42 (959): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:21:25 (977): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:23:43 (985): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:26:04 (994): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:28:21 (1004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:30:23 (1030): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:32:45 (1039): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:35:17 (1074): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:37:19 (1101): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:13:20 (1108): No heartbeat from core client for 30 sec - exiting 04:13:21 (1108): No heartbeat from core client for 30 sec - exiting 04:13:22 (1108): No heartbeat from core client for 30 sec - exiting 04:13:23 (1108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_u1qe_2020_40_008336230/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_u1qe_2020_40_008336230/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_u1qe_2020_40_008336230/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_u1qe_2020_40_008336230/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_u1qe_2020_40_008336230/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_u1qe_2020_40_008336230/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_u1qe_2020_40_008336230/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_u1qe_2020_40_008336230/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_u1qe_2020_40_008336230/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_u1qe_2020_40_008336230/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_u1qe_2020_40_008336230/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_u1qe_2020_40_008336230/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Oct 2013 21:42:36	1291356	16048901	hadcm3n_u1qe_2020_40_008336230_2	259,200	627,866	2.4223
06 Oct 2013 03:01:29	1291356	16048901	hadcm3n_u1qe_2020_40_008336230_2	233,280	563,992	2.4177
05 Oct 2013 02:31:21	1291356	16048901	hadcm3n_u1qe_2020_40_008336230_2	207,360	499,469	2.4087
05 Oct 2013 01:30:54	1291356	16048901	hadcm3n_u1qe_2020_40_008336230_2	181,440	436,715	2.4069
05 Oct 2013 01:30:54	1291356	16048901	hadcm3n_u1qe_2020_40_008336230_2	155,520	374,226	2.4063
03 Oct 2013 09:51:48	1291356	16048901	hadcm3n_u1qe_2020_40_008336230_2	129,600	310,298	2.3943
03 Oct 2013 09:51:48	1291356	16048901	hadcm3n_u1qe_2020_40_008336230_2	103,680	247,612	2.3882
03 Oct 2013 09:51:48	1291356	16048901	hadcm3n_u1qe_2020_40_008336230_2	77,760	186,871	2.4032
03 Oct 2013 09:51:48	1291356	16048901	hadcm3n_u1qe_2020_40_008336230_2	51,840	123,743	2.3870
30 Sep 2013 06:03:56	1291356	16048901	hadcm3n_u1qe_2020_40_008336230_2	25,920	62,561	2.4136