Task 12993322

Name	hadcm3n_s63i_1940_40_007301131_0
Workunit	7498555
Created	21 Jun 2011, 2:22:42 UTC
Sent	21 Jun 2011, 2:23:41 UTC
Report deadline	20 Sep 2011, 9:50:52 UTC
Received	29 Jun 2011, 15:43:39 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1151942
Run time	7 days 4 hours 12 min 25 sec
CPU time	6 days 11 hours 12 min 52 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.24 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 07:10:24 (28399): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:15:23 (29000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:10:42 (29606): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:12:01 (29973): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:31:32 (30030): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:33:46 (19831): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:35:58 (19888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:35:59 (19888): No heartbeat from core client for 30 sec - exiting 15:38:10 (20497): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:40:22 (20554): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:43:15 (21159): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:12:10 (21217): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:12:11 (21217): No heartbeat from core client for 30 sec - exiting 11:15:23 (9189): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:15:24 (9189): No heartbeat from core client for 30 sec - exiting 12:22:02 (28696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:19:35 (3557): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:31:47 (22382): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:32:56 (23607): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:33:50 (1326): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:14:05 (8219): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:14:06 (8219): No heartbeat from core client for 30 sec - exiting 04:50:43 (13441): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:50:44 (13441): No heartbeat from core client for 30 sec - exiting 04:50:45 (13441): No heartbeat from core client for 30 sec - exiting 04:50:46 (13441): No heartbeat from core client for 30 sec - exiting 04:50:47 (13441): No heartbeat from core client for 30 sec - exiting 04:50:48 (13441): No heartbeat from core client for 30 sec - exiting 04:50:49 (13441): No heartbeat from core client for 30 sec - exiting 04:50:50 (13441): No heartbeat from core client for 30 sec - exiting 04:50:51 (13441): No heartbeat from core client for 30 sec - exiting 04:50:52 (13441): No heartbeat from core client for 30 sec - exiting 05:32:53 (17146): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:52:44 (21079): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:55:11 (23367): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:56:53 (23993): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:38:41 (24073): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:39:57 (10189): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:06:45 (19980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:46 (19980): No heartbeat from core client for 30 sec - exiting 07:22:12 (30658): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:22:13 (30658): No heartbeat from core client for 30 sec - exiting 07:22:14 (30658): No heartbeat from core client for 30 sec - exiting 07:22:15 (30658): No heartbeat from core client for 30 sec - exiting 07:22:16 (30658): No heartbeat from core client for 30 sec - exiting 07:22:17 (30658): No heartbeat from core client for 30 sec - exiting 07:22:18 (30658): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 04:02:20 (29854): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:50:43 (25872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:50:44 (25872): No heartbeat from core client for 30 sec - exiting 06:50:45 (25872): No heartbeat from core client for 30 sec - exiting 06:50:46 (25872): No heartbeat from core client for 30 sec - exiting 06:50:47 (25872): No heartbeat from core client for 30 sec - exiting 06:50:48 (25872): No heartbeat from core client for 30 sec - exiting 06:50:49 (25872): No heartbeat from core client for 30 sec - exiting 06:50:50 (25872): No heartbeat from core client for 30 sec - exiting 06:50:51 (25872): No heartbeat from core client for 30 sec - exiting 06:50:52 (25872): No heartbeat from core client for 30 sec - exiting 06:50:53 (25872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:43:54 (4324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:14:09 (4608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:40:59 (21496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:41:01 (21496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Jun 2011 10:29:40	1151942	12993322	hadcm3n_s63i_1940_40_007301131_0	233,280	556,379	2.3850
28 Jun 2011 05:06:51	1151942	12993322	hadcm3n_s63i_1940_40_007301131_0	207,360	494,422	2.3844
27 Jun 2011 04:27:36	1151942	12993322	hadcm3n_s63i_1940_40_007301131_0	181,440	432,260	2.3824
26 Jun 2011 04:20:15	1151942	12993322	hadcm3n_s63i_1940_40_007301131_0	155,520	369,590	2.3765
25 Jun 2011 05:11:05	1151942	12993322	hadcm3n_s63i_1940_40_007301131_0	129,600	307,794	2.3750
24 Jun 2011 07:46:08	1151942	12993322	hadcm3n_s63i_1940_40_007301131_0	103,680	244,531	2.3585
23 Jun 2011 12:32:13	1151942	12993322	hadcm3n_s63i_1940_40_007301131_0	77,760	182,642	2.3488
22 Jun 2011 17:30:55	1151942	12993322	hadcm3n_s63i_1940_40_007301131_0	51,840	122,084	2.3550
21 Jun 2011 22:46:55	1151942	12993322	hadcm3n_s63i_1940_40_007301131_0	25,920	61,512	2.3731