Task 13129172

Name	hadcm3n_ymzj_1900_40_007362649_1
Workunit	7560079
Created	6 Jul 2011, 15:27:02 UTC
Sent	6 Jul 2011, 18:13:36 UTC
Report deadline	6 Oct 2011, 1:40:47 UTC
Received	22 Jul 2011, 11:32:06 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1118905
Run time	14 days 18 hours 11 min 41 sec
CPU time	13 days 13 hours 14 min 59 sec
Validate state	Invalid
Credit	7,153.92
Device peak FLOPS	3.07 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... Called boinc_finish 08:21:44 (7062): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:23:05 (19209): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:25:15 (19225): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:27:27 (19243): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:28:16 (19258): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:29:02 (19273): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:29:49 (19288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:30:36 (19303): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:31:23 (19318): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:32:10 (19334): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:34:16 (19350): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:35:04 (19363): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:35:46 (19376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:43:32 (19388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:44:23 (19409): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:45:09 (19422): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:45:56 (19434): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:46:43 (19444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:30 (19455): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:48:17 (19465): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:49:03 (19475): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:30:55 (19485): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish 19:23:33 (20394): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:24:54 (20957): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:29:20 (20970): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:30:05 (20983): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:30:51 (20993): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:31:37 (21003): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:32:23 (21013): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:33:09 (21023): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:33:55 (21033): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:34:41 (21044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:35:28 (21055): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:36:14 (21065): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:37:01 (21075): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:37:48 (21085): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:38:34 (21095): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... 12:01:45 (9805): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:57 (21472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:03:38 (21486): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:05:11 (21501): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:05:57 (21516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:06:43 (21530): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:07:32 (21544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:08:15 (21558): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:09:48 (21572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:10:34 (21588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:11:20 (21602): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish 21:46:07 (21953): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 19:08:25	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	596,160	1,220,301	2.0469
25 Jul 2011 19:08:25	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	570,240	1,168,495	2.0491
25 Jul 2011 18:54:04	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	544,320	1,114,612	2.0477
25 Jul 2011 18:12:03	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	518,400	1,061,201	2.0471
25 Jul 2011 17:46:31	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	492,480	1,007,208	2.0452
25 Jul 2011 17:21:02	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	466,560	953,185	2.0430
25 Jul 2011 16:24:27	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	440,640	898,375	2.0388
25 Jul 2011 16:24:27	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	414,720	843,838	2.0347
25 Jul 2011 15:30:30	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	388,800	789,441	2.0305
25 Jul 2011 14:38:42	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	362,880	736,205	2.0288
25 Jul 2011 13:08:21	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	336,960	686,519	2.0374
25 Jul 2011 13:08:21	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	311,040	640,323	2.0587
25 Jul 2011 13:08:21	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	285,120	588,716	2.0648
25 Jul 2011 13:08:21	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	259,200	535,170	2.0647
25 Jul 2011 13:08:21	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	233,280	481,510	2.0641
25 Jul 2011 13:08:21	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	207,360	427,785	2.0630
25 Jul 2011 13:08:21	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	181,440	373,970	2.0611
10 Jul 2011 14:29:04	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	155,520	320,800	2.0628
09 Jul 2011 23:02:53	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	129,600	266,994	2.0601
09 Jul 2011 07:40:01	1118905	13129172	hadcm3n_ymzj_1900_40_007362649_1	103,680	213,292	2.0572