Task 15782949

Name	hadcm3n_n6lt_1880_40_008284593_3
Workunit	8435728
Created	14 May 2013, 10:27:36 UTC
Sent	14 May 2013, 10:28:25 UTC
Report deadline	13 Aug 2013, 17:55:36 UTC
Received	13 Jun 2013, 4:32:56 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1040449
Run time	26 days 16 hours 4 min 7 sec
CPU time	23 days 4 hours 57 min 37 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	2.68 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <stderr_txt> 06:36:20 (4852): No heartbeat from core client for 30 sec - exiting 06:36:21 (4852): No heartbeat from core client for 30 sec - exiting 06:36:22 (4852): No heartbeat from core client for 30 sec - exiting 06:36:23 (4852): No heartbeat from core client for 30 sec - exiting 06:36:24 (4852): No heartbeat from core client for 30 sec - exiting 06:36:25 (4852): No heartbeat from core client for 30 sec - exiting 06:36:26 (4852): No heartbeat from core client for 30 sec - exiting 06:36:27 (4852): No heartbeat from core client for 30 sec - exiting 06:36:28 (4852): No heartbeat from core client for 30 sec - exiting 06:36:30 (4852): No heartbeat from core client for 30 sec - exiting 06:36:31 (4852): No heartbeat from core client for 30 sec - exiting 06:36:32 (4852): No heartbeat from core client for 30 sec - exiting 06:36:33 (4852): No heartbeat from core client for 30 sec - exiting 06:36:34 (4852): No heartbeat from core client for 30 sec - exiting 06:36:35 (4852): No heartbeat from core client for 30 sec - exiting 06:36:36 (4852): No heartbeat from core client for 30 sec - exiting 06:36:37 (4852): No heartbeat from core client for 30 sec - exiting 06:36:38 (4852): No heartbeat from core client for 30 sec - exiting 06:36:39 (4852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:36:40 (4852): No heartbeat from core client for 30 sec - exiting 06:36:42 (4852): No heartbeat from core client for 30 sec - exiting 06:36:43 (4852): No heartbeat from core client for 30 sec - exiting 06:36:44 (4852): No heartbeat from core client for 30 sec - exiting 06:36:45 (4852): No heartbeat from core client for 30 sec - exiting 06:36:46 (4852): No heartbeat from core client for 30 sec - exiting 06:36:47 (4852): No heartbeat from core client for 30 sec - exiting 06:36:48 (4852): No heartbeat from core client for 30 sec - exiting 06:36:49 (4852): No heartbeat from core client for 30 sec - exiting 06:36:50 (4852): No heartbeat from core client for 30 sec - exiting 12:37:09 (3924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:37:10 (3924): No heartbeat from core client for 30 sec - exiting 12:37:11 (3924): No heartbeat from core client for 30 sec - exiting 13:26:09 (2124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:26:11 (2124): No heartbeat from core client for 30 sec - exiting 14:14:47 (500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:14:49 (500): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:39:36 (3236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:31:46 (5676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:31:47 (5676): No heartbeat from core client for 30 sec - exiting 00:31:48 (5676): No heartbeat from core client for 30 sec - exiting 14:11:41 (3828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:32:02 (5208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:48:59 (4296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:08 (6004): No heartbeat from core client for 30 sec - exiting 16:50:10 (6004): No heartbeat from core client for 30 sec - exiting 16:50:11 (6004): No heartbeat from core client for 30 sec - exiting 16:50:12 (6004): No heartbeat from core client for 30 sec - exiting 16:50:13 (6004): No heartbeat from core client for 30 sec - exiting 16:50:14 (6004): No heartbeat from core client for 30 sec - exiting 16:50:15 (6004): No heartbeat from core client for 30 sec - exiting 16:50:16 (6004): No heartbeat from core client for 30 sec - exiting 16:50:17 (6004): No heartbeat from core client for 30 sec - exiting 16:50:18 (6004): No heartbeat from core client for 30 sec - exiting 16:50:19 (6004): No heartbeat from core client for 30 sec - exiting 16:50:20 (6004): No heartbeat from core client for 30 sec - exiting 16:50:22 (6004): No heartbeat from core client for 30 sec - exiting 16:50:23 (6004): No heartbeat from core client for 30 sec - exiting 16:50:24 (6004): No heartbeat from core client for 30 sec - exiting 16:50:25 (6004): No heartbeat from core client for 30 sec - exiting 16:50:26 (6004): No heartbeat from core client for 30 sec - exiting 16:50:27 (6004): No heartbeat from core client for 30 sec - exiting 16:50:28 (6004): No heartbeat from core client for 30 sec - exiting 16:50:29 (6004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:30 (6004): No heartbeat from core client for 30 sec - exiting 16:50:31 (6004): No heartbeat from core client for 30 sec - exiting 16:50:32 (6004): No heartbeat from core client for 30 sec - exiting 16:50:34 (6004): No heartbeat from core client for 30 sec - exiting 16:50:35 (6004): No heartbeat from core client for 30 sec - exiting 16:50:36 (6004): No heartbeat from core client for 30 sec - exiting 16:50:37 (6004): No heartbeat from core client for 30 sec - exiting 16:51:31 (5644): No heartbeat from core client for 30 sec - exiting 16:51:32 (5644): No heartbeat from core client for 30 sec - exiting 16:51:33 (5644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:23:02 (4276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:28:43 (1120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:45:26 (2548): No heartbeat from core client for 30 sec - exiting 20:45:27 (2548): No heartbeat from core client for 30 sec - exiting 20:45:29 (2548): No heartbeat from core client for 30 sec - exiting 20:45:30 (2548): No heartbeat from core client for 30 sec - exiting 20:45:31 (2548): No heartbeat from core client for 30 sec - exiting 20:45:32 (2548): No heartbeat from core client for 30 sec - exiting 20:45:33 (2548): No heartbeat from core client for 30 sec - exiting 20:45:34 (2548): No heartbeat from core client for 30 sec - exiting 20:45:35 (2548): No heartbeat from core client for 30 sec - exiting 20:45:36 (2548): No heartbeat from core client for 30 sec - exiting 20:45:37 (2548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:45:38 (2548): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:13:52 (4400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:13:53 (4400): No heartbeat from core client for 30 sec - exiting 13:13:54 (4400): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:04:54 (4860): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 17:04:56 (4860): No heartbeat from core client for 30 sec - exiting 17:04:57 (4860): No heartbeat from core client for 30 sec - exiting 00:15:26 (5604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:27:28 (5228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:30:49 (6860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:34:45 (1344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:37:44 (1240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:48:47 (440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:52:41 (6988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:55:46 (4660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:01:03 (5104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:11:27 (4692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 13:25:47 (6048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:29:37 (1124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6952, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6952, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6952, iMonCtr=1 Model crash detected, will try to restart... 13:49:22 (6952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6936, iMonCtr=1 Model crash detected, will try to restart... 13:51:34 (6936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=212, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=212, iMonCtr=1 Model crash detected, will try to restart... 15:27:41 (212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:23:32 (5708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Jun 2013 03:36:52	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	1,036,800	2,004,342	1.9332
12 Jun 2013 10:28:17	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	1,010,880	1,951,848	1.9308
11 Jun 2013 17:22:29	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	984,960	1,898,900	1.9279
11 Jun 2013 00:29:33	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	959,040	1,846,411	1.9253
10 Jun 2013 07:32:19	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	933,120	1,793,856	1.9224
09 Jun 2013 14:41:59	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	907,200	1,741,317	1.9194
08 Jun 2013 21:40:04	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	881,280	1,688,913	1.9164
08 Jun 2013 04:36:55	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	855,360	1,636,383	1.9131
07 Jun 2013 11:31:39	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	829,440	1,583,812	1.9095
06 Jun 2013 18:13:42	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	803,520	1,533,437	1.9084
05 Jun 2013 22:26:14	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	777,600	1,482,158	1.9061
05 Jun 2013 05:23:56	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	751,680	1,429,708	1.9020
04 Jun 2013 12:21:27	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	725,760	1,377,063	1.8974
03 Jun 2013 19:40:41	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	699,840	1,324,622	1.8927
03 Jun 2013 01:48:02	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	673,920	1,272,244	1.8878
02 Jun 2013 06:37:58	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	648,000	1,230,745	1.8993
01 Jun 2013 04:30:44	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	622,080	1,187,694	1.9092
31 May 2013 10:52:04	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	596,160	1,138,351	1.9095
30 May 2013 18:16:27	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	570,240	1,087,785	1.9076
30 May 2013 00:23:56	1040449	15782949	hadcm3n_n6lt_1880_40_008284593_3	544,320	1,036,986	1.9051