Task 16033918

Name	hadcm3n_o1dl_1940_40_008378747_4
Workunit	8529606
Created	25 Sep 2013, 5:31:04 UTC
Sent	25 Sep 2013, 5:32:36 UTC
Report deadline	25 Dec 2013, 12:59:47 UTC
Received	15 Oct 2013, 8:22:31 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1260847
Run time	9 days 10 hours 43 min 20 sec
CPU time	9 days 7 hours 22 min 12 sec
Validate state	Invalid
Credit	7,776.00
Device peak FLOPS	3.60 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7992, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7992, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:10:43 (8112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:34:13 (7340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:28:44 (7596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:41:46 (9812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Oct 2013 03:02:51	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	648,000	802,640	1.2386
14 Oct 2013 15:33:48	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	622,080	773,902	1.2441
14 Oct 2013 01:58:51	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	596,160	741,737	1.2442
13 Oct 2013 18:34:28	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	570,240	719,030	1.2609
13 Oct 2013 11:51:07	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	544,320	695,285	1.2773
13 Oct 2013 02:37:01	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	518,400	670,900	1.2942
12 Oct 2013 19:08:24	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	492,480	644,291	1.3083
12 Oct 2013 07:34:19	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	466,560	611,341	1.3103
11 Oct 2013 16:19:56	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	440,640	580,511	1.3174
11 Oct 2013 01:36:25	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	414,720	547,446	1.3200
10 Oct 2013 09:31:34	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	388,800	516,667	1.3289
07 Oct 2013 15:36:34	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	362,880	486,177	1.3398
06 Oct 2013 06:58:53	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	336,960	452,256	1.3422
05 Oct 2013 06:13:03	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	311,040	418,197	1.3445
03 Oct 2013 23:26:29	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	285,120	384,524	1.3486
02 Oct 2013 22:41:15	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	259,200	350,443	1.3520
02 Oct 2013 05:22:30	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	233,280	314,206	1.3469
01 Oct 2013 20:38:18	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	207,360	283,891	1.3691
01 Oct 2013 08:00:23	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	181,440	249,007	1.3724
30 Sep 2013 07:54:44	1260847	16033918	hadcm3n_o1dl_1940_40_008378747_4	155,520	213,068	1.3700