Task 15490411

Name	hadcm3n_39a3_1940_40_008262072_0
Workunit	8417196
Created	21 Dec 2012, 0:24:05 UTC
Sent	21 Dec 2012, 0:27:21 UTC
Report deadline	22 Mar 2013, 7:54:32 UTC
Received	12 Feb 2013, 3:01:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1221449
Run time	20 days 17 hours 9 min 32 sec
CPU time	17 days 9 hours 43 min 11 sec
Validate state	Invalid
Credit	10,886.40
Device peak FLOPS	2.82 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3372, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5752, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4360, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4360, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4016, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4280, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3616, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4480, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4708, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3588, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3936, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3936, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3936, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CSuspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4404, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4404, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4404, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2532, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4968, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4968, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4968, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4992, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4568, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4616, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5672, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4984, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Feb 2013 20:45:11	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	907,200	1,700,191	1.8741
10 Feb 2013 18:59:22	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	881,280	1,657,503	1.8808
10 Feb 2013 02:20:27	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	855,360	1,610,426	1.8827
09 Feb 2013 12:02:37	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	829,440	1,560,141	1.8810
08 Feb 2013 21:48:32	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	803,520	1,509,730	1.8789
08 Feb 2013 06:31:14	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	777,600	1,461,245	1.8792
07 Feb 2013 15:26:26	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	751,680	1,412,495	1.8791
06 Feb 2013 03:47:05	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	725,760	1,364,806	1.8805
05 Feb 2013 13:45:23	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	699,840	1,315,752	1.8801
01 Feb 2013 14:35:46	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	673,920	1,267,458	1.8807
30 Jan 2013 13:35:16	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	648,000	1,219,208	1.8815
29 Jan 2013 22:11:47	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	622,080	1,171,533	1.8833
28 Jan 2013 15:25:33	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	596,160	1,123,127	1.8839
28 Jan 2013 09:04:35	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	570,240	1,075,195	1.8855
28 Jan 2013 09:04:35	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	544,320	1,027,084	1.8869
25 Jan 2013 09:45:29	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	518,400	977,415	1.8854
23 Jan 2013 11:03:36	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	492,480	929,686	1.8878
17 Jan 2013 13:40:53	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	466,560	881,054	1.8884
15 Jan 2013 15:10:52	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	440,640	832,346	1.8889
14 Jan 2013 09:23:52	1221449	15490411	hadcm3n_39a3_1940_40_008262072_0	414,720	782,706	1.8873