Task 12987412

Name	hadcm3n_s1bu_1940_40_007298624_1
Workunit	7496048
Created	20 Jun 2011, 14:10:07 UTC
Sent	20 Jun 2011, 14:10:08 UTC
Report deadline	19 Sep 2011, 21:37:19 UTC
Received	2 Jul 2011, 17:09:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	775427
Run time	7 days 16 hours 21 min 53 sec
CPU time	6 days 6 hours 45 min 29 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.33 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.26</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5864, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4548, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4312, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6000, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6036, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5340, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5340, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4824, iMonCtr=1 Model crash detected, will try to restart... 16:27:02 (4876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:53:16 (2004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:08:16 (5420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:29:54 (7032): No heartbeat from core client for 30 sec - exiting 14:29:56 (7032): No heartbeat from core client for 30 sec - exiting 14:29:57 (7032): No heartbeat from core client for 30 sec - exiting 14:30:00 (7032): No heartbeat from core client for 30 sec - exiting 14:30:01 (7032): No heartbeat from core client for 30 sec - exiting 14:30:02 (7032): No heartbeat from core client for 30 sec - exiting 14:30:03 (7032): No heartbeat from core client for 30 sec - exiting 14:30:04 (7032): No heartbeat from core client for 30 sec - exiting 14:30:05 (7032): No heartbeat from core client for 30 sec - exiting 14:30:07 (7032): No heartbeat from core client for 30 sec - exiting 14:30:09 (7032): No heartbeat from core client for 30 sec - exiting 14:30:10 (7032): No heartbeat from core client for 30 sec - exiting 14:30:12 (7032): No heartbeat from core client for 30 sec - exiting 14:30:13 (7032): No heartbeat from core client for 30 sec - exiting 14:30:14 (7032): No heartbeat from core client for 30 sec - exiting 14:30:15 (7032): No heartbeat from core client for 30 sec - exiting 14:30:16 (7032): No heartbeat from core client for 30 sec - exiting 14:30:17 (7032): No heartbeat from core client for 30 sec - exiting 14:30:18 (7032): No heartbeat from core client for 30 sec - exiting 14:30:19 (7032): No heartbeat from core client for 30 sec - exiting 14:30:20 (7032): No heartbeat from core client for 30 sec - exiting 14:30:21 (7032): No heartbeat from core client for 30 sec - exiting 14:30:22 (7032): No heartbeat from core client for 30 sec - exiting 14:30:23 (7032): No heartbeat from core client for 30 sec - exiting 14:30:24 (7032): No heartbeat from core client for 30 sec - exiting 14:30:25 (7032): No heartbeat from core client for 30 sec - exiting 14:30:26 (7032): No heartbeat from core client for 30 sec - exiting 14:30:27 (7032): No heartbeat from core client for 30 sec - exiting 14:30:28 (7032): No heartbeat from core client for 30 sec - exiting 14:30:29 (7032): No heartbeat from core client for 30 sec - exiting 14:30:30 (7032): No heartbeat from core client for 30 sec - exiting 14:30:31 (7032): No heartbeat from core client for 30 sec - exiting 14:30:32 (7032): No heartbeat from core client for 30 sec - exiting 14:30:33 (7032): No heartbeat from core client for 30 sec - exiting 14:30:34 (7032): No heartbeat from core client for 30 sec - exiting 14:30:35 (7032): No heartbeat from core client for 30 sec - exiting 14:30:36 (7032): No heartbeat from core client for 30 sec - exiting 14:30:37 (7032): No heartbeat from core client for 30 sec - exiting 14:30:38 (7032): No heartbeat from core client for 30 sec - exiting 14:30:39 (7032): No heartbeat from core client for 30 sec - exiting 14:30:40 (7032): No heartbeat from core client for 30 sec - exiting 14:30:41 (7032): No heartbeat from core client for 30 sec - exiting 14:30:42 (7032): No heartbeat from core client for 30 sec - exiting 14:30:43 (7032): No heartbeat from core client for 30 sec - exiting 14:30:44 (7032): No heartbeat from core client for 30 sec - exiting 14:30:45 (7032): No heartbeat from core client for 30 sec - exiting 14:30:46 (7032): No heartbeat from core client for 30 sec - exiting 14:30:47 (7032): No heartbeat from core client for 30 sec - exiting 14:30:48 (7032): No heartbeat from core client for 30 sec - exiting 14:30:49 (7032): No heartbeat from core client for 30 sec - exiting 14:30:50 (7032): No heartbeat from core client for 30 sec - exiting 14:30:51 (7032): No heartbeat from core client for 30 sec - exiting 14:30:52 (7032): No heartbeat from core client for 30 sec - exiting 14:30:53 (7032): No heartbeat from core client for 30 sec - exiting 14:30:54 (7032): No heartbeat from core client for 30 sec - exiting 14:30:55 (7032): No heartbeat from core client for 30 sec - exiting 14:30:56 (7032): No heartbeat from core client for 30 sec - exiting 14:30:57 (7032): No heartbeat from core client for 30 sec - exiting 14:30:58 (7032): No heartbeat from core client for 30 sec - exiting 14:30:59 (7032): No heartbeat from core client for 30 sec - exiting 14:31:00 (7032): No heartbeat from core client for 30 sec - exiting 14:31:01 (7032): No heartbeat from core client for 30 sec - exiting 14:31:02 (7032): No heartbeat from core client for 30 sec - exiting 14:31:03 (7032): No heartbeat from core client for 30 sec - exiting 14:31:04 (7032): No heartbeat from core client for 30 sec - exiting 14:31:05 (7032): No heartbeat from core client for 30 sec - exiting 14:31:06 (7032): No heartbeat from core client for 30 sec - exiting 14:31:07 (7032): No heartbeat from core client for 30 sec - exiting 14:31:08 (7032): No heartbeat from core client for 30 sec - exiting 14:31:09 (7032): No heartbeat from core client for 30 sec - exiting 14:31:10 (7032): No heartbeat from core client for 30 sec - exiting 14:31:11 (7032): No heartbeat from core client for 30 sec - exiting 14:31:12 (7032): No heartbeat from core client for 30 sec - exiting 14:31:13 (7032): No heartbeat from core client for 30 sec - exiting 14:31:14 (7032): No heartbeat from core client for 30 sec - exiting 14:31:15 (7032): No heartbeat from core client for 30 sec - exiting 14:31:16 (7032): No heartbeat from core client for 30 sec - exiting 14:31:17 (7032): No heartbeat from core client for 30 sec - exiting 14:31:18 (7032): No heartbeat from core client for 30 sec - exiting 14:31:19 (7032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4756, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 21:44:41 (5004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3728, iMonCtr=1 Model crash detected, will try to restart... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Jul 2011 13:13:19	775427	12987412	hadcm3n_s1bu_1940_40_007298624_1	233,280	614,644	2.6348
01 Jul 2011 17:40:29	775427	12987412	hadcm3n_s1bu_1940_40_007298624_1	207,360	546,214	2.6341
30 Jun 2011 19:32:19	775427	12987412	hadcm3n_s1bu_1940_40_007298624_1	181,440	477,191	2.6300
28 Jun 2011 17:17:05	775427	12987412	hadcm3n_s1bu_1940_40_007298624_1	155,520	408,969	2.6297
26 Jun 2011 22:56:02	775427	12987412	hadcm3n_s1bu_1940_40_007298624_1	129,600	331,023	2.5542
26 Jun 2011 04:15:12	775427	12987412	hadcm3n_s1bu_1940_40_007298624_1	103,680	265,923	2.5648
25 Jun 2011 00:24:01	775427	12987412	hadcm3n_s1bu_1940_40_007298624_1	77,760	200,348	2.5765
23 Jun 2011 19:17:04	775427	12987412	hadcm3n_s1bu_1940_40_007298624_1	51,840	132,134	2.5489
21 Jun 2011 19:19:58	775427	12987412	hadcm3n_s1bu_1940_40_007298624_1	25,920	64,994	2.5075