Task 12989065

Name	hadcm3n_s35u_1940_40_007299326_1
Workunit	7496750
Created	20 Jun 2011, 18:01:35 UTC
Sent	20 Jun 2011, 18:03:09 UTC
Report deadline	20 Sep 2011, 1:30:20 UTC
Received	30 Jun 2011, 14:08:09 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1103150
Run time	5 days 8 hours 49 min 42 sec
CPU time	4 days 20 hours 56 min 50 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.64 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 00:37:26 (4408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:12:32 (7584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:33:24 (10160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:22:12 (6312): No heartbeat from core client for 30 sec - exiting 22:22:13 (6312): No heartbeat from core client for 30 sec - exiting 22:22:14 (6312): No heartbeat from core client for 30 sec - exiting 22:22:15 (6312): No heartbeat from core client for 30 sec - exiting 22:22:16 (6312): No heartbeat from core client for 30 sec - exiting 22:22:17 (6312): No heartbeat from core client for 30 sec - exiting 22:22:18 (6312): No heartbeat from core client for 30 sec - exiting 22:22:19 (6312): No heartbeat from core client for 30 sec - exiting 22:22:20 (6312): No heartbeat from core client for 30 sec - exiting 22:22:21 (6312): No heartbeat from core client for 30 sec - exiting 22:22:22 (6312): No heartbeat from core client for 30 sec - exiting 22:22:23 (6312): No heartbeat from core client for 30 sec - exiting 22:22:24 (6312): No heartbeat from core client for 30 sec - exiting 22:22:25 (6312): No heartbeat from core client for 30 sec - exiting 22:22:26 (6312): No heartbeat from core client for 30 sec - exiting 22:22:27 (6312): No heartbeat from core client for 30 sec - exiting 22:22:28 (6312): No heartbeat from core client for 30 sec - exiting 22:22:29 (6312): No heartbeat from core client for 30 sec - exiting 22:22:30 (6312): No heartbeat from core client for 30 sec - exiting 22:22:31 (6312): No heartbeat from core client for 30 sec - exiting 22:22:32 (6312): No heartbeat from core client for 30 sec - exiting 22:22:33 (6312): No heartbeat from core client for 30 sec - exiting 22:22:34 (6312): No heartbeat from core client for 30 sec - exiting 22:22:35 (6312): No heartbeat from core client for 30 sec - exiting 22:22:36 (6312): No heartbeat from core client for 30 sec - exiting 22:22:37 (6312): No heartbeat from core client for 30 sec - exiting 22:22:38 (6312): No heartbeat from core client for 30 sec - exiting 22:22:39 (6312): No heartbeat from core client for 30 sec - exiting 22:22:40 (6312): No heartbeat from core client for 30 sec - exiting 22:22:41 (6312): No heartbeat from core client for 30 sec - exiting 22:22:42 (6312): No heartbeat from core client for 30 sec - exiting 22:22:43 (6312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:10:10 (7356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:41:06 (5328): No heartbeat from core client for 30 sec - exiting 12:41:07 (5328): No heartbeat from core client for 30 sec - exiting 12:41:08 (5328): No heartbeat from core client for 30 sec - exiting 12:41:09 (5328): No heartbeat from core client for 30 sec - exiting 12:41:10 (5328): No heartbeat from core client for 30 sec - exiting 12:41:11 (5328): No heartbeat from core client for 30 sec - exiting 12:41:12 (5328): No heartbeat from core client for 30 sec - exiting 12:41:13 (5328): No heartbeat from core client for 30 sec - exiting 12:41:14 (5328): No heartbeat from core client for 30 sec - exiting 12:41:15 (5328): No heartbeat from core client for 30 sec - exiting 12:41:16 (5328): No heartbeat from core client for 30 sec - exiting 12:41:17 (5328): No heartbeat from core client for 30 sec - exiting 12:41:18 (5328): No heartbeat from core client for 30 sec - exiting 12:41:19 (5328): No heartbeat from core client for 30 sec - exiting 12:41:20 (5328): No heartbeat from core client for 30 sec - exiting 12:41:21 (5328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:41:22 (5328): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:32:11 (7296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:08:05 (7252): No heartbeat from core client for 30 sec - exiting 20:08:06 (7252): No heartbeat from core client for 30 sec - exiting 20:08:07 (7252): No heartbeat from core client for 30 sec - exiting 20:08:08 (7252): No heartbeat from core client for 30 sec - exiting 20:08:09 (7252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:08:10 (7252): No heartbeat from core client for 30 sec - exiting 20:08:11 (7252): No heartbeat from core client for 30 sec - exiting 20:08:12 (7252): No heartbeat from core client for 30 sec - exiting 20:08:13 (7252): No heartbeat from core client for 30 sec - exiting 20:08:14 (7252): No heartbeat from core client for 30 sec - exiting 20:08:15 (7252): No heartbeat from core client for 30 sec - exiting 20:08:16 (7252): No heartbeat from core client for 30 sec - exiting 20:08:17 (7252): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8592, iMonCtr=1 Model crash detected, will try to restart... 08:35:25 (6956): No heartbeat from core client for 30 sec - exiting 08:35:26 (6956): No heartbeat from core client for 30 sec - exiting 08:35:27 (6956): No heartbeat from core client for 30 sec - exiting 08:35:28 (6956): No heartbeat from core client for 30 sec - exiting 08:35:29 (6956): No heartbeat from core client for 30 sec - exiting 08:35:30 (6956): No heartbeat from core client for 30 sec - exiting 08:35:31 (6956): No heartbeat from core client for 30 sec - exiting 08:35:32 (6956): No heartbeat from core client for 30 sec - exiting 08:35:33 (6956): No heartbeat from core client for 30 sec - exiting 08:35:34 (6956): No heartbeat from core client for 30 sec - exiting 08:35:35 (6956): No heartbeat from core client for 30 sec - exiting 08:35:36 (6956): No heartbeat from core client for 30 sec - exiting 08:35:37 (6956): No heartbeat from core client for 30 sec - exiting 08:35:38 (6956): No heartbeat from core client for 30 sec - exiting 08:35:39 (6956): No heartbeat from core client for 30 sec - exiting 08:35:40 (6956): No heartbeat from core client for 30 sec - exiting 08:35:41 (6956): No heartbeat from core client for 30 sec - exiting 08:35:42 (6956): No heartbeat from core client for 30 sec - exiting 08:35:43 (6956): No heartbeat from core client for 30 sec - exiting 08:35:44 (6956): No heartbeat from core client for 30 sec - exiting 08:35:45 (6956): No heartbeat from core client for 30 sec - exiting 08:35:46 (6956): No heartbeat from core client for 30 sec - exiting 08:35:47 (6956): No heartbeat from core client for 30 sec - exiting 08:35:48 (6956): No heartbeat from core client for 30 sec - exiting 08:35:49 (6956): No heartbeat from core client for 30 sec - exiting 08:35:50 (6956): No heartbeat from core client for 30 sec - exiting 08:35:51 (6956): No heartbeat from core client for 30 sec - exiting 08:35:52 (6956): No heartbeat from core client for 30 sec - exiting 08:35:53 (6956): No heartbeat from core client for 30 sec - exiting 08:35:54 (6956): No heartbeat from core client for 30 sec - exiting 08:35:55 (6956): No heartbeat from core client for 30 sec - exiting 08:35:56 (6956): No heartbeat from core client for 30 sec - exiting 08:35:57 (6956): No heartbeat from core client for 30 sec - exiting 08:35:58 (6956): No heartbeat from core client for 30 sec - exiting 08:35:59 (6956): No heartbeat from core client for 30 sec - exiting 08:36:00 (6956): No heartbeat from core client for 30 sec - exiting 08:36:01 (6956): No heartbeat from core client for 30 sec - exiting 08:36:02 (6956): No heartbeat from core client for 30 sec - exiting 08:36:03 (6956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:23:14 (7432): No heartbeat from core client for 30 sec - exiting 09:23:15 (7432): No heartbeat from core client for 30 sec - exiting 09:23:16 (7432): No heartbeat from core client for 30 sec - exiting 09:23:17 (7432): No heartbeat from core client for 30 sec - exiting 09:23:18 (7432): No heartbeat from core client for 30 sec - exiting 09:23:19 (7432): No heartbeat from core client for 30 sec - exiting 09:23:20 (7432): No heartbeat from core client for 30 sec - exiting 09:23:21 (7432): No heartbeat from core client for 30 sec - exiting 09:23:22 (7432): No heartbeat from core client for 30 sec - exiting 09:23:23 (7432): No heartbeat from core client for 30 sec - exiting 09:23:24 (7432): No heartbeat from core client for 30 sec - exiting 09:23:25 (7432): No heartbeat from core client for 30 sec - exiting 09:23:26 (7432): No heartbeat from core client for 30 sec - exiting 09:23:27 (7432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Jun 2011 12:32:21	1103150	12989065	hadcm3n_s35u_1940_40_007299326_1	233,280	434,846	1.8641
29 Jun 2011 14:26:47	1103150	12989065	hadcm3n_s35u_1940_40_007299326_1	207,360	383,179	1.8479
27 Jun 2011 22:00:23	1103150	12989065	hadcm3n_s35u_1940_40_007299326_1	181,440	332,683	1.8336
26 Jun 2011 04:05:07	1103150	12989065	hadcm3n_s35u_1940_40_007299326_1	155,520	289,354	1.8606
25 Jun 2011 08:02:33	1103150	12989065	hadcm3n_s35u_1940_40_007299326_1	129,600	242,439	1.8707
24 Jun 2011 02:58:22	1103150	12989065	hadcm3n_s35u_1940_40_007299326_1	103,680	194,411	1.8751
23 Jun 2011 00:34:01	1103150	12989065	hadcm3n_s35u_1940_40_007299326_1	77,760	145,122	1.8663
22 Jun 2011 10:52:51	1103150	12989065	hadcm3n_s35u_1940_40_007299326_1	51,840	96,959	1.8704
21 Jun 2011 09:14:34	1103150	12989065	hadcm3n_s35u_1940_40_007299326_1	25,920	48,133	1.8570