Task 13603885

Name	hadcm3n_o4tv_1980_40_007537385_1
Workunit	7734617
Created	5 Nov 2011, 16:05:55 UTC
Sent	5 Nov 2011, 16:31:04 UTC
Report deadline	4 Feb 2012, 23:58:15 UTC
Received	20 Nov 2011, 16:46:57 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1170477
Run time	5 days 17 hours 23 min 38 sec
CPU time	5 days 3 hours 14 min 43 sec
Validate state	Invalid
Credit	2,177.28
Device peak FLOPS	2.79 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:01:50 (7960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:02:23 (7960): No heartbeat from core client for 30 sec - exiting 11:02:24 (7960): No heartbeat from core client for 30 sec - exiting 11:02:25 (7960): No heartbeat from core client for 30 sec - exiting 11:02:26 (7960): No heartbeat from core client for 30 sec - exiting 11:02:27 (7960): No heartbeat from core client for 30 sec - exiting 11:02:28 (7960): No heartbeat from core client for 30 sec - exiting 11:02:29 (7960): No heartbeat from core client for 30 sec - exiting 11:02:30 (7960): No heartbeat from core client for 30 sec - exiting 11:02:31 (7960): No heartbeat from core client for 30 sec - exiting 11:02:32 (7960): No heartbeat from core client for 30 sec - exiting 11:02:33 (7960): No heartbeat from core client for 30 sec - exiting 11:02:34 (7960): No heartbeat from core client for 30 sec - exiting 11:02:35 (7960): No heartbeat from core client for 30 sec - exiting 11:02:36 (7960): No heartbeat from core client for 30 sec - exiting 11:02:37 (7960): No heartbeat from core client for 30 sec - exiting 11:02:38 (7960): No heartbeat from core client for 30 sec - exiting 11:02:39 (7960): No heartbeat from core client for 30 sec - exiting 11:02:40 (7960): No heartbeat from core client for 30 sec - exiting 11:02:41 (7960): No heartbeat from core client for 30 sec - exiting 11:02:42 (7960): No heartbeat from core client for 30 sec - exiting 11:02:43 (7960): No heartbeat from core client for 30 sec - exiting 11:02:44 (7960): No heartbeat from core client for 30 sec - exiting 11:02:45 (7960): No heartbeat from core client for 30 sec - exiting 11:02:46 (7960): No heartbeat from core client for 30 sec - exiting 11:02:47 (7960): No heartbeat from core client for 30 sec - exiting 11:02:48 (7960): No heartbeat from core client for 30 sec - exiting 11:02:49 (7960): No heartbeat from core client for 30 sec - exiting 11:02:50 (7960): No heartbeat from core client for 30 sec - exiting 11:02:51 (7960): No heartbeat from core client for 30 sec - exiting 11:02:52 (7960): No heartbeat from core client for 30 sec - exiting 11:02:53 (7960): No heartbeat from core client for 30 sec - exiting 11:02:54 (7960): No heartbeat from core client for 30 sec - exiting 11:02:55 (7960): No heartbeat from core client for 30 sec - exiting 11:02:56 (7960): No heartbeat from core client for 30 sec - exiting 11:02:57 (7960): No heartbeat from core client for 30 sec - exiting 11:02:58 (7960): No heartbeat from core client for 30 sec - exiting 11:02:59 (7960): No heartbeat from core client for 30 sec - exiting 11:03:00 (7960): No heartbeat from core client for 30 sec - exiting 11:03:01 (7960): No heartbeat from core client for 30 sec - exiting 11:03:02 (7960): No heartbeat from core client for 30 sec - exiting 11:03:03 (7960): No heartbeat from core client for 30 sec - exiting 11:03:04 (7960): No heartbeat from core client for 30 sec - exiting 11:03:05 (7960): No heartbeat from core client for 30 sec - exiting 11:03:06 (7960): No heartbeat from core client for 30 sec - exiting 11:03:07 (7960): No heartbeat from core client for 30 sec - exiting 11:03:08 (7960): No heartbeat from core client for 30 sec - exiting 11:03:09 (7960): No heartbeat from core client for 30 sec - exiting 11:03:10 (7960): No heartbeat from core client for 30 sec - exiting 11:03:11 (7960): No heartbeat from core client for 30 sec - exiting 11:03:12 (7960): No heartbeat from core client for 30 sec - exiting 11:03:13 (7960): No heartbeat from core client for 30 sec - exiting 11:03:14 (7960): No heartbeat from core client for 30 sec - exiting 11:03:15 (7960): No heartbeat from core client for 30 sec - exiting 11:03:16 (7960): No heartbeat from core client for 30 sec - exiting 11:03:17 (7960): No heartbeat from core client for 30 sec - exiting 11:03:18 (7960): No heartbeat from core client for 30 sec - exiting 11:03:19 (7960): No heartbeat from core client for 30 sec - exiting 11:03:20 (7960): No heartbeat from core client for 30 sec - exiting 11:03:21 (7960): No heartbeat from core client for 30 sec - exiting 11:03:22 (7960): No heartbeat from core client for 30 sec - exiting 11:03:23 (7960): No heartbeat from core client for 30 sec - exiting 11:03:24 (7960): No heartbeat from core client for 30 sec - exiting 11:03:25 (7960): No heartbeat from core client for 30 sec - exiting 11:03:26 (7960): No heartbeat from core client for 30 sec - exiting 11:03:27 (7960): No heartbeat from core client for 30 sec - exiting 11:03:28 (7960): No heartbeat from core client for 30 sec - exiting 11:03:29 (7960): No heartbeat from core client for 30 sec - exiting 11:03:30 (7960): No heartbeat from core client for 30 sec - exiting 11:03:31 (7960): No heartbeat from core client for 30 sec - exiting 11:03:32 (7960): No heartbeat from core client for 30 sec - exiting 11:03:33 (7960): No heartbeat from core client for 30 sec - exiting 11:03:34 (7960): No heartbeat from core client for 30 sec - exiting 11:03:35 (7960): No heartbeat from core client for 30 sec - exiting 11:03:36 (7960): No heartbeat from core client for 30 sec - exiting 11:03:37 (7960): No heartbeat from core client for 30 sec - exiting 11:03:38 (7960): No heartbeat from core client for 30 sec - exiting 11:03:39 (7960): No heartbeat from core client for 30 sec - exiting 11:03:40 (7960): No heartbeat from core client for 30 sec - exiting 11:03:41 (7960): No heartbeat from core client for 30 sec - exiting 11:03:42 (7960): No heartbeat from core client for 30 sec - exiting 11:03:43 (7960): No heartbeat from core client for 30 sec - exiting 11:03:44 (7960): No heartbeat from core client for 30 sec - exiting 11:03:45 (7960): No heartbeat from core client for 30 sec - exiting 11:03:46 (7960): No heartbeat from core client for 30 sec - exiting 11:03:47 (7960): No heartbeat from core client for 30 sec - exiting 11:03:48 (7960): No heartbeat from core client for 30 sec - exiting 11:03:49 (7960): No heartbeat from core client for 30 sec - exiting 11:03:50 (7960): No heartbeat from core client for 30 sec - exiting 11:03:51 (7960): No heartbeat from core client for 30 sec - exiting 11:03:52 (7960): No heartbeat from core client for 30 sec - exiting 11:03:53 (7960): No heartbeat from core client for 30 sec - exiting 11:03:54 (7960): No heartbeat from core client for 30 sec - exiting 11:03:55 (7960): No heartbeat from core client for 30 sec - exiting 11:03:56 (7960): No heartbeat from core client for 30 sec - exiting 11:03:57 (7960): No heartbeat from core client for 30 sec - exiting 11:03:58 (7960): No heartbeat from core client for 30 sec - exiting 11:03:59 (7960): No heartbeat from core client for 30 sec - exiting 11:04:00 (7960): No heartbeat from core client for 30 sec - exiting 11:04:01 (7960): No heartbeat from core client for 30 sec - exiting 11:04:02 (7960): No heartbeat from core client for 30 sec - exiting 11:04:03 (7960): No heartbeat from core client for 30 sec - exiting 11:04:04 (7960): No heartbeat from core client for 30 sec - exiting 11:04:05 (7960): No heartbeat from core client for 30 sec - exiting 11:04:06 (7960): No heartbeat from core client for 30 sec - exiting 11:04:07 (7960): No heartbeat from core client for 30 sec - exiting 11:04:08 (7960): No heartbeat from core client for 30 sec - exiting 11:04:09 (7960): No heartbeat from core client for 30 sec - exiting 11:04:10 (7960): No heartbeat from core client for 30 sec - exiting 11:04:11 (7960): No heartbeat from core client for 30 sec - exiting 11:04:12 (7960): No heartbeat from core client for 30 sec - exiting 11:04:13 (7960): No heartbeat from core client for 30 sec - exiting 11:04:14 (7960): No heartbeat from core client for 30 sec - exiting 11:04:15 (7960): No heartbeat from core client for 30 sec - exiting 11:04:16 (7960): No heartbeat from core client for 30 sec - exiting 11:04:17 (7960): No heartbeat from core client for 30 sec - exiting 11:04:18 (7960): No heartbeat from core client for 30 sec - exiting 11:04:19 (7960): No heartbeat from core client for 30 sec - exiting 11:04:20 (7960): No heartbeat from core client for 30 sec - exiting 11:04:21 (7960): No heartbeat from core client for 30 sec - exiting 11:04:22 (7960): No heartbeat from core client for 30 sec - exiting 11:04:23 (7960): No heartbeat from core client for 30 sec - exiting 11:04:24 (7960): No heartbeat from core client for 30 sec - exiting 11:04:25 (7960): No heartbeat from core client for 30 sec - exiting 11:04:26 (7960): No heartbeat from core client for 30 sec - exiting 11:04:27 (7960): No heartbeat from core client for 30 sec - exiting 11:04:28 (7960): No heartbeat from core client for 30 sec - exiting 11:04:29 (7960): No heartbeat from core client for 30 sec - exiting 11:04:30 (7960): No heartbeat from core client for 30 sec - exiting 11:04:31 (7960): No heartbeat from core client for 30 sec - exiting 11:04:32 (7960): No heartbeat from core client for 30 sec - exiting 11:04:33 (7960): No heartbeat from core client for 30 sec - exiting 11:04:34 (7960): No heartbeat from core client for 30 sec - exiting 11:04:36 (7960): No heartbeat from core client for 30 sec - exiting 11:04:37 (7960): No heartbeat from core client for 30 sec - exiting 11:04:38 (7960): No heartbeat from core client for 30 sec - exiting 11:04:39 (7960): No heartbeat from core client for 30 sec - exiting 11:04:40 (7960): No heartbeat from core client for 30 sec - exiting 11:04:41 (7960): No heartbeat from core client for 30 sec - exiting 11:04:42 (7960): No heartbeat from core client for 30 sec - exiting 11:04:43 (7960): No heartbeat from core client for 30 sec - exiting 11:04:44 (7960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:27:42 (5992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:13:51 (5892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:28:38 (4608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... forrtl: The requested operation cannot be performed on a file with a user-mapped section open. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5764, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:53:56 (5312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:54:00 (5312): No heartbeat from core client for 30 sec - exiting 16:54:31 (5312): No heartbeat from core client for 30 sec - exiting 16:54:32 (5312): No heartbeat from core client for 30 sec - exiting 16:54:33 (5312): No heartbeat from core client for 30 sec - exiting 16:54:34 (5312): No heartbeat from core client for 30 sec - exiting 16:54:35 (5312): No heartbeat from core client for 30 sec - exiting 16:54:36 (5312): No heartbeat from core client for 30 sec - exiting 16:54:37 (5312): No heartbeat from core client for 30 sec - exiting 16:54:38 (5312): No heartbeat from core client for 30 sec - exiting 16:54:39 (5312): No heartbeat from core client for 30 sec - exiting 16:54:40 (5312): No heartbeat from core client for 30 sec - exiting 16:54:41 (5312): No heartbeat from core client for 30 sec - exiting 16:54:42 (5312): No heartbeat from core client for 30 sec - exiting 16:54:43 (5312): No heartbeat from core client for 30 sec - exiting 16:54:44 (5312): No heartbeat from core client for 30 sec - exiting 16:54:45 (5312): No heartbeat from core client for 30 sec - exiting 16:54:46 (5312): No heartbeat from core client for 30 sec - exiting 16:54:47 (5312): No heartbeat from core client for 30 sec - exiting 16:54:48 (5312): No heartbeat from core client for 30 sec - exiting 16:54:49 (5312): No heartbeat from core client for 30 sec - exiting 16:54:50 (5312): No heartbeat from core client for 30 sec - exiting 16:54:51 (5312): No heartbeat from core client for 30 sec - exiting 16:54:52 (5312): No heartbeat from core client for 30 sec - exiting 16:54:53 (5312): No heartbeat from core client for 30 sec - exiting 16:54:54 (5312): No heartbeat from core client for 30 sec - exiting 16:54:55 (5312): No heartbeat from core client for 30 sec - exiting 16:54:56 (5312): No heartbeat from core client for 30 sec - exiting 16:54:57 (5312): No heartbeat from core client for 30 sec - exiting 16:54:58 (5312): No heartbeat from core client for 30 sec - exiting 16:54:59 (5312): No heartbeat from core client for 30 sec - exiting 16:55:00 (5312): No heartbeat from core client for 30 sec - exiting 16:55:01 (5312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Nov 2011 14:40:20	1170477	13603885	hadcm3n_o4tv_1980_40_007537385_1	181,440	390,546	2.1525
17 Nov 2011 21:25:45	1170477	13603885	hadcm3n_o4tv_1980_40_007537385_1	155,520	334,759	2.1525
17 Nov 2011 09:28:34	1170477	13603885	hadcm3n_o4tv_1980_40_007537385_1	129,600	278,202	2.1466
17 Nov 2011 09:28:34	1170477	13603885	hadcm3n_o4tv_1980_40_007537385_1	103,680	221,735	2.1386
17 Nov 2011 09:28:34	1170477	13603885	hadcm3n_o4tv_1980_40_007537385_1	77,760	166,068	2.1356
09 Nov 2011 09:07:37	1170477	13603885	hadcm3n_o4tv_1980_40_007537385_1	51,840	109,985	2.1216
07 Nov 2011 09:16:05	1170477	13603885	hadcm3n_o4tv_1980_40_007537385_1	25,920	53,690	2.0714