Task 15522050

Name	hadcm3n_3mio_1940_40_008260580_1
Workunit	8415704
Created	3 Jan 2013, 4:27:46 UTC
Sent	3 Jan 2013, 4:28:14 UTC
Report deadline	4 Apr 2013, 11:55:25 UTC
Received	15 Jan 2013, 0:31:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1225070
Run time	4 days 11 hours 53 min 52 sec
CPU time	4 days 3 hours 19 min 58 sec
Validate state	Invalid
Credit	3,732.48
Device peak FLOPS	3.31 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5164, iMonCtr=1 Model crash detected, will try to restart... 19:50:41 (4288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:18:52 (5276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:19:38 (2104): No heartbeat from core client for 30 sec - exiting 21:19:39 (2104): No heartbeat from core client for 30 sec - exiting 21:19:40 (2104): No heartbeat from core client for 30 sec - exiting 21:19:41 (2104): No heartbeat from core client for 30 sec - exiting 21:19:42 (2104): No heartbeat from core client for 30 sec - exiting 21:19:43 (2104): No heartbeat from core client for 30 sec - exiting 21:19:44 (2104): No heartbeat from core client for 30 sec - exiting 21:19:45 (2104): No heartbeat from core client for 30 sec - exiting 21:19:46 (2104): No heartbeat from core client for 30 sec - exiting 21:19:47 (2104): No heartbeat from core client for 30 sec - exiting 21:19:48 (2104): No heartbeat from core client for 30 sec - exiting 21:19:49 (2104): No heartbeat from core client for 30 sec - exiting 21:19:50 (2104): No heartbeat from core client for 30 sec - exiting 21:19:51 (2104): No heartbeat from core client for 30 sec - exiting 21:19:52 (2104): No heartbeat from core client for 30 sec - exiting 21:19:53 (2104): No heartbeat from core client for 30 sec - exiting 21:19:54 (2104): No heartbeat from core client for 30 sec - exiting 21:19:55 (2104): No heartbeat from core client for 30 sec - exiting 21:19:56 (2104): No heartbeat from core client for 30 sec - exiting 21:19:57 (2104): No heartbeat from core client for 30 sec - exiting 21:19:58 (2104): No heartbeat from core client for 30 sec - exiting 21:19:59 (2104): No heartbeat from core client for 30 sec - exiting 21:20:00 (2104): No heartbeat from core client for 30 sec - exiting 21:20:01 (2104): No heartbeat from core client for 30 sec - exiting 21:20:02 (2104): No heartbeat from core client for 30 sec - exiting 21:20:03 (2104): No heartbeat from core client for 30 sec - exiting 21:20:04 (2104): No heartbeat from core client for 30 sec - exiting 21:20:05 (2104): No heartbeat from core client for 30 sec - exiting 21:20:06 (2104): No heartbeat from core client for 30 sec - exiting 21:20:07 (2104): No heartbeat from core client for 30 sec - exiting 21:20:08 (2104): No heartbeat from core client for 30 sec - exiting 21:20:09 (2104): No heartbeat from core client for 30 sec - exiting 21:20:10 (2104): No heartbeat from core client for 30 sec - exiting 21:20:11 (2104): No heartbeat from core client for 30 sec - exiting 21:20:12 (2104): No heartbeat from core client for 30 sec - exiting 21:20:13 (2104): No heartbeat from core client for 30 sec - exiting 21:20:14 (2104): No heartbeat from core client for 30 sec - exiting 21:20:15 (2104): No heartbeat from core client for 30 sec - exiting 21:20:16 (2104): No heartbeat from core client for 30 sec - exiting 21:20:17 (2104): No heartbeat from core client for 30 sec - exiting 21:20:18 (2104): No heartbeat from core client for 30 sec - exiting 21:20:19 (2104): No heartbeat from core client for 30 sec - exiting 21:20:20 (2104): No heartbeat from core client for 30 sec - exiting 21:20:21 (2104): No heartbeat from core client for 30 sec - exiting 21:20:22 (2104): No heartbeat from core client for 30 sec - exiting 21:20:23 (2104): No heartbeat from core client for 30 sec - exiting 21:20:24 (2104): No heartbeat from core client for 30 sec - exiting 21:20:25 (2104): No heartbeat from core client for 30 sec - exiting 21:20:26 (2104): No heartbeat from core client for 30 sec - exiting 21:20:27 (2104): No heartbeat from core client for 30 sec - exiting 21:20:28 (2104): No heartbeat from core client for 30 sec - exiting 21:20:29 (2104): No heartbeat from core client for 30 sec - exiting 21:20:30 (2104): No heartbeat from core client for 30 sec - exiting 21:20:31 (2104): No heartbeat from core client for 30 sec - exiting 21:20:32 (2104): No heartbeat from core client for 30 sec - exiting 21:20:33 (2104): No heartbeat from core client for 30 sec - exiting 21:20:34 (2104): No heartbeat from core client for 30 sec - exiting 21:20:35 (2104): No heartbeat from core client for 30 sec - exiting 21:20:36 (2104): No heartbeat from core client for 30 sec - exiting 21:20:37 (2104): No heartbeat from core client for 30 sec - exiting 21:20:38 (2104): No heartbeat from core client for 30 sec - exiting 21:20:39 (2104): No heartbeat from core client for 30 sec - exiting 21:20:40 (2104): No heartbeat from core client for 30 sec - exiting 21:20:41 (2104): No heartbeat from core client for 30 sec - exiting 21:20:42 (2104): No heartbeat from core client for 30 sec - exiting 21:20:43 (2104): No heartbeat from core client for 30 sec - exiting 21:20:44 (2104): No heartbeat from core client for 30 sec - exiting 21:20:45 (2104): No heartbeat from core client for 30 sec - exiting 21:20:46 (2104): No heartbeat from core client for 30 sec - exiting 21:20:47 (2104): No heartbeat from core client for 30 sec - exiting 21:20:48 (2104): No heartbeat from core client for 30 sec - exiting 21:20:49 (2104): No heartbeat from core client for 30 sec - exiting 21:20:50 (2104): No heartbeat from core client for 30 sec - exiting 21:20:51 (2104): No heartbeat from core client for 30 sec - exiting 21:20:52 (2104): No heartbeat from core client for 30 sec - exiting 21:20:53 (2104): No heartbeat from core client for 30 sec - exiting 21:20:54 (2104): No heartbeat from core client for 30 sec - exiting 21:20:55 (2104): No heartbeat from core client for 30 sec - exiting 21:20:56 (2104): No heartbeat from core client for 30 sec - exiting 21:20:57 (2104): No heartbeat from core client for 30 sec - exiting 21:20:58 (2104): No heartbeat from core client for 30 sec - exiting 21:20:59 (2104): No heartbeat from core client for 30 sec - exiting 21:21:00 (2104): No heartbeat from core client for 30 sec - exiting 21:21:01 (2104): No heartbeat from core client for 30 sec - exiting 21:21:02 (2104): No heartbeat from core client for 30 sec - exiting 21:21:03 (2104): No heartbeat from core client for 30 sec - exiting 21:21:04 (2104): No heartbeat from core client for 30 sec - exiting 21:21:05 (2104): No heartbeat from core client for 30 sec - exiting 21:21:06 (2104): No heartbeat from core client for 30 sec - exiting 21:21:07 (2104): No heartbeat from core client for 30 sec - exiting 21:21:08 (2104): No heartbeat from core client for 30 sec - exiting 21:21:09 (2104): No heartbeat from core client for 30 sec - exiting 21:21:10 (2104): No heartbeat from core client for 30 sec - exiting 21:21:11 (2104): No heartbeat from core client for 30 sec - exiting 21:21:12 (2104): No heartbeat from core client for 30 sec - exiting 21:21:13 (2104): No heartbeat from core client for 30 sec - exiting 21:21:14 (2104): No heartbeat from core client for 30 sec - exiting 21:21:15 (2104): No heartbeat from core client for 30 sec - exiting 21:21:16 (2104): No heartbeat from core client for 30 sec - exiting 21:21:17 (2104): No heartbeat from core client for 30 sec - exiting 21:21:18 (2104): No heartbeat from core client for 30 sec - exiting 21:21:19 (2104): No heartbeat from core client for 30 sec - exiting 21:21:20 (2104): No heartbeat from core client for 30 sec - exiting 21:21:21 (2104): No heartbeat from core client for 30 sec - exiting 21:21:22 (2104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5212, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Jan 2013 22:31:01	1225070	15522050	hadcm3n_3mio_1940_40_008260580_1	311,040	362,568	1.1657
14 Jan 2013 05:12:52	1225070	15522050	hadcm3n_3mio_1940_40_008260580_1	285,120	329,928	1.1572
13 Jan 2013 05:18:55	1225070	15522050	hadcm3n_3mio_1940_40_008260580_1	259,200	306,092	1.1809
11 Jan 2013 07:04:00	1225070	15522050	hadcm3n_3mio_1940_40_008260580_1	233,280	276,907	1.1870
10 Jan 2013 22:47:27	1225070	15522050	hadcm3n_3mio_1940_40_008260580_1	207,360	245,457	1.1837
10 Jan 2013 05:18:04	1225070	15522050	hadcm3n_3mio_1940_40_008260580_1	181,440	214,862	1.1842
09 Jan 2013 20:35:19	1225070	15522050	hadcm3n_3mio_1940_40_008260580_1	155,520	183,906	1.1825
09 Jan 2013 04:59:24	1225070	15522050	hadcm3n_3mio_1940_40_008260580_1	129,600	152,879	1.1796
08 Jan 2013 20:15:10	1225070	15522050	hadcm3n_3mio_1940_40_008260580_1	103,680	122,689	1.1833
08 Jan 2013 03:40:54	1225070	15522050	hadcm3n_3mio_1940_40_008260580_1	77,760	91,669	1.1789
06 Jan 2013 03:14:54	1225070	15522050	hadcm3n_3mio_1940_40_008260580_1	51,840	62,560	1.2068
04 Jan 2013 04:39:28	1225070	15522050	hadcm3n_3mio_1940_40_008260580_1	25,920	30,372	1.1718