Task 15465172

Name	hadcm3n_zl1m_1880_40_008250683_2
Workunit	8405807
Created	28 Nov 2012, 12:43:15 UTC
Sent	28 Nov 2012, 12:43:24 UTC
Report deadline	27 Feb 2013, 20:10:35 UTC
Received	17 Dec 2012, 15:22:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1253823
Run time	2 days 13 hours 11 min 36 sec
CPU time	1 days 0 hours 19 min 22 sec
Validate state	Invalid
Credit	311.04
Device peak FLOPS	2.45 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 02:14:22 (8228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:14:24 (8228): No heartbeat from core client for 30 sec - exiting 02:14:27 (8228): No heartbeat from core client for 30 sec - exiting 02:14:30 (8228): No heartbeat from core client for 30 sec - exiting 02:16:59 (7664): No heartbeat from core client for 30 sec - exiting 02:17:00 (7664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:32:27 (7384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:32:30 (7384): No heartbeat from core client for 30 sec - exiting 02:47:32 (9884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:47:35 (9884): No heartbeat from core client for 30 sec - exiting 02:47:38 (9884): No heartbeat from core client for 30 sec - exiting 03:20:40 (6936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:20:43 (6936): No heartbeat from core client for 30 sec - exiting 03:55:02 (400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:55:04 (400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 03:30:50 (5320): No heartbeat from core client for 30 sec - exiting 03:30:51 (5320): No heartbeat from core client for 30 sec - exiting 03:30:52 (5320): No heartbeat from core client for 30 sec - exiting 03:30:53 (5320): No heartbeat from core client for 30 sec - exiting 03:30:54 (5320): No heartbeat from core client for 30 sec - exiting 03:30:55 (5320): No heartbeat from core client for 30 sec - exiting 03:30:56 (5320): No heartbeat from core client for 30 sec - exiting 03:30:57 (5320): No heartbeat from core client for 30 sec - exiting 03:30:58 (5320): No heartbeat from core client for 30 sec - exiting 03:30:59 (5320): No heartbeat from core client for 30 sec - exiting 03:31:00 (5320): No heartbeat from core client for 30 sec - exiting 03:31:01 (5320): No heartbeat from core client for 30 sec - exiting 03:31:02 (5320): No heartbeat from core client for 30 sec - exiting 03:31:03 (5320): No heartbeat from core client for 30 sec - exiting 03:31:04 (5320): No heartbeat from core client for 30 sec - exiting 03:31:05 (5320): No heartbeat from core client for 30 sec - exiting 03:31:06 (5320): No heartbeat from core client for 30 sec - exiting 03:31:07 (5320): No heartbeat from core client for 30 sec - exiting 03:31:08 (5320): No heartbeat from core client for 30 sec - exiting 03:31:09 (5320): No heartbeat from core client for 30 sec - exiting 03:31:10 (5320): No heartbeat from core client for 30 sec - exiting 03:31:11 (5320): No heartbeat from core client for 30 sec - exiting 03:31:12 (5320): No heartbeat from core client for 30 sec - exiting 03:31:13 (5320): No heartbeat from core client for 30 sec - exiting 03:31:14 (5320): No heartbeat from core client for 30 sec - exiting 03:31:15 (5320): No heartbeat from core client for 30 sec - exiting 03:31:16 (5320): No heartbeat from core client for 30 sec - exiting 03:31:17 (5320): No heartbeat from core client for 30 sec - exiting 03:31:18 (5320): No heartbeat from core client for 30 sec - exiting 03:31:19 (5320): No heartbeat from core client for 30 sec - exiting 03:31:20 (5320): No heartbeat from core client for 30 sec - exiting 03:31:21 (5320): No heartbeat from core client for 30 sec - exiting 03:31:22 (5320): No heartbeat from core client for 30 sec - exiting 03:31:23 (5320): No heartbeat from core client for 30 sec - exiting 03:31:24 (5320): No heartbeat from core client for 30 sec - exiting 03:31:25 (5320): No heartbeat from core client for 30 sec - exiting 03:31:26 (5320): No heartbeat from core client for 30 sec - exiting 03:31:27 (5320): No heartbeat from core client for 30 sec - exiting 03:31:28 (5320): No heartbeat from core client for 30 sec - exiting 03:31:29 (5320): No heartbeat from core client for 30 sec - exiting 03:31:30 (5320): No heartbeat from core client for 30 sec - exiting 03:31:31 (5320): No heartbeat from core client for 30 sec - exiting 03:31:32 (5320): No heartbeat from core client for 30 sec - exiting 03:31:33 (5320): No heartbeat from core client for 30 sec - exiting 03:31:34 (5320): No heartbeat from core client for 30 sec - exiting 03:31:35 (5320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:27:36 (5288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:27:37 (5288): No heartbeat from core client for 30 sec - exiting 06:27:39 (5288): No heartbeat from core client for 30 sec - exiting 06:32:52 (3944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:32:54 (3944): No heartbeat from core client for 30 sec - exiting 06:36:34 (5428): No heartbeat from core client for 30 sec - exiting 06:36:36 (5428): No heartbeat from core client for 30 sec - exiting 06:36:39 (5428): No heartbeat from core client for 30 sec - exiting 06:36:40 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:22:14 (4800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:22:16 (4800): No heartbeat from core client for 30 sec - exiting 07:33:11 (4844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:33:14 (4844): No heartbeat from core client for 30 sec - exiting 07:52:52 (5188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:52:54 (5188): No heartbeat from core client for 30 sec - exiting 07:55:35 (5908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:58:54 (5672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:07:43 (2884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:07:47 (2884): No heartbeat from core client for 30 sec - exiting 08:14:33 (5852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:35 (5852): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2676, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2676, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2676, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2676, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:05:06 (2676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:45:31 (9444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:48:59 (10184): No heartbeat from core client for 30 sec - exiting 06:49:00 (10184): No heartbeat from core client for 30 sec - exiting 06:49:02 (10184): No heartbeat from core client for 30 sec - exiting 06:49:06 (10184): No heartbeat from core client for 30 sec - exiting 06:49:11 (10184): No heartbeat from core client for 30 sec - exiting 06:49:13 (10184): No heartbeat from core client for 30 sec - exiting 06:49:16 (10184): No heartbeat from core client for 30 sec - exiting 06:49:19 (10184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:16 (9704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:07:44 (6700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:07:46 (6700): No heartbeat from core client for 30 sec - exiting 07:31:55 (6892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:31:58 (6892): No heartbeat from core client for 30 sec - exiting 08:02:21 (2904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:02:23 (2904): No heartbeat from core client for 30 sec - exiting 08:16:25 (8288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:30:58 (10172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:34:59 (8348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:52 (2024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:53 (2024): No heartbeat from core client for 30 sec - exiting 08:45:21 (9248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:45:22 (9248): No heartbeat from core client for 30 sec - exiting 08:47:58 (8560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:23 (5068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:13:36 (10116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:13:37 (10116): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5860, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5860, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Nov 2012 07:27:12	1253823	15465172	hadcm3n_zl1m_1880_40_008250683_2	25,920	58,033	2.2389