Task 13019174

Name	hadcm3n_t02k_1940_40_007313506_0
Workunit	7510936
Created	28 Jun 2011, 11:02:23 UTC
Sent	28 Jun 2011, 11:02:39 UTC
Report deadline	27 Sep 2011, 18:29:50 UTC
Received	25 Jul 2011, 21:28:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1114842
Run time	19 days 19 hours 28 min 5 sec
CPU time	17 days 9 hours 8 min 27 sec
Validate state	Invalid
Credit	8,398.08
Device peak FLOPS	3.09 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:57:16 (5460): No heartbeat from core client for 30 sec - exiting 20:57:17 (5460): No heartbeat from core client for 30 sec - exiting 20:57:18 (5460): No heartbeat from core client for 30 sec - exiting 20:57:19 (5460): No heartbeat from core client for 30 sec - exiting 20:57:20 (5460): No heartbeat from core client for 30 sec - exiting 20:57:21 (5460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:57:22 (5460): No heartbeat from core client for 30 sec - exiting 20:57:23 (5460): No heartbeat from core client for 30 sec - exiting 20:57:24 (5460): No heartbeat from core client for 30 sec - exiting 20:57:25 (5460): No heartbeat from core client for 30 sec - exiting 20:57:26 (5460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:49:00 (7144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4432, iMonCtr=1 Model crash detected, will try to restart... 08:51:01 (5932): No heartbeat from core client for 30 sec - exiting 08:51:02 (5932): No heartbeat from core client for 30 sec - exiting 08:51:03 (5932): No heartbeat from core client for 30 sec - exiting 08:51:04 (5932): No heartbeat from core client for 30 sec - exiting 08:51:05 (5932): No heartbeat from core client for 30 sec - exiting 08:51:06 (5932): No heartbeat from core client for 30 sec - exiting 08:51:07 (5932): No heartbeat from core client for 30 sec - exiting 08:51:08 (5932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:13:27 (3476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:19:07 (8516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:45:44 (5656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:35:49 (2688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:18:51 (6984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:12:04 (6828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:25:32 (7208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6276, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6276, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6276, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6276, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6276, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:14:06 (3348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:14:07 (3348): No heartbeat from core client for 30 sec - exiting 12:14:08 (3348): No heartbeat from core client for 30 sec - exiting 12:14:09 (3348): No heartbeat from core client for 30 sec - exiting 12:14:10 (3348): No heartbeat from core client for 30 sec - exiting 12:14:11 (3348): No heartbeat from core client for 30 sec - exiting 12:14:12 (3348): No heartbeat from core client for 30 sec - exiting 12:14:13 (3348): No heartbeat from core client for 30 sec - exiting 12:14:14 (3348): No heartbeat from core client for 30 sec - exiting 12:14:15 (3348): No heartbeat from core client for 30 sec - exiting 12:14:16 (3348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6732, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 22:47:45	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	699,840	1,501,129	2.1450
25 Jul 2011 21:13:00	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	673,920	1,454,363	2.1581
25 Jul 2011 20:25:16	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	648,000	1,403,801	2.1664
25 Jul 2011 19:03:59	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	622,080	1,348,976	2.1685
25 Jul 2011 19:03:59	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	596,160	1,299,037	2.1790
25 Jul 2011 18:51:07	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	570,240	1,245,604	2.1844
25 Jul 2011 17:51:42	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	544,320	1,185,249	2.1775
25 Jul 2011 17:19:16	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	518,400	1,120,103	2.1607
25 Jul 2011 16:03:22	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	492,480	1,057,704	2.1477
25 Jul 2011 15:27:49	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	466,560	1,002,056	2.1478
25 Jul 2011 13:04:17	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	440,640	941,258	2.1361
25 Jul 2011 13:04:17	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	414,720	887,460	2.1399
25 Jul 2011 13:04:17	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	388,800	832,692	2.1417
25 Jul 2011 13:04:17	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	362,880	776,427	2.1396
25 Jul 2011 13:04:17	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	336,960	726,236	2.1553
25 Jul 2011 13:04:17	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	311,040	675,358	2.1713
10 Jul 2011 23:55:14	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	285,120	622,041	2.1817
10 Jul 2011 06:20:54	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	259,200	570,134	2.1996
09 Jul 2011 13:47:02	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	233,280	516,622	2.2146
08 Jul 2011 15:06:28	1114842	13019174	hadcm3n_t02k_1940_40_007313506_0	207,360	465,793	2.2463