Task 15459764

Name	hadcm3n_zbpx_1880_40_008246442_3
Workunit	8401566
Created	24 Nov 2012, 0:54:11 UTC
Sent	24 Nov 2012, 0:54:36 UTC
Report deadline	23 Feb 2013, 8:21:47 UTC
Received	23 Dec 2012, 10:43:27 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1025553
Run time	14 days 0 hours 5 min 13 sec
CPU time	12 days 11 hours 21 min 54 sec
Validate state	Invalid
Credit	5,598.72
Device peak FLOPS	3.31 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 22:24:27 (3892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:03:02 (2668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:09 (4236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:27:24 (4764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:27:25 (4764): No heartbeat from core client for 30 sec - exiting 12:27:44 (2016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:27:48 (2016): No heartbeat from core client for 30 sec - exiting 22:57:13 (1628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:38:03 (4820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:49:27 (4444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:49:36 (4444): No heartbeat from core client for 30 sec - exiting 04:49:37 (4444): No heartbeat from core client for 30 sec - exiting 04:49:38 (4444): No heartbeat from core client for 30 sec - exiting 04:49:39 (4444): No heartbeat from core client for 30 sec - exiting 09:29:35 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:27:43 (2800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:27:45 (4676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:24:14 (4188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:40:43 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:10:50 (772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:10:58 (772): No heartbeat from core client for 30 sec - exiting 16:10:59 (772): No heartbeat from core client for 30 sec - exiting 16:11:00 (772): No heartbeat from core client for 30 sec - exiting 16:11:01 (772): No heartbeat from core client for 30 sec - exiting 16:11:02 (772): No heartbeat from core client for 30 sec - exiting 16:11:03 (772): No heartbeat from core client for 30 sec - exiting 16:11:04 (772): No heartbeat from core client for 30 sec - exiting 16:11:05 (772): No heartbeat from core client for 30 sec - exiting 16:11:06 (772): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:41:58 (4804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:41:59 (4804): No heartbeat from core client for 30 sec - exiting 00:42:00 (4804): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 13:33:00 (1176): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 17:20:41 (5092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:20:42 (1156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:20:45 (1156): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 03:18:28 (1880): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=988, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=988, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=988, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=988, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:54:48 (3696): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:19:57 (3344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:20:05 (3344): No heartbeat from core client for 30 sec - exiting 06:20:06 (3344): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:13:01 (5088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:39:14 (4264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:39:15 (4264): No heartbeat from core client for 30 sec - exiting 13:39:16 (4264): No heartbeat from core client for 30 sec - exiting 13:39:17 (4264): No heartbeat from core client for 30 sec - exiting 13:39:18 (4264): No heartbeat from core client for 30 sec - exiting 13:39:19 (4264): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:12:06 (504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:46:21 (4224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:01:41 (1448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:01:43 (1448): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Dec 2012 02:56:48	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	466,560	1,148,491	2.4616
22 Dec 2012 08:54:25	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	440,640	1,112,137	2.5239
21 Dec 2012 21:55:14	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	414,720	1,074,759	2.5915
21 Dec 2012 08:06:33	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	388,800	1,037,477	2.6684
20 Dec 2012 21:25:01	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	362,880	1,000,048	2.7559
01 Dec 2012 14:52:38	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	336,960	473,133	1.4041
30 Nov 2012 16:54:35	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	311,040	437,796	1.4075
29 Nov 2012 03:06:17	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	285,120	401,352	1.4077
28 Nov 2012 13:48:00	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	259,200	364,699	1.4070
27 Nov 2012 23:29:37	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	233,280	328,341	1.4075
27 Nov 2012 12:54:30	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	207,360	292,292	1.4096
27 Nov 2012 02:42:24	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	181,440	256,228	1.4122
26 Nov 2012 16:22:35	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	155,520	220,358	1.4169
26 Nov 2012 06:14:33	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	129,600	184,403	1.4229
25 Nov 2012 19:50:19	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	103,680	148,407	1.4314
25 Nov 2012 09:30:33	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	77,760	111,694	1.4364
24 Nov 2012 22:53:11	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	51,840	74,459	1.4363
24 Nov 2012 12:44:18	1025553	15459764	hadcm3n_zbpx_1880_40_008246442_3	25,920	37,621	1.4514