Task 15549321

Name	hadcm3n_n0eu_1880_40_008286428_0
Workunit	8437563
Created	17 Jan 2013, 19:50:05 UTC
Sent	17 Jan 2013, 20:20:22 UTC
Report deadline	19 Apr 2013, 3:47:33 UTC
Received	28 Feb 2013, 19:58:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1213041
Run time	12 days 13 hours 4 min 34 sec
CPU time	11 days 20 hours 1 min 51 sec
Validate state	Invalid
Credit	5,598.72
Device peak FLOPS	3.05 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4228, iMonCtr=1 Model crashSuspended CPDN Monitor - Suspend request from BOINC... 16:55:09 (3452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:12:27 (4940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:27:42 (3456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:53 (4616): No heartbeat from core client for 30 sec - exiting 16:55:55 (4616): No heartbeat from core client for 30 sec - exiting 16:55:56 (4616): No heartbeat from core client for 30 sec - exiting 16:55:57 (4616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:17:15 (4484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:20:25 (3540): No heartbeat from core client for 30 sec - exiting 12:20:26 (3540): No heartbeat from core client for 30 sec - exiting 12:20:27 (3540): No heartbeat from core client for 30 sec - exiting 12:20:28 (3540): No heartbeat from core client for 30 sec - exiting 12:20:29 (3540): No heartbeat from core client for 30 sec - exiting 12:20:30 (3540): No heartbeat from core client for 30 sec - exiting 12:20:32 (3540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:20:33 (3540): No heartbeat from core client for 30 sec - exiting 16:45:42 (3964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:44:05 (4444): No heartbeat from core client for 30 sec - exiting 17:44:06 (4444): No heartbeat from core client for 30 sec - exiting 17:44:07 (4444): No heartbeat from core client for 30 sec - exiting 17:44:08 (4444): No heartbeat from core client for 30 sec - exiting 17:44:09 (4444): No heartbeat from core client for 30 sec - exiting 17:44:10 (4444): No heartbeat from core client for 30 sec - exiting 17:44:11 (4444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:49:59 (3388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:01:58 (3196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:43:35 (4116): No heartbeat from core client for 30 sec - exiting 10:43:36 (4116): No heartbeat from core client for 30 sec - exiting 10:43:37 (4116): No heartbeat from core client for 30 sec - exiting 10:43:38 (4116): No heartbeat from core client for 30 sec - exiting 10:43:39 (4116): No heartbeat from core client for 30 sec - exiting 10:43:40 (4116): No heartbeat from core client for 30 sec - exiting 10:43:41 (4116): No heartbeat from core client for 30 sec - exiting 10:43:42 (4116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:16:28 (4336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4524, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4524, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4524, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4524, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4524, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4524, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish 17:11:40 (3620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Feb 2013 19:33:51	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	466,560	971,251	2.0817
23 Feb 2013 21:30:01	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	440,640	915,600	2.0779
22 Feb 2013 16:20:44	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	414,720	859,831	2.0733
20 Feb 2013 21:21:32	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	388,800	804,544	2.0693
19 Feb 2013 13:20:27	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	362,880	749,009	2.0641
17 Feb 2013 19:43:12	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	336,960	694,564	2.0613
16 Feb 2013 15:05:36	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	311,040	638,572	2.0530
13 Feb 2013 17:51:28	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	285,120	584,204	2.0490
10 Feb 2013 18:01:46	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	259,200	529,952	2.0446
09 Feb 2013 12:02:37	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	233,280	476,600	2.0430
04 Feb 2013 20:37:56	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	207,360	421,698	2.0337
02 Feb 2013 21:03:32	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	181,440	365,706	2.0156
31 Jan 2013 20:58:13	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	155,520	310,386	1.9958
29 Jan 2013 16:45:46	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	129,600	257,083	1.9837
27 Jan 2013 14:52:14	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	103,680	203,543	1.9632
25 Jan 2013 22:07:54	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	77,760	152,166	1.9569
22 Jan 2013 22:51:18	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	51,840	101,415	1.9563
19 Jan 2013 17:00:59	1213041	15549321	hadcm3n_n0eu_1880_40_008286428_0	25,920	50,253	1.9388