Task 14278394

Name	hadcm3n_o0lq_1980_40_007831348_0
Workunit	7986460
Created	17 Mar 2012, 15:55:25 UTC
Sent	17 Mar 2012, 15:56:59 UTC
Report deadline	16 Jun 2012, 23:24:10 UTC
Received	14 Jun 2012, 15:47:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1189258
Run time	12 days 14 hours 51 min 44 sec
CPU time	10 days 14 hours 36 min 20 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	2.24 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3680, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5604, selfPID=5604, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3976, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o0lqko.pji3c10 Error converting file to netcdf: dataout/o0lqko.pii3c10 Error converting file to netcdf: dataout/o0lqko.pfi3c10 Error converting file to netcdf: dataout/o0lqka.phi3c10 Error converting file to netcdf: dataout/o0lqka.pgi3c10 Error converting file to netcdf: dataout/o0lqka.pei3c10 Error converting file to netcdf: dataout/o0lqka.pdi3c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... C14:41:53 (4112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4088, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5952, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3828, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4536, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3032, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2772, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1340, iMonCtr=1 Model crash detected, will try to restart... 17:24:41 (3544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1360, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3772, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3260, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3756, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:01:17 (2964): No heartbeat from core client for 30 sec - exiting 16:01:18 (2964): No heartbeat from core client for 30 sec - exiting 16:01:19 (2964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:53:06 (3460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:03:06 (3432): No heartbeat from core client for 30 sec - exiting 15:03:07 (3432): No heartbeat from core client for 30 sec - exiting 15:03:08 (3432): No heartbeat from core client for 30 sec - exiting 15:03:09 (3432): No heartbeat from core client for 30 sec - exiting 15:03:10 (3432): No heartbeat from core client for 30 sec - exiting 15:03:11 (3432): No heartbeat from core client for 30 sec - exiting 15:03:13 (3432): No heartbeat from core client for 30 sec - exiting 15:03:14 (3432): No heartbeat from core client for 30 sec - exiting 15:03:15 (3432): No heartbeat from core client for 30 sec - exiting 15:03:16 (3432): No heartbeat from core client for 30 sec - exiting 15:03:17 (3432): No heartbeat from core client for 30 sec - exiting 15:03:18 (3432): No heartbeat from core client for 30 sec - exiting 15:03:19 (3432): No heartbeat from core client for 30 sec - exiting 15:03:20 (3432): No heartbeat from core client for 30 sec - exiting 15:03:21 (3432): No heartbeat from core client for 30 sec - exiting 15:03:22 (3432): No heartbeat from core client for 30 sec - exiting 15:03:23 (3432): No heartbeat from core client for 30 sec - exiting 15:03:25 (3432): No heartbeat from core client for 30 sec - exiting 15:03:26 (3432): No heartbeat from core client for 30 sec - exiting 15:03:27 (3432): No heartbeat from core client for 30 sec - exiting 15:03:28 (3432): No heartbeat from core client for 30 sec - exiting 15:03:29 (3432): No heartbeat from core client for 30 sec - exiting 15:03:30 (3432): No heartbeat from core client for 30 sec - exiting 15:03:31 (3432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:03:32 (3432): No heartbeat from core client for 30 sec - exiting 15:04:52 (1204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:21:34 (3548): No heartbeat from core client for 30 sec - exiting 17:21:35 (3548): No heartbeat from core client for 30 sec - exiting 17:21:36 (3548): No heartbeat from core client for 30 sec - exiting 17:21:37 (3548): No heartbeat from core client for 30 sec - exiting 17:21:38 (3548): No heartbeat from core client for 30 sec - exiting 17:21:39 (3548): No heartbeat from core client for 30 sec - exiting 17:21:40 (3548): No heartbeat from core client for 30 sec - exiting 17:21:41 (3548): No heartbeat from core client for 30 sec - exiting 17:21:42 (3548): No heartbeat from core client for 30 sec - exiting 17:21:44 (3548): No heartbeat from core client for 30 sec - exiting 17:21:45 (3548): No heartbeat from core client for 30 sec - exiting 17:21:46 (3548): No heartbeat from core client for 30 sec - exiting 17:21:47 (3548): No heartbeat from core client for 30 sec - exiting 17:21:48 (3548): No heartbeat from core client for 30 sec - exiting 17:21:49 (3548): No heartbeat from core client for 30 sec - exiting 17:21:50 (3548): No heartbeat from core client for 30 sec - exiting 17:21:51 (3548): No heartbeat from core client for 30 sec - exiting 17:21:52 (3548): No heartbeat from core client for 30 sec - exiting 17:21:53 (3548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:21:54 (3548): No heartbeat from core client for 30 sec - exiting 17:22:46 (2028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:10:15 (3576): No heartbeat from core client for 30 sec - exiting 07:10:16 (3576): No heartbeat from core client for 30 sec - exiting 07:10:17 (3576): No heartbeat from core client for 30 sec - exiting 07:10:18 (3576): No heartbeat from core client for 30 sec - exiting 07:10:19 (3576): No heartbeat from core client for 30 sec - exiting 07:10:20 (3576): No heartbeat from core client for 30 sec - exiting 07:10:21 (3576): No heartbeat from core client for 30 sec - exiting 07:10:22 (3576): No heartbeat from core client for 30 sec - exiting 07:10:23 (3576): No heartbeat from core client for 30 sec - exiting 07:10:24 (3576): No heartbeat from core client for 30 sec - exiting 07:10:26 (3576): No heartbeat from core client for 30 sec - exiting 07:10:27 (3576): No heartbeat from core client for 30 sec - exiting 07:10:28 (3576): No heartbeat from core client for 30 sec - exiting 07:10:29 (3576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:10:30 (3576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 15:51:17 (3860): No heartbeat from core client for 30 sec - exiting 15:51:18 (3860): No heartbeat from core client for 30 sec - exiting 15:51:19 (3860): No heartbeat from core client for 30 sec - exiting 15:51:20 (3860): No heartbeat from core client for 30 sec - exiting 15:51:21 (3860): No heartbeat from core client for 30 sec - exiting 15:51:22 (3860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3068, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:48:30 (2712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1208, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2616, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3708, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3384, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3812, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:29:30 (3592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:28:33 (4696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:11:52 (3888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3920, iMonCtr=1 Model crash detected, will try to restart... 15:36:31 (3772): No heartbeat from core client for 30 sec - exiting 15:36:32 (3772): No heartbeat from core client for 30 sec - exiting 15:36:33 (3772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:36:34 (3772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=552, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:22:56 (3240): No heartbeat from core client for 30 sec - exiting 09:22:57 (3240): No heartbeat from core client for 30 sec - exiting 09:22:58 (3240): No heartbeat from core client for 30 sec - exiting 09:22:59 (3240): No heartbeat from core client for 30 sec - exiting 09:23:00 (3240): No heartbeat from core client for 30 sec - exiting 09:23:01 (3240): No heartbeat from core client for 30 sec - exiting 09:23:02 (3240): No heartbeat from core client for 30 sec - exiting 09:23:03 (3240): No heartbeat from core client for 30 sec - exiting 09:23:04 (3240): No heartbeat from core client for 30 sec - exiting 09:23:05 (3240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2492, iMonCtr=1 Model crash detected, will try to restart... 17:30:45 (2564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:46:52 (2508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:46:54 (2508): No heartbeat from core client for 30 sec - exiting 20:46:55 (2508): No heartbeat from core client for 30 sec - exiting 20:46:56 (2508): No heartbeat from core client for 30 sec - exiting 20:46:57 (2508): No heartbeat from core client for 30 sec - exiting 20:48:12 (6868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=1 Model crash detected, will try to restart... forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=1 Model crash detected, will try to restart... forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=1 Model crash detected, will try to restart... forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=1 Model crash detected, will try to restart... forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3832, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Jun 2012 19:57:58	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	492,480	904,289	1.8362
05 Jun 2012 16:25:49	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	466,560	853,370	1.8291
28 May 2012 15:48:08	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	440,640	804,828	1.8265
26 May 2012 11:59:46	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	414,720	753,643	1.8172
23 May 2012 14:24:01	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	388,800	706,878	1.8181
20 May 2012 16:01:10	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	362,880	659,416	1.8172
16 May 2012 17:32:13	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	336,960	611,759	1.8155
13 May 2012 10:14:21	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	311,040	564,513	1.8149
06 May 2012 15:52:11	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	285,120	515,988	1.8097
04 May 2012 12:09:05	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	259,200	470,155	1.8139
30 Apr 2012 16:31:51	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	233,280	422,860	1.8127
28 Apr 2012 16:50:09	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	207,360	376,624	1.8163
25 Apr 2012 11:51:43	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	181,440	331,333	1.8261
21 Apr 2012 20:59:53	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	155,520	281,089	1.8074
18 Apr 2012 15:57:03	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	129,600	228,907	1.7663
15 Apr 2012 11:01:35	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	103,680	183,394	1.7688
08 Apr 2012 12:32:55	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	77,760	139,226	1.7905
03 Apr 2012 17:40:31	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	51,840	93,393	1.8016
24 Mar 2012 23:34:35	1189258	14278394	hadcm3n_o0lq_1980_40_007831348_0	25,920	46,582	1.7971