Task 15502254

Name	hadcm3n_o49g_2140_40_008269427_0
Workunit	8424551
Created	23 Dec 2012, 22:58:09 UTC
Sent	25 Dec 2012, 0:04:45 UTC
Report deadline	26 Mar 2013, 7:31:56 UTC
Received	1 Mar 2013, 1:23:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1227604
Run time	16 days 7 hours 46 min 31 sec
CPU time	13 days 4 hours 51 min 12 sec
Validate state	Invalid
Credit	9,642.24
Device peak FLOPS	2.89 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3744, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6932, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 12:22:06 AM No files match the supplied pattern. MainError: 12:22:06 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:54:08 (5484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:56:28 (8128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 07:44:36 AM No files match the supplied pattern. MainError: 07:44:36 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 12:58:32 AM No files match the supplied pattern. MainError: 12:58:32 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 06:20:27 PM No files match the supplied pattern. MainError: 06:20:27 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 03:03:16 PM No files match the supplied pattern. MainError: 03:03:16 PM No files match the supplied pattern. MainError: 01:47:56 AM No files match the supplied pattern. MainError: 01:47:56 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 10:55:32 AM No files match the supplied pattern. MainError: 10:55:32 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 10:49:04 PM No files match the supplied pattern. MainError: 10:49:04 PM No files match the supplied pattern. MainError: 10:16:52 AM No files match the supplied pattern. MainError: 10:16:52 AM No files match the supplied pattern. MainError: 09:41:03 PM No files match the supplied pattern. MainError: 09:41:03 PM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Error converting file to netcdf: dataout/o49gko.pj10c10 Error converting file to netcdf: dataout/o49gka.ph12c10 Error converting file to netcdf: dataout/o49gka.pg12c10 Error converting file to netcdf: dataout/o49gka.pe12c10 MainError: 12:12:30 AM No files match the supplied pattern. MainError: 12:12:30 AM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Mar 2013 00:25:26	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	803,520	1,147,211	1.4277
17 Feb 2013 22:05:41	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	751,680	1,073,509	1.4281
17 Feb 2013 10:42:19	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	725,760	1,036,457	1.4281
16 Feb 2013 23:04:57	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	699,840	999,546	1.4282
15 Feb 2013 11:12:36	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	673,920	962,468	1.4282
14 Feb 2013 02:29:20	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	648,000	923,556	1.4252
13 Feb 2013 15:35:39	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	622,080	887,095	1.4260
11 Feb 2013 18:21:46	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	596,160	844,726	1.4169
11 Feb 2013 00:59:28	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	570,240	807,188	1.4155
10 Feb 2013 07:46:41	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	544,320	770,436	1.4154
08 Feb 2013 12:25:04	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	518,400	734,008	1.4159
07 Feb 2013 15:51:36	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	492,480	698,039	1.4174
05 Feb 2013 22:25:40	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	466,560	656,248	1.4066
03 Feb 2013 19:14:50	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	440,640	619,924	1.4069
28 Jan 2013 23:42:59	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	414,720	582,779	1.4052
28 Jan 2013 08:19:27	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	388,800	546,446	1.4055
27 Jan 2013 20:46:28	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	362,880	509,863	1.4050
25 Jan 2013 21:32:29	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	336,960	473,006	1.4037
24 Jan 2013 16:25:52	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	311,040	436,032	1.4019
24 Jan 2013 13:55:22	1227604	15502254	hadcm3n_o49g_2140_40_008269427_0	285,120	399,684	1.4018