Task 15749169

Name	hadcm3n_o2af_2140_40_008269536_4
Workunit	8424660
Created	24 Apr 2013, 13:31:01 UTC
Sent	24 Apr 2013, 13:31:08 UTC
Report deadline	24 Jul 2013, 20:58:19 UTC
Received	20 May 2013, 12:19:53 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1181449
Run time	12 days 8 hours 0 min 50 sec
CPU time	11 days 2 hours 6 min 48 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.31 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6404, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6404, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4152, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 08:18:25 PM No files match the supplied pattern. MainError: 08:18:25 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 12:41:23 AM No files match the supplied pattern. MainError: 12:41:23 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, seCPDN Monitor - Quit request from BOINC... MainError: 03:25:56 PM No files match the supplied pattern. MainError: 03:25:56 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... 21:52:05 (6128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... MainError: 10:39:29 AM No files match the supplied pattern. MainError: 10:39:29 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 07:48:57 PM No files match the supplied pattern. MainError: 07:48:57 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 11:24:06 AM No files match the supplied pattern. MainError: 11:24:06 AM No files match the supplied pattern. MainError: 10:32:15 PM No files match the supplied pattern. MainError: 10:32:15 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 08:39:15 PM No files match the supplied pattern. MainError: 08:39:15 PM No files match the supplied pattern. MainError: 04:51:12 PM No files match the supplied pattern. MainError: 04:51:12 PM No files match the supplied pattern. MainError: 01:36:37 AM No files match the supplied pattern. MainError: 01:36:37 AM No files match the supplied pattern. Error converting file to netcdf: dataout/o2afka.ph11c10 Error converting file to netcdf: dataout/o2afka.pg11c10 Error converting file to netcdf: dataout/o2afka.pe11c10 MainError: 11:22:23 AM No files match the supplied pattern. MainError: 11:22:23 AM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 May 2013 12:22:07	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	777,600	1,021,030	1.3131
20 May 2013 02:18:38	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	751,680	986,085	1.3118
17 May 2013 17:20:23	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	725,760	951,232	1.3107
16 May 2013 21:35:21	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	699,840	917,536	1.3111
15 May 2013 23:08:29	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	673,920	883,970	1.3117
15 May 2013 11:26:26	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	648,000	849,418	1.3108
14 May 2013 20:29:42	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	622,080	815,234	1.3105
13 May 2013 10:39:54	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	596,160	780,686	1.3095
12 May 2013 15:30:37	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	570,240	747,948	1.3116
09 May 2013 00:43:47	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	544,320	714,180	1.3121
07 May 2013 20:19:24	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	518,400	680,412	1.3125
07 May 2013 10:47:10	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	492,480	647,138	1.3140
07 May 2013 01:34:43	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	466,560	614,539	1.3172
06 May 2013 16:35:47	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	440,640	581,818	1.3204
05 May 2013 20:13:36	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	414,720	547,997	1.3214
05 May 2013 10:20:21	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	388,800	513,333	1.3203
04 May 2013 17:23:14	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	362,880	479,486	1.3213
03 May 2013 17:26:55	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	336,960	445,511	1.3221
02 May 2013 22:30:22	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	311,040	410,837	1.3208
02 May 2013 12:26:54	1181449	15749169	hadcm3n_o2af_2140_40_008269536_4	285,120	376,703	1.3212