Task 15529516

Name	hadcm3n_398h_1940_40_008268095_1
Workunit	8423219
Created	12 Jan 2013, 9:13:25 UTC
Sent	12 Jan 2013, 9:13:28 UTC
Report deadline	13 Apr 2013, 16:40:39 UTC
Received	29 Sep 2013, 20:01:20 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1187881
Run time	51 days 15 hours 20 min 31 sec
CPU time	49 days 22 hours 41 min 14 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	0.62 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/398hko.pjf0c10 is not a valid UM file. Error converting file to netcdf: dataout/398hko.pjf0c10 Error: Input file: dataout/398hko.pif0c10 is not a valid UM file. Error converting file to netcdf: dataout/398hko.pif0c10 Error: Input file: dataout/398hko.pff0c10 is not a valid UM file. Error converting file to netcdf: dataout/398hko.pff0c10 Error: Input file: dataout/398hko.pcf0c10 is not a valid UM file. Error converting file to netcdf: dataout/398hko.pcf0c10 Error: Input file: dataout/398hko.pbf0c10 is not a valid UM file. Error converting file to netcdf: dataout/398hko.pbf0c10 Error: Input file: dataout/398hko.paf0c10 is not a valid UM file. Error converting file to netcdf: dataout/398hko.paf0c10 Error: Input file: dataout/398hka.phf0c10 is not a valid UM file. Error converting file to netcdf: dataout/398hka.phf0c10 Error: Input file: dataout/398hka.pgf0c10 is not a valid UM file. Error converting file to netcdf: dataout/398hka.pgf0c10 Error: Input file: dataout/398hka.pef0c10 is not a valid UM file. Error converting file to netcdf: dataout/398hka.pef0c10 Error: Input file: dataout/398hka.pdf0c10 is not a valid UM file. Error converting file to netcdf: dataout/398hka.pdf0c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:42:09 (5498): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:21:14 (2591): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 SETPOS: Seek Failed: No space left on device SETPOS: Unit 22 to Word Address 3715072 Failed with Error Code -1 Model crashed: SETPOS: Unit 22 to Word Address 3715072 Failed with Error Code -1 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 SETPOS: Seek Failed: No space left on device SETPOS: Unit 22 to Word Address 202Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14822, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_398h_1940_40_008268095/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450E2C Unknown Unknown Unknown hadcm3n_um_6.07_i 0824EBA6 Unknown Unknown Unknown hadcm3n_um_6.07_i 082712DC Unknown Unknown Unknown hadcm3n_um_6.07_i 08177076 Unknown Unknown Unknown hadcm3n_um_6.07_i 081795D4 Unknown Unknown Unknown hadcm3n_um_6.07_i 081908AF Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F7550935 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14822, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Sep 2013 17:51:09	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	492,480	4,202,450	8.5332
15 Sep 2013 16:31:21	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	466,560	3,992,247	8.5568
22 Aug 2013 15:31:39	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	440,640	3,758,776	8.5303
14 Aug 2013 18:58:32	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	414,720	3,524,047	8.4974
14 Aug 2013 18:00:07	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	388,800	3,293,533	8.4710
14 Aug 2013 18:00:08	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	362,880	3,060,580	8.4341
23 Jul 2013 22:12:51	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	336,960	2,831,455	8.4029
07 Jul 2013 16:37:56	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	311,040	2,605,368	8.3763
11 Jun 2013 16:21:55	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	285,120	2,401,500	8.4228
02 Jun 2013 15:00:58	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	259,200	2,195,997	8.4722
18 May 2013 21:37:52	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	233,280	1,993,539	8.5457
12 May 2013 23:03:55	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	207,360	1,769,487	8.5334
02 May 2013 11:31:37	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	181,440	1,545,543	8.5182
21 Mar 2013 14:50:27	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	155,520	1,323,020	8.5071
10 Mar 2013 11:01:56	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	129,600	1,106,681	8.5392
22 Feb 2013 21:29:07	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	103,680	890,304	8.5870
12 Feb 2013 00:47:41	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	77,760	674,309	8.6717
09 Feb 2013 02:20:16	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	51,840	458,090	8.8366
30 Jan 2013 11:33:32	1187881	15529516	hadcm3n_398h_1940_40_008268095_1	25,920	240,938	9.2954