Task 15487005

Name	hadcm3n_3ir2_1940_40_008259326_0
Workunit	8414450
Created	20 Dec 2012, 14:10:44 UTC
Sent	20 Dec 2012, 14:11:22 UTC
Report deadline	21 Mar 2013, 21:38:33 UTC
Received	16 Jan 2013, 23:56:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1148818
Run time	16 days 8 hours 32 min 53 sec
CPU time	12 days 1 hours 44 min 53 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	2.28 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/3ir2ko.pje8c10 is not a valid UM file. Error converting file to netcdf: dataout/3ir2ko.pje8c10 Error: Input file: dataout/3ir2ko.pie8c10 is not a valid UM file. Error converting file to netcdf: dataout/3ir2ko.pie8c10 Error: Input file: dataout/3ir2ko.pfe8c10 is not a valid UM file. Error converting file to netcdf: dataout/3ir2ko.pfe8c10 Error: Input file: dataout/3ir2ka.phe8c10 is not a valid UM file. Error converting file to netcdf: dataout/3ir2ka.phe8c10 Error: Input file: dataout/3ir2ka.pge8c10 is not a valid UM file. Error converting file to netcdf: dataout/3ir2ka.pge8c10 Error: Input file: dataout/3ir2ka.pee8c10 is not a valid UM file. Error converting file to netcdf: dataout/3ir2ka.pee8c10 Error: Input file: dataout/3ir2ka.pde8c10 is not a valid UM file. Error converting file to netcdf: dataout/3ir2ka.pde8c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:51:44 (5219): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:55:28 (6368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:57:50 (7368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:00:14 (7469): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:03:20 (8441): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:09:36 (8511): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:19:20 (10259): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:29:30 (12301): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:36:07 (14332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:23:41 (15628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:44:54 (24843): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:01:08 (29842): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:11 (25179): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:36:05 (31366): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:38:46 (32382): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:16:57 (12447): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:29:08 (20553): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:35:58 (22624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 14:34:27 (30521): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /data/boinc-client/projects/climateprediction.net/hadcm3n_3ir2_1940_40_008259326/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450E2C Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0822D68F Unknown Unknown Unknown hadcm3n_um_6.07_i 0818A767 Unknown Unknown Unknown hadcm3n_um_6.07_i 0818D749 Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F7600BD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4131, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /data/boinc-client/projects/climateprediction.net/hadcm3n_3ir2_1940_40_008259326/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450E2C Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0822D68F Unknown Unknown Unknown hadcm3n_um_6.07_i 0818A767 Unknown Unknown Unknown hadcm3n_um_6.07_i 0818D749 Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F755BBD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, sCalled boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Jan 2013 02:16:08	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	440,640	1,226,118	2.7826
15 Jan 2013 00:31:43	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	414,720	1,150,310	2.7737
14 Jan 2013 00:11:38	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	388,800	1,073,466	2.7610
04 Jan 2013 22:10:39	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	362,880	1,002,361	2.7622
04 Jan 2013 00:08:10	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	336,960	930,589	2.7617
03 Jan 2013 00:39:08	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	311,040	860,863	2.7677
01 Jan 2013 19:38:24	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	285,120	788,425	2.7652
31 Dec 2012 13:20:21	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	259,200	715,044	2.7587
30 Dec 2012 07:54:15	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	233,280	644,217	2.7616
29 Dec 2012 03:49:17	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	207,360	572,163	2.7593
27 Dec 2012 21:41:31	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	181,440	498,564	2.7478
26 Dec 2012 14:40:14	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	155,520	425,348	2.7350
25 Dec 2012 14:00:34	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	129,600	351,309	2.7107
24 Dec 2012 13:25:25	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	103,680	278,060	2.6819
23 Dec 2012 07:42:47	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	77,760	206,167	2.6513
22 Dec 2012 09:29:31	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	51,840	136,855	2.6399
21 Dec 2012 13:36:43	1148818	15487005	hadcm3n_3ir2_1940_40_008259326_0	25,920	68,607	2.6469