Task 13318675

Name	hadcm3n_t210_1940_40_007433084_0
Workunit	7630587
Created	31 Aug 2011, 21:11:00 UTC
Sent	31 Aug 2011, 21:15:21 UTC
Report deadline	1 Dec 2011, 4:42:32 UTC
Received	13 Oct 2011, 20:58:31 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	876324
Run time	22 days 2 hours 29 min 50 sec
CPU time	16 days 16 hours 55 min 7 sec
Validate state	Invalid
Credit	11,197.44
Device peak FLOPS	3.46 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.21</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/t210ko.pjf5c10 is not a valid UM file. Error converting file to netcdf: dataout/t210ko.pjf5c10 Error: Input file: dataout/t210ko.pif5c10 is not a valid UM file. Error converting file to netcdf: dataout/t210ko.pif5c10 Error: Input file: dataout/t210ko.pff5c10 is not a valid UM file. Error converting file to netcdf: dataout/t210ko.pff5c10 Error: Input file: dataout/t210ka.phf5c10 is not a valid UM file. Error converting file to netcdf: dataout/t210ka.phf5c10 Error: Input file: dataout/t210ka.pgf5c10 is not a valid UM file. Error converting file to netcdf: dataout/t210ka.pgf5c10 Error: Input file: dataout/t210ka.pef5c10 is not a valid UM file. Error converting file to netcdf: dataout/t210ka.pef5c10 Error: Input file: dataout/t210ka.pdf5c10 is not a valid UM file. Error converting file to netcdf: dataout/t210ka.pdf5c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:15:48 (44995): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SETPOS: Seek Failed: No space left on device SETPOS: Unit 42 to Word Address 14221312 Failed with Error Code -1 Model crashed: SETPOS: Unit 42 to Word Address 14221312 Failed with Error Code -1 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t210_1940_40_007433084/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0042E622 Unknown Unknown Unknown hadcm3n_um_6.07_i 0042D0EB Unknown Unknown Unknown hadcm3n_um_6.07_i 003EF4DD Unknown Unknown Unknown hadcm3n_um_6.07_i 003A66F5 Unknown Unknown Unknown hadcm3n_um_6.07_i 003A5E57 Unknown Unknown Unknown hadcm3n_um_6.07_i 003E1E32 Unknown Unknown Unknown hadcm3n_um_6.07_i 0020C599 Unknown Unknown Unknown hadcm3n_um_6.07_i 00160E7A Unknown Unknown Unknown hadcm3n_um_6.07_i 00164453 Unknown Unknown Unknown hadcm3n_um_6.07_i 0038CE7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0038A1ED Unknown Unknown Unknown hadcm3n_um_6.07_i 000048E1 Unknown Unknown Unknown hadcm3n_um_6.07_i 000027FB Unknown Unknown Unknown hadcm3n_um_6.07_i 00002729 Unknown Unknown Unknown Unknown 00000003 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=72409, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t210_1940_40_007433084/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0042E622 Unknown Unknown Unknown hadcm3n_um_6.07_i 0042D0EB Unknown Unknown Unknown hadcm3n_um_6.07_i 003EF4DD Unknown Unknown Unknown hadcm3n_um_6.07_i 003A66F5 Unknown Unknown Unknown hadcm3n_um_6.07_i 003A5E57 Unknown Unknown Unknown hadcm3n_um_6.07_i 003E1E32 Unknown Unknown Unknown hadcm3n_um_6.07_i 0020C599 Unknown Unknown Unknown hadcm3n_um_6.07_i 00160E7A Unknown Unknown Unknown hadcm3n_um_6.07_i 00164453 Unknown Unknown Unknown hadcm3n_um_6.07_i 0038CE7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0038A1ED Unknown Unknown Unknown hadcm3n_um_6.07_i 000048E1 Unknown Unknown Unknown hadcm3n_um_6.07_i 000027FB Unknown Unknown Unknown hadcm3n_um_6.07_i 00002729 Unknown Unknown Unknown Unknown 00000003 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=72409, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_t210_1940_40_007433084/dataout/stdout_um.txt ImCalled boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Oct 2011 05:51:12	876324	13318675	hadcm3n_t210_1940_40_007433084_0	933,120	1,425,225	1.5274
11 Oct 2011 10:08:44	876324	13318675	hadcm3n_t210_1940_40_007433084_0	907,200	1,385,592	1.5273
10 Oct 2011 14:31:26	876324	13318675	hadcm3n_t210_1940_40_007433084_0	881,280	1,347,732	1.5293
09 Oct 2011 23:25:24	876324	13318675	hadcm3n_t210_1940_40_007433084_0	855,360	1,308,649	1.5299
09 Oct 2011 04:02:38	876324	13318675	hadcm3n_t210_1940_40_007433084_0	829,440	1,267,779	1.5285
08 Oct 2011 05:23:57	876324	13318675	hadcm3n_t210_1940_40_007433084_0	803,520	1,228,321	1.5287
07 Oct 2011 00:11:01	876324	13318675	hadcm3n_t210_1940_40_007433084_0	777,600	1,188,187	1.5280
05 Oct 2011 06:53:04	876324	13318675	hadcm3n_t210_1940_40_007433084_0	751,680	1,148,141	1.5274
04 Oct 2011 03:57:13	876324	13318675	hadcm3n_t210_1940_40_007433084_0	725,760	1,108,673	1.5276
30 Sep 2011 09:11:37	876324	13318675	hadcm3n_t210_1940_40_007433084_0	699,840	1,069,712	1.5285
29 Sep 2011 05:50:42	876324	13318675	hadcm3n_t210_1940_40_007433084_0	673,920	1,030,260	1.5288
28 Sep 2011 05:26:54	876324	13318675	hadcm3n_t210_1940_40_007433084_0	648,000	990,126	1.5280
26 Sep 2011 23:16:14	876324	13318675	hadcm3n_t210_1940_40_007433084_0	622,080	951,130	1.5290
25 Sep 2011 21:52:39	876324	13318675	hadcm3n_t210_1940_40_007433084_0	596,160	911,524	1.5290
25 Sep 2011 05:28:30	876324	13318675	hadcm3n_t210_1940_40_007433084_0	570,240	872,445	1.5300
24 Sep 2011 09:47:09	876324	13318675	hadcm3n_t210_1940_40_007433084_0	544,320	832,931	1.5302
23 Sep 2011 02:31:06	876324	13318675	hadcm3n_t210_1940_40_007433084_0	518,400	793,646	1.5310
21 Sep 2011 20:39:32	876324	13318675	hadcm3n_t210_1940_40_007433084_0	492,480	753,060	1.5291
21 Sep 2011 01:37:10	876324	13318675	hadcm3n_t210_1940_40_007433084_0	466,560	712,921	1.5280
20 Sep 2011 07:58:05	876324	13318675	hadcm3n_t210_1940_40_007433084_0	440,640	673,426	1.5283