Task 16203052

Name	hadcm3n_o8rl_1900_40_008466484_2
Workunit	8617323
Created	7 Jan 2014, 14:55:39 UTC
Sent	7 Jan 2014, 14:55:42 UTC
Report deadline	8 Apr 2014, 22:22:53 UTC
Received	11 Mar 2014, 3:53:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1213699
Run time	32 days 23 hours 36 min 46 sec
CPU time	32 days 11 hours 20 min 54 sec
Validate state	Invalid
Credit	7,153.92
Device peak FLOPS	1.10 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/o8rlko.pjc0c10 is not a valid UM file. Error converting file to netcdf: dataout/o8rlko.pjc0c10 Error: Input file: dataout/o8rlko.pic0c10 is not a valid UM file. Error converting file to netcdf: dataout/o8rlko.pic0c10 Error: Input file: dataout/o8rlko.pfc0c10 is not a valid UM file. Error converting file to netcdf: dataout/o8rlko.pfc0c10 Error: Input file: dataout/o8rlko.pcc0c10 is not a valid UM file. Error converting file to netcdf: dataout/o8rlko.pcc0c10 Error: Input file: dataout/o8rlko.pbc0c10 is not a valid UM file. Error converting file to netcdf: dataout/o8rlko.pbc0c10 Error: Input file: dataout/o8rlko.pac0c10 is not a valid UM file. Error converting file to netcdf: dataout/o8rlko.pac0c10 Error: Input file: dataout/o8rlka.phc0c10 is not a valid UM file. Error converting file to netcdf: dataout/o8rlka.phc0c10 Error: Input file: dataout/o8rlka.pgc0c10 is not a valid UM file. Error converting file to netcdf: dataout/o8rlka.pgc0c10 Error: Input file: dataout/o8rlka.pec0c10 is not a valid UM file. Error converting file to netcdf: dataout/o8rlka.pec0c10 Error: Input file: dataout/o8rlka.pdc0c10 is not a valid UM file. Error converting file to netcdf: dataout/o8rlka.pdc0c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7736400] [0xb7736424] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75571df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb755a825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75424d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5437, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7703400] [0xb7703424] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75241df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb7527825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb750f4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5437, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb773e400] [0xb773e424] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb755f1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb7562825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb754a4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5437, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7760400] [0xb7760424] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75811df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb7584825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb756c4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5437, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb772c400] [0xb772c424] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb754d1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb7550825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75384d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5437, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb774c400] [0xb774c424] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb756d1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb7570825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75584d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5437, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Mar 2014 20:31:55	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	596,160	2,762,857	4.6344
07 Mar 2014 06:22:31	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	570,240	2,642,332	4.6337
05 Mar 2014 09:28:25	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	544,320	2,521,121	4.6317
25 Feb 2014 02:33:34	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	518,400	2,401,214	4.6320
23 Feb 2014 00:04:50	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	492,480	2,281,318	4.6323
20 Feb 2014 19:20:32	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	466,560	2,161,127	4.6320
18 Feb 2014 15:07:04	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	440,640	2,040,718	4.6313
15 Feb 2014 10:42:03	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	414,720	1,919,656	4.6288
11 Feb 2014 19:49:11	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	388,800	1,798,626	4.6261
01 Feb 2014 23:11:45	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	362,880	1,679,356	4.6279
31 Jan 2014 13:45:46	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	336,960	1,560,748	4.6318
30 Jan 2014 18:13:22	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	311,040	1,441,478	4.6344
30 Jan 2014 18:13:22	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	285,120	1,322,814	4.6395
25 Jan 2014 01:20:49	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	259,200	1,197,712	4.6208
24 Jan 2014 13:43:24	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	233,280	1,070,335	4.5882
22 Jan 2014 01:37:02	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	207,360	943,305	4.5491
17 Jan 2014 07:41:56	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	181,440	823,958	4.5412
15 Jan 2014 22:56:06	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	155,520	708,033	4.5527
14 Jan 2014 14:26:42	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	129,600	592,232	4.5697
13 Jan 2014 05:32:10	1213699	16203052	hadcm3n_o8rl_1900_40_008466484_2	103,680	476,242	4.5934