Task 16046057

Name	hadcm3n_oflr_1900_40_008475346_0
Workunit	8626185
Created	27 Sep 2013, 10:36:22 UTC
Sent	27 Sep 2013, 15:41:49 UTC
Report deadline	27 Dec 2013, 23:09:00 UTC
Received	9 Nov 2013, 14:08:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1245110
Run time	6 days 1 hours 5 min 40 sec
CPU time	5 days 22 hours 55 min 23 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.67 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.6.41</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc SIGABRT: abort called Stack trace (15 frames): /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7744400] [0xb7744424] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75691df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb756c825] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8401b90] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7af5] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7b32] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7c5a] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f825e] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f829d] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839dfbf] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf3] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75544d3] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21613, iMonCtr=1 Model crash detected, will try to restart... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc SIGABRT: abort called Stack trace (15 frames): /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb77b5400] [0xb77b5424] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75da1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb75dd825] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8401b90] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7af5] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7b32] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7c5a] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f825e] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f829d] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839dfbf] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf3] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75c54d3] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21613, iMonCtr=1 Model crash detected, will try to restart... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc SIGABRT: abort called Stack trace (15 frames): /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7782400] [0xb7782424] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75a71df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb75aa825] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8401b90] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7af5] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7b32] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7c5a] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f825e] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f829d] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839dfbf] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf3] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75924d3] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21613, iMonCtr=1 Model crash detected, will try to restart... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc SIGABRT: abort called Stack trace (15 frames): /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7729400] [0xb7729424] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb754e1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb7551825] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8401b90] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7af5] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7b32] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7c5a] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f825e] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f829d] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839dfbf] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf3] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75394d3] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21613, iMonCtr=1 Model crash detected, will try to restart... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc SIGABRT: abort called Stack trace (15 frames): /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb77bb400] [0xb77bb424] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75e01df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb75e3825] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8401b90] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7af5] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7b32] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7c5a] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f825e] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f829d] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839dfbf] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf3] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75cb4d3] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21613, iMonCtr=1 Model crash detected, will try to restart... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc SIGABRT: abort called Stack trace (15 frames): /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7752400] [0xb7752424] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb75771df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb757a825] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8401b90] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7af5] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7b32] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f7c5a] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f825e] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83f829d] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839dfbf] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf3] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75624d3] /home/huo/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21613, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Nov 2013 13:07:21	1245110	16046057	hadcm3n_oflr_1900_40_008475346_0	259,200	512,988	1.9791
04 Nov 2013 14:56:26	1245110	16046057	hadcm3n_oflr_1900_40_008475346_0	233,280	458,821	1.9668
29 Oct 2013 03:40:37	1245110	16046057	hadcm3n_oflr_1900_40_008475346_0	207,360	404,916	1.9527
26 Oct 2013 13:09:17	1245110	16046057	hadcm3n_oflr_1900_40_008475346_0	181,440	354,168	1.9520
25 Oct 2013 23:02:33	1245110	16046057	hadcm3n_oflr_1900_40_008475346_0	155,520	303,603	1.9522
25 Oct 2013 08:53:24	1245110	16046057	hadcm3n_oflr_1900_40_008475346_0	129,600	253,084	1.9528
24 Oct 2013 18:40:33	1245110	16046057	hadcm3n_oflr_1900_40_008475346_0	103,680	202,454	1.9527
24 Oct 2013 04:26:58	1245110	16046057	hadcm3n_oflr_1900_40_008475346_0	77,760	151,822	1.9524
23 Oct 2013 14:18:54	1245110	16046057	hadcm3n_oflr_1900_40_008475346_0	51,840	101,200	1.9522
23 Oct 2013 00:02:49	1245110	16046057	hadcm3n_oflr_1900_40_008475346_0	25,920	50,610	1.9525