Task 15450389

Name	hadcm3n_z979_1880_40_008249888_0
Workunit	8405012
Created	22 Nov 2012, 1:58:58 UTC
Sent	22 Nov 2012, 1:59:41 UTC
Report deadline	21 Feb 2013, 9:26:52 UTC
Received	15 Dec 2012, 17:58:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1140527
Run time	19 days 20 hours 34 min 3 sec
CPU time	15 days 14 hours 2 min 39 sec
Validate state	Invalid
Credit	8,398.08
Device peak FLOPS	2.89 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:02:22 (24288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:22:56 (30523): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:37:54 (5917): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:09 (7001): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:46:00 (7493): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:46:01 (7493): No heartbeat from core client for 30 sec - exiting 03:46:02 (7493): No heartbeat from core client for 30 sec - exiting 03:46:03 (7493): No heartbeat from core client for 30 sec - exiting 03:46:04 (7493): No heartbeat from core client for 30 sec - exiting 03:46:05 (7493): No heartbeat from core client for 30 sec - exiting 03:46:06 (7493): No heartbeat from core client for 30 sec - exiting 03:46:07 (7493): No heartbeat from core client for 30 sec - exiting 04:11:58 (7614): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:32:54 (20365): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 07:54:39 (22855): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:08:18 (24296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:37:43 (25375): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:38:26 (27262): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:52:02 (27443): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:52:03 (27443): No heartbeat from core client for 30 sec - exiting 08:52:04 (27443): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 09:17:12 (28366): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:37:58 (30006): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:03:33 (31407): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:06:33 (778): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:06:34 (778): No heartbeat from core client for 30 sec - exiting 10:13:52 (1114): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:13:53 (1114): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... 15:59:13 (9114): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Stack trace (10 frames): /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7751400] [0xf7751430] /lib32/libc.so.6(gsignal+0x51)[0xf75b9951] /lib32/libc.so.6(abort+0x182)[0xf75bcd82] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75a5bd6] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22262, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77c6400] [0xf77c6430] /lib32/libc.so.6(gsignal+0x51)[0xf762e951] /lib32/libc.so.6(abort+0x182)[0xf7631d82] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf761abd6] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22262, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7766400] [0xf7766430] /lib32/libc.so.6(gsignal+0x51)[0xf75ce951] /lib32/libc.so.6(abort+0x182)[0xf75d1d82] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75babd6] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22262, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf779f400] [0xf779f430] /lib32/libc.so.6(gsignal+0x51)[0xf7607951] /lib32/libc.so.6(abort+0x182)[0xf760ad82] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75f3bd6] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22262, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77d9400] [0xf77d9430] /lib32/libc.so.6(gsignal+0x51)[0xf7641951] /lib32/libc.so.6(abort+0x182)[0xf7644d82] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf762dbd6] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22262, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7729400] [0xf7729430] /lib32/libc.so.6(gsignal+0x51)[0xf7591951] /lib32/libc.so.6(abort+0x182)[0xf7594d82] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf757dbd6] /igel2/Vol1/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22262, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Dec 2012 23:16:31	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	699,840	1,304,947	1.8646
14 Dec 2012 14:47:39	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	673,920	1,256,266	1.8641
14 Dec 2012 14:47:39	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	648,000	1,207,180	1.8629
14 Dec 2012 14:47:39	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	622,080	1,160,778	1.8660
14 Dec 2012 14:47:39	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	596,160	1,115,109	1.8705
14 Dec 2012 14:47:39	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	570,240	1,066,805	1.8708
14 Dec 2012 14:47:39	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	544,320	1,020,198	1.8743
14 Dec 2012 14:47:39	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	518,400	973,190	1.8773
14 Dec 2012 14:47:39	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	492,480	923,874	1.8760
14 Dec 2012 14:47:39	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	466,560	875,729	1.8770
14 Dec 2012 14:47:39	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	440,640	826,955	1.8767
07 Dec 2012 20:53:03	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	414,720	778,268	1.8766
07 Dec 2012 01:25:45	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	388,800	730,466	1.8788
06 Dec 2012 10:03:35	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	362,880	684,383	1.8860
05 Dec 2012 12:30:13	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	336,960	635,904	1.8872
04 Dec 2012 11:37:50	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	311,040	586,684	1.8862
03 Dec 2012 13:10:24	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	285,120	537,001	1.8834
02 Dec 2012 12:54:23	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	259,200	487,186	1.8796
01 Dec 2012 09:56:30	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	233,280	437,232	1.8743
30 Nov 2012 10:52:22	1140527	15450389	hadcm3n_z979_1880_40_008249888_0	207,360	387,433	1.8684