Task 13928331

Name	hadcm3n_u240_1980_40_007683945_0
Workunit	7839032
Created	16 Jan 2012, 8:46:59 UTC
Sent	16 Jan 2012, 8:53:15 UTC
Report deadline	16 Apr 2012, 16:20:26 UTC
Received	6 Feb 2012, 3:37:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1227722
Run time	18 days 9 hours 37 min 13 sec
CPU time	13 days 0 hours 56 min 18 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.50 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 1 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 forrtl: No space left on device forrtl: severe (38): error during write, unit -1, file stdout Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450B99 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 083009BD Unknown Unknown Unknown hadcm3n_um_6.07_i 0838EDB1 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F75633E6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8775, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 forrtl: No space left on device forrtl: severe (38): error during write, unit -1, file stdout Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450B99 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 083009BD Unknown Unknown Unknown hadcm3n_um_6.07_i 0838EDB1 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 UnController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1076, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 forrtl: No space left on device forrtl: severe (38): error during write, unit -1, file stdout Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450B99 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 083009BD Unknown Unknown Unknown hadcm3n_um_6.07_i 0838EDB1 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F75E13E6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1076, iMonCtr=1 Model crash detected, will try to restart... Signal 2 received, exiting... Called boinc_finish Signal 2 received, exiting... Called boinc_finish SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf778b400] [0xf778b425] /lib32/libc.so.6(gsignal+0x49)[0xf74a58a9] /lib32/libc.so.6(abort+0x17c)[0xf74a719c] /lib32/libc.so.6(+0x6a02b)[0xf74e102b] /lib32/libc.so.6(+0x7031f)[0xf74e731f] /lib32/libc.so.6(+0x71c83)[0xf74e8c83] /lib32/libc.so.6(cfree+0x69)[0xf74ebe49] /usr/lib/gcc/x86_64-pc-linux-gnu/4.5.3/32/libstdc++.so.6(_ZdlPv+0x21)[0xf76d0b01] /usr/lib/gcc/x86_64-pc-linux-gnu/4.5.3/32/libstdc++.so.6(_ZdaPv+0x1d)[0xf76d0b7d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf74913e6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Feb 2012 03:39:03	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	777,600	1,126,572	1.4488
05 Feb 2012 13:19:15	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	751,680	1,085,376	1.4439
05 Feb 2012 01:06:54	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	725,760	1,119,491	1.5425
04 Feb 2012 12:40:04	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	699,840	1,078,390	1.5409
03 Feb 2012 17:55:47	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	673,920	1,037,202	1.5391
03 Feb 2012 01:30:11	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	648,000	1,028,243	1.5868
02 Feb 2012 04:07:17	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	622,080	987,372	1.5872
01 Feb 2012 04:30:19	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	596,160	946,866	1.5883
31 Jan 2012 04:37:49	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	570,240	906,298	1.5893
30 Jan 2012 04:49:01	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	544,320	865,826	1.5907
29 Jan 2012 04:54:28	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	518,400	825,213	1.5918
28 Jan 2012 05:14:25	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	492,480	784,624	1.5932
27 Jan 2012 13:50:22	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	466,560	743,429	1.5934
27 Jan 2012 02:11:13	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	440,640	702,286	1.5938
26 Jan 2012 13:12:39	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	414,720	660,997	1.5938
26 Jan 2012 01:34:58	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	388,800	619,762	1.5940
25 Jan 2012 11:49:08	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	362,880	578,479	1.5941
24 Jan 2012 23:20:16	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	336,960	537,013	1.5937
24 Jan 2012 11:31:22	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	311,040	495,791	1.5940
23 Jan 2012 15:31:59	1165864	13928331	hadcm3n_u240_1980_40_007683945_0	285,120	454,247	1.5932