Task 15938469

Name	hadcm3n_700c_1980_40_008411269_1
Workunit	8562125
Created	24 Aug 2013, 13:27:46 UTC
Sent	24 Aug 2013, 13:29:10 UTC
Report deadline	23 Nov 2013, 20:56:21 UTC
Received	17 Nov 2013, 12:00:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1188539
Run time	7 days 8 hours 15 min 6 sec
CPU time	6 days 14 hours 7 min 25 sec
Validate state	Invalid
Credit	4,043.52
Device peak FLOPS	1.05 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.7</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:16:57 (2836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:39:52 (2919): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:41:23 (3228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:42:25 (4230): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_700c_1980_40_008411269/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450B99 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0810D371 Unknown Unknown Unknown hadcm3n_um_6.07_i 081826E6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0818CDAC Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 B754B905 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2651, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_700c_1980_40_008411269/dataout/stdout_um.txt Image PC Routine Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2651, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_700c_1980_40_008411269/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450B99 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0810D371 Unknown Unknown Unknown hadcm3n_um_6.07_i 081826E6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0818CDAC Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 B752F905 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2651, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_700c_1980_40_008411269/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450B99 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0810D371 Unknown Unknown Unknown hadcm3n_um_6.07_i 081826E6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0818CDAC Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 B7551905 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2651, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Nov 2013 18:57:42	1188539	15938469	hadcm3n_700c_1980_40_008411269_1	336,960	546,192	1.6209
28 Oct 2013 21:12:01	1188539	15938469	hadcm3n_700c_1980_40_008411269_1	311,040	507,715	1.6323
27 Oct 2013 19:37:51	1188539	15938469	hadcm3n_700c_1980_40_008411269_1	285,120	468,847	1.6444
21 Oct 2013 03:01:42	1188539	15938469	hadcm3n_700c_1980_40_008411269_1	259,200	430,144	1.6595
15 Oct 2013 06:19:19	1188539	15938469	hadcm3n_700c_1980_40_008411269_1	233,280	389,949	1.6716
13 Oct 2013 10:09:42	1188539	15938469	hadcm3n_700c_1980_40_008411269_1	207,360	341,780	1.6482
06 Oct 2013 09:30:21	1188539	15938469	hadcm3n_700c_1980_40_008411269_1	181,440	297,655	1.6405
29 Sep 2013 10:24:25	1188539	15938469	hadcm3n_700c_1980_40_008411269_1	155,520	255,948	1.6458
10 Sep 2013 16:46:38	1188539	15938469	hadcm3n_700c_1980_40_008411269_1	129,600	214,214	1.6529
27 Aug 2013 06:00:05	1188539	15938469	hadcm3n_700c_1980_40_008411269_1	103,680	169,762	1.6374
26 Aug 2013 16:51:08	1188539	15938469	hadcm3n_700c_1980_40_008411269_1	77,760	123,246	1.5850
26 Aug 2013 03:57:59	1188539	15938469	hadcm3n_700c_1980_40_008411269_1	51,840	78,199	1.5085
25 Aug 2013 15:29:29	1188539	15938469	hadcm3n_700c_1980_40_008411269_1	25,920	39,464	1.5225