Task 13404132

Name	hadcm3n_o6mv_1940_40_007444081_4
Workunit	7641584
Created	21 Sep 2011, 14:10:37 UTC
Sent	21 Sep 2011, 14:11:28 UTC
Report deadline	21 Dec 2011, 21:38:39 UTC
Received	18 Dec 2011, 18:19:51 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1040277
Run time	10 days 0 hours 11 min 32 sec
CPU time	8 days 20 hours 44 min 22 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.77 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Con10:49:47 (5036): No heartbeat from core client for 30 sec - exiting 10:49:48 (5036): No heartbeat from core client for 30 sec - exiting 10:49:50 (5036): No heartbeat from core client for 30 sec - exiting 10:49:51 (5036): No heartbeat from core client for 30 sec - exiting 10:49:52 (5036): No heartbeat from core client for 30 sec - exiting 10:49:53 (5036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:52:29 (4856): No heartbeat from core client for 30 sec - exiting 09:52:31 (4856): No heartbeat from core client for 30 sec - exiting 09:52:32 (4856): No heartbeat from core client for 30 sec - exiting 09:52:33 (4856): No heartbeat from core client for 30 sec - exiting 09:52:34 (4856): No heartbeat from core client for 30 sec - exiting 09:52:38 (4856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7460, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:16:48 (4888): No heartbeat from core client for 30 sec - exiting 11:16:49 (4888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:09:54 (5120): No heartbeat from core client for 30 sec - exiting 10:09:57 (5120): No heartbeat from core client for 30 sec - exiting 10:09:58 (5120): No heartbeat from core client for 30 sec - exiting 10:10:00 (5120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9040, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:10:35 (4944): No heartbeat from core client for 30 sec - exiting 18:10:40 (4944): No heartbeat from core client for 30 sec - exiting 18:10:41 (4944): No heartbeat from core client for 30 sec - exiting 18:10:42 (4944): No heartbeat from core client for 30 sec - exiting 18:10:43 (4944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:36:47 (4796): No heartbeat from core client for 30 sec - exiting 20:36:54 (4796): No heartbeat from core client for 30 sec - exiting 20:36:56 (4796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:50:58 (5092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4616, iMonCtr=1 Model crash detected, will try to restart... 15:24:28 (4872): No heartbeat from core client for 30 sec - exiting 15:24:29 (4872): No heartbeat from core client for 30 sec - exiting 15:24:31 (4872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5424, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:53:02 (4880): No heartbeat from core client for 30 sec - exiting 17:53:05 (4880): No heartbeat from core client for 30 sec - exiting 17:53:10 (4880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:40:06 (4548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:40:13 (4548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:50:15 (5044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:50:17 (5044): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4376, iMonCtr=1 Model crash detected, will try to restart... 18:11:58 (4812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:40 (4716): No heartbeat from core client for 30 sec - exiting 18:25:45 (4716): No heartbeat from core client for 30 sec - exiting 18:25:48 (4716): No heartbeat from core client for 30 sec - exiting 18:25:49 (4716): No heartbeat from core client for 30 sec - exiting 18:25:51 (4716): No heartbeat from core client for 30 sec - exiting 18:25:52 (4716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4228, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5568, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12608, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4360, iMonCtr=1 Model crash detected, will try to restart... 13:22:10 (3640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4540, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4540, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5016, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5016, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5016, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5016, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4332, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4368, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4368, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4400, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4400, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4400, iMonCtr=1 Model crash detected, will try to restart... CSignal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Dec 2011 17:26:14	1040277	13404132	hadcm3n_o6mv_1940_40_007444081_4	259,200	765,852	2.9547
11 Dec 2011 21:04:47	1040277	13404132	hadcm3n_o6mv_1940_40_007444081_4	233,280	683,858	2.9315
06 Dec 2011 14:01:28	1040277	13404132	hadcm3n_o6mv_1940_40_007444081_4	207,360	599,497	2.8911
03 Nov 2011 21:50:45	1040277	13404132	hadcm3n_o6mv_1940_40_007444081_4	181,440	516,745	2.8480
31 Oct 2011 19:22:39	1040277	13404132	hadcm3n_o6mv_1940_40_007444081_4	155,520	439,820	2.8281
31 Oct 2011 16:24:07	1040277	13404132	hadcm3n_o6mv_1940_40_007444081_4	129,600	374,775	2.8918
31 Oct 2011 14:54:34	1040277	13404132	hadcm3n_o6mv_1940_40_007444081_4	103,680	307,739	2.9682
17 Oct 2011 20:53:19	1040277	13404132	hadcm3n_o6mv_1940_40_007444081_4	77,760	236,199	3.0375
12 Oct 2011 16:01:55	1040277	13404132	hadcm3n_o6mv_1940_40_007444081_4	51,840	170,665	3.2921
29 Sep 2011 20:55:04	1040277	13404132	hadcm3n_o6mv_1940_40_007444081_4	25,920	88,960	3.4321