Task 15786519

Name	hadcm3n_4gob_1940_40_008302995_3
Workunit	8454130
Created	16 May 2013, 11:28:06 UTC
Sent	16 May 2013, 11:28:16 UTC
Report deadline	15 Aug 2013, 18:55:27 UTC
Received	5 Sep 2013, 13:29:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1254204
Run time	17 days 16 hours 26 min 7 sec
CPU time	16 days 4 hours 24 min 32 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.44 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2004, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3172, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3400, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3528, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5796, iMonCtr=1 Model crash detected, will try to restart... 11:45:49 (3492): No heartbeat from core client for 30 sec - exiting 11:45:51 (3492): No heartbeat from core client for 30 sec - exiting 11:45:52 (3492): No heartbeat from core client for 30 sec - exiting 11:45:53 (3492): No heartbeat from core client for 30 sec - exiting 11:45:54 (3492): No heartbeat from core client for 30 sec - exiting 11:45:55 (3492): No heartbeat from core client for 30 sec - exiting 11:45:56 (3492): No heartbeat from core client for 30 sec - exiting 11:45:57 (3492): No heartbeat from core client for 30 sec - exiting 11:45:58 (3492): No heartbeat from core client for 30 sec - exiting 11:45:59 (3492): No heartbeat from core client for 30 sec - exiting 11:46:00 (3492): No heartbeat from core client for 30 sec - exiting 11:46:01 (3492): No heartbeat from core client for 30 sec - exiting 11:46:03 (3492): No heartbeat from core client for 30 sec - exiting 11:46:04 (3492): No heartbeat from core client for 30 sec - exiting 11:46:05 (3492): No heartbeat from core client for 30 sec - exiting 11:46:06 (3492): No heartbeat from core client for 30 sec - exiting 11:46:07 (3492): No heartbeat from core client for 30 sec - exiting 11:46:08 (3492): No heartbeat from core client for 30 sec - exiting 11:46:09 (3492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:46:10 (3492): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5292, iMonCtr=1 Model crash detected, will try to restart... 12:50:13 (5796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5580, iMonCtr=1 Model crash detected, will try to restart... Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5400, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=476, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1088, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=372, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4176, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4188, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2860, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5692, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2864, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on 4gobko.daf9140 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5248, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5928, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2800, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3120, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5404, iMonCtr=1 Model crash detected, will try to restart... 12:19:33 (4624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1192, iMonCtr=1 Model crash detected, will try to restart... Ocean Restart file copy failed on 4gobko.dag34i0 Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5836, iMonCtr=1 Model crash detected, will try to restart... Ocean Restart file copy failed on 4gobko.dag3bh0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2448, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5628, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5768, iMonCtr=1 Model crash detected, will try to restart... Ocean Restart file copy failed on 4gobko.dag59a0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1888, iMonCtr=1 Model crash detected, will try to restart... Ocean Restart file copy failed on 4gobko.dag71p0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1096, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5668, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5964, iMonCtr=1 Model crash detected, will try to restart... 18:55:12 (5152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4840, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4488, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5924, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1504, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3872, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77967225 read attempt to address 0x40F718F1 Engaging BOINC Windows Runtime Debugger... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Sep 2013 12:08:01	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	777,600	1,388,832	1.7860
27 Aug 2013 09:25:50	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	751,680	1,340,633	1.7835
18 Aug 2013 14:25:55	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	725,760	1,289,700	1.7770
16 Aug 2013 08:34:35	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	699,840	1,241,354	1.7738
16 Aug 2013 08:34:35	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	673,920	1,191,810	1.7685
16 Aug 2013 08:34:35	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	648,000	1,144,364	1.7660
16 Aug 2013 08:34:35	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	622,080	1,097,237	1.7638
16 Aug 2013 08:34:35	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	596,160	1,049,304	1.7601
23 Jul 2013 22:15:49	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	570,240	1,001,192	1.7557
23 Jul 2013 16:13:49	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	544,320	953,159	1.7511
10 Jul 2013 12:46:47	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	518,400	905,100	1.7459
08 Jul 2013 09:32:29	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	492,480	857,287	1.7408
03 Jul 2013 12:27:05	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	466,560	809,303	1.7346
28 Jun 2013 12:10:02	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	440,640	762,746	1.7310
20 Jun 2013 13:35:02	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	414,720	717,634	1.7304
19 Jun 2013 23:58:42	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	388,800	670,183	1.7237
19 Jun 2013 11:02:26	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	362,880	624,927	1.7221
16 Jun 2013 23:21:30	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	336,960	580,030	1.7214
13 Jun 2013 22:36:46	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	311,040	535,724	1.7224
12 Jun 2013 20:37:17	1254204	15786519	hadcm3n_4gob_1940_40_008302995_3	285,120	491,437	1.7236