Task 14840547

Name	hadcm3n_o5pb_2100_40_008025860_1
Workunit	8180974
Created	25 Jun 2012, 1:43:48 UTC
Sent	25 Jun 2012, 1:43:56 UTC
Report deadline	24 Sep 2012, 9:11:07 UTC
Received	15 Dec 2012, 22:38:36 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1092111
Run time	25 days 16 hours 22 min 6 sec
CPU time	20 days 19 hours 54 min 32 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.59 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4692, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5304, iMonCtr=1 Model crash detected, will try to resCPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2180, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4228, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4228, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4228, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1456, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1264, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1932, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7944, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CSuspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3884, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5856, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3808, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5764, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4684, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3520, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4008, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5964, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4024, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5460, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3900, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5268, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=332, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3376, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4084, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=904, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2600, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=708, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 17:39:41 (2316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:39:42 (2316): No heartbeat from core client for 30 sec - exiting 17:39:43 (2316): No heartbeat from core client for 30 sec - exiting 17:39:44 (2316): No heartbeat from core client for 30 sec - exiting 17:39:45 (2316): No heartbeat from core client for 30 sec - exiting 17:39:46 (2316): No heartbeat from core client for 30 sec - exiting 17:39:47 (2316): No heartbeat from core client for 30 sec - exiting 17:39:48 (2316): No heartbeat from core client for 30 sec - exiting 17:39:49 (2316): No heartbeat from core client for 30 sec - exiting 17:39:50 (2316): No heartbeat from core client for 30 sec - exiting 17:39:51 (2316): No heartbeat from core client for 30 sec - exiting 17:39:52 (2316): No heartbeat from core client for 30 sec - exiting 17:39:53 (2316): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4156, iMonCtr=1 Model crash detected, will try to restart... 15:36:40 (4500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:36:42 (4500): No heartbeat from core client for 30 sec - exiting 15:36:43 (4500): No heartbeat from core client for 30 sec - exiting 15:36:44 (4500): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77556E5F read attempt to address 0x403C9E42 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_o5pb_2100_40_008025860/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Dec 2012 21:36:51	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	518,400	1,799,660	3.4716
04 Dec 2012 21:21:32	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	492,480	1,707,722	3.4676
25 Nov 2012 04:54:36	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	466,560	1,613,321	3.4579
19 Nov 2012 23:49:57	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	440,640	1,519,994	3.4495
15 Nov 2012 02:33:36	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	414,720	1,426,406	3.4394
31 Oct 2012 04:13:18	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	388,800	1,333,319	3.4293
27 Oct 2012 04:08:22	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	362,880	1,242,941	3.4252
15 Oct 2012 03:09:47	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	336,960	1,151,284	3.4167
30 Sep 2012 02:05:17	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	311,040	1,059,238	3.4055
18 Sep 2012 02:14:02	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	285,120	968,071	3.3953
02 Sep 2012 02:49:23	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	259,200	876,726	3.3824
22 Aug 2012 21:53:54	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	233,280	785,429	3.3669
18 Aug 2012 18:43:59	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	207,360	698,918	3.3706
17 Aug 2012 16:01:28	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	181,440	616,622	3.3985
14 Aug 2012 21:22:29	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	155,520	530,023	3.4081
09 Aug 2012 22:59:52	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	129,600	440,602	3.3997
01 Aug 2012 04:44:41	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	103,680	350,335	3.3790
19 Jul 2012 02:14:54	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	77,760	260,683	3.3524
08 Jul 2012 00:18:23	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	51,840	175,228	3.3802
02 Jul 2012 22:01:02	1092111	14840547	hadcm3n_o5pb_2100_40_008025860_1	25,920	89,763	3.4631