Task 13663698

Name	hadcm3n_t239_1980_40_007426767_2
Workunit	7624270
Created	26 Nov 2011, 14:14:01 UTC
Sent	26 Nov 2011, 14:14:15 UTC
Report deadline	25 Feb 2012, 21:41:26 UTC
Received	8 Jan 2012, 11:54:14 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	784541
Run time	16 days 18 hours 17 min 13 sec
CPU time	16 days 1 hours 21 min 17 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.33 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 16:28:31 (4064): No heartbeat from core client for 30 sec - exiting 16:28:32 (4064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:28:48 (2732): No heartbeat from core client for 30 sec - exiting 17:28:49 (2732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:08:25 (2864): No heartbeat from core client for 30 sec - exiting 18:08:26 (2864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:31:45 (3204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:01:22 (4020): No heartbeat from core client for 30 sec - exiting 01:01:26 (4020): No heartbeat from core client for 30 sec - exiting 01:01:27 (4020): No heartbeat from core client for 30 sec - exiting 01:01:28 (4020): No heartbeat from core client for 30 sec - exiting 01:01:29 (4020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:30:16 (3920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:01:30 (2860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:28:31 (2172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:08:29 (3184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:08:37 (3184): No heartbeat from core client for 30 sec - exiting 16:23:03 (3252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:34:59 (3944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3404, iMonCtr=1 Model crash detected, will try to restart... 10:14:14 (3136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:14:17 (3136): No heartbeat from core client for 30 sec - exiting 15:22:41 (3960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:22:43 (3960): No heartbeat from core client for 30 sec - exiting 07:09:54 (2980): No heartbeat from core client for 30 sec - exiting 07:09:55 (2980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2816, iMonCtr=1 Model crash detected, will try to restart... 09:25:55 (4040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:17:47 (2968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:28:25 (3988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:01:16 (3112): No heartbeat from core client for 30 sec - exiting 06:01:25 (3112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:09:55 (3612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:09:59 (3612): No heartbeat from core client for 30 sec - exiting 17:40:43 (4044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:43:49 (3032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2628, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 20:39:29 (1216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:03:36 (2992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1108, iMonCtr=1 Model crash detected, will try to restart... 18:09:11 (4000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2664, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3008, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 05:01:24 (4044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:55:22 (3992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:36:04 (2168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:36:08 (2168): No heartbeat from core client for 30 sec - exiting 19:55:00 (3828): No heartbeat from core client for 30 sec - exiting 19:55:01 (3828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1076, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 19:24:25 (556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:34 (2492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:14:26 (3228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:52:06 (3124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3872, iMonCtr=1 Model crash detected, will try to restart... 06:01:28 (1264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:01:33 (2648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:10:22 (4092): No heartbeat from core client for 30 sec - exiting 06:10:27 (4092): No heartbeat from core client for 30 sec - exiting 06:10:28 (4092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:03:44 (1616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:32:24 (4072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:01:16 (2976): No heartbeat from core client for 30 sec - exiting 13:01:17 (2976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77D078F5 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77F36E0F read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_t239_1980_40_007426767/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Jan 2012 14:58:31	784541	13663698	hadcm3n_t239_1980_40_007426767_2	518,400	1,374,312	2.6511
06 Jan 2012 05:47:24	784541	13663698	hadcm3n_t239_1980_40_007426767_2	492,480	1,305,564	2.6510
04 Jan 2012 15:07:39	784541	13663698	hadcm3n_t239_1980_40_007426767_2	466,560	1,236,807	2.6509
01 Jan 2012 12:28:01	784541	13663698	hadcm3n_t239_1980_40_007426767_2	440,640	1,166,906	2.6482
29 Dec 2011 07:18:27	784541	13663698	hadcm3n_t239_1980_40_007426767_2	414,720	1,097,721	2.6469
27 Dec 2011 16:59:22	784541	13663698	hadcm3n_t239_1980_40_007426767_2	388,800	1,029,831	2.6487
26 Dec 2011 02:03:53	784541	13663698	hadcm3n_t239_1980_40_007426767_2	362,880	963,373	2.6548
24 Dec 2011 20:20:25	784541	13663698	hadcm3n_t239_1980_40_007426767_2	336,960	895,526	2.6577
21 Dec 2011 13:44:15	784541	13663698	hadcm3n_t239_1980_40_007426767_2	311,040	826,445	2.6570
18 Dec 2011 14:28:39	784541	13663698	hadcm3n_t239_1980_40_007426767_2	285,120	756,537	2.6534
17 Dec 2011 04:52:58	784541	13663698	hadcm3n_t239_1980_40_007426767_2	259,200	688,757	2.6572
14 Dec 2011 16:50:23	784541	13663698	hadcm3n_t239_1980_40_007426767_2	233,280	620,420	2.6596
13 Dec 2011 09:46:49	784541	13663698	hadcm3n_t239_1980_40_007426767_2	207,360	552,789	2.6658
12 Dec 2011 13:54:59	784541	13663698	hadcm3n_t239_1980_40_007426767_2	181,440	486,130	2.6793
06 Dec 2011 15:27:02	784541	13663698	hadcm3n_t239_1980_40_007426767_2	155,520	417,415	2.6840
05 Dec 2011 21:00:29	784541	13663698	hadcm3n_t239_1980_40_007426767_2	129,600	350,613	2.7053
04 Dec 2011 15:11:32	784541	13663698	hadcm3n_t239_1980_40_007426767_2	103,680	281,807	2.7180
03 Dec 2011 20:06:17	784541	13663698	hadcm3n_t239_1980_40_007426767_2	77,760	214,681	2.7608
01 Dec 2011 14:02:46	784541	13663698	hadcm3n_t239_1980_40_007426767_2	51,840	146,103	2.8183
29 Nov 2011 13:01:43	784541	13663698	hadcm3n_t239_1980_40_007426767_2	25,920	75,782	2.9237