Task 16047027

Name	hadcm3n_o2v3_2060_40_008406997_2
Workunit	8557853
Created	27 Sep 2013, 15:30:17 UTC
Sent	27 Sep 2013, 15:35:50 UTC
Report deadline	27 Dec 2013, 23:03:01 UTC
Received	10 Nov 2013, 17:05:27 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1254500
Run time	41 days 18 hours 29 min 35 sec
CPU time	21 days 6 hours 22 min 58 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	1.71 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> 19:36:36 (4152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:38:55 (2380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:41:26 (4492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4132, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4132, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2328, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2328, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2328, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2328, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish 09:12:54 (4900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5388, iMonCtr=1 Model crash detected, will try to restart... 23:53:34 (4684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:53:35 (4684): No heartbeat from core client for 30 sec - exiting 23:53:36 (4684): No heartbeat from core client for 30 sec - exiting 04:48:19 (5560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5808, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 forrtl: fBXNÉ\ªÈó«Ìæª èÜ¹ñB Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2156, iMonCtr=1 Model crash detected, will try to restart... forrtl: fBXNÉ\ªÈó«Ìæª èÜ¹ñB Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2156, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4844, iMonCtr=1 Model crash detected, will try to restart... 21:14:11 (4136): No heartbeat from core client for 30 sec - exiting 21:14:12 (4136): No heartbeat from core client for 30 sec - exiting 21:14:13 (4136): No heartbeat from core client for 30 sec - exiting 21:14:14 (4136): No heartbeat from core client for 30 sec - exiting 21:14:15 (4136): No heartbeat from core client for 30 sec - exiting 21:14:17 (4136): No heartbeat from core client for 30 sec - exiting 21:14:18 (4136): No heartbeat from core client for 30 sec - exiting 21:14:19 (4136): No heartbeat from core client for 30 sec - exiting 21:14:20 (4136): No heartbeat from core client for 30 sec - exiting 21:14:21 (4136): No heartbeat from core client for 30 sec - exiting 21:14:22 (4136): No heartbeat from core client for 30 sec - exiting 21:14:23 (4136): No heartbeat from core client for 30 sec - exiting 21:14:24 (4136): No heartbeat from core client for 30 sec - exiting 21:14:25 (4136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:15:43 (5520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77C00BB0 read attempt to address 0x0B010204 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_o2v3_2060_40_008406997/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Nov 2013 15:14:57	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	777,600	1,837,365	2.3629
07 Nov 2013 18:27:57	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	751,680	1,747,438	2.3247
06 Nov 2013 10:33:19	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	725,760	1,707,116	2.3522
05 Nov 2013 18:32:38	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	699,840	1,667,180	2.3822
04 Nov 2013 12:09:54	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	673,920	1,627,127	2.4144
03 Nov 2013 05:12:09	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	648,000	1,572,942	2.4274
01 Nov 2013 19:38:11	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	622,080	1,495,135	2.4034
31 Oct 2013 15:54:03	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	596,160	1,440,772	2.4168
30 Oct 2013 14:00:45	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	570,240	1,386,898	2.4321
29 Oct 2013 07:07:00	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	544,320	1,332,528	2.4481
28 Oct 2013 07:55:18	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	518,400	1,279,016	2.4672
26 Oct 2013 20:23:24	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	492,480	1,220,480	2.4782
25 Oct 2013 02:04:57	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	466,560	1,128,377	2.4185
22 Oct 2013 22:52:01	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	440,640	980,494	2.2252
21 Oct 2013 08:23:24	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	414,720	878,947	2.1194
20 Oct 2013 09:04:36	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	388,800	828,416	2.1307
18 Oct 2013 13:32:54	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	362,880	710,268	1.9573
16 Oct 2013 15:38:37	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	336,960	572,367	1.6986
15 Oct 2013 15:46:11	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	311,040	515,540	1.6575
14 Oct 2013 13:04:57	1254500	16047027	hadcm3n_o2v3_2060_40_008406997_2	285,120	451,726	1.5843