Task 13400166

Name	hadcm3n_o2d1_1980_40_007457050_3
Workunit	7654553
Created	19 Sep 2011, 7:07:34 UTC
Sent	19 Sep 2011, 7:13:45 UTC
Report deadline	19 Dec 2011, 14:40:56 UTC
Received	6 Dec 2011, 0:59:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	940669
Run time	14 days 1 hours 25 min 52 sec
CPU time	13 days 3 hours 53 min 38 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.32 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:10:37 (77932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 21:40:07 (79660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:16:20 (1592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:28:45 (4520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:56:10 (2840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5104, iMonCtr=1 Model crash detected, will try to restart... 02:08:12 (1632): No heartbeat from core client for 30 sec - exiting 02:08:13 (1632): No heartbeat from core client for 30 sec - exiting 02:08:14 (1632): No heartbeat from core client for 30 sec - exiting 02:08:15 (1632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2252, selfPID=2252, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5884, selfPID=5884, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 07:38:16 (52748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. 07:40:55 (49968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 CPDN Monitor -BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Dec 2011 22:56:27	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	518,400	1,127,696	2.1753
05 Dec 2011 08:35:52	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	492,480	1,077,228	2.1874
04 Dec 2011 03:48:46	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	466,560	1,316,547	2.8218
21 Nov 2011 00:31:27	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	440,640	903,447	2.0503
20 Nov 2011 08:42:48	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	414,720	852,289	2.0551
19 Nov 2011 17:05:54	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	388,800	801,040	2.0603
19 Nov 2011 01:26:47	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	362,880	749,616	2.0657
18 Nov 2011 08:24:05	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	336,960	698,179	2.0720
17 Nov 2011 11:29:05	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	311,040	646,284	2.0778
15 Nov 2011 17:30:12	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	285,120	595,015	2.0869
31 Oct 2011 17:38:46	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	259,200	543,706	2.0976
31 Oct 2011 15:48:24	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	233,280	492,619	2.1117
31 Oct 2011 13:29:47	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	207,360	439,863	2.1213
09 Oct 2011 04:07:41	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	181,440	385,835	2.1265
06 Oct 2011 06:30:51	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	155,520	331,474	2.1314
03 Oct 2011 00:59:54	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	129,600	275,398	2.1250
02 Oct 2011 07:34:47	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	103,680	219,292	2.1151
01 Oct 2011 14:43:52	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	77,760	163,273	2.0997
30 Sep 2011 22:43:46	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	51,840	107,307	2.0700
21 Sep 2011 07:40:43	940669	13400166	hadcm3n_o2d1_1980_40_007457050_3	25,920	53,090	2.0482