Task 15823966

Name	hadcm3n_o53f_1980_40_008387993_0
Workunit	8538852
Created	3 Jun 2013, 11:50:33 UTC
Sent	3 Jun 2013, 12:06:45 UTC
Report deadline	2 Sep 2013, 19:33:56 UTC
Received	23 Sep 2013, 12:39:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1060699
Run time	5 days 1 hours 44 min 25 sec
CPU time	4 days 7 hours 30 min 29 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.40 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:00:21 (704): No heartbeat from core client for 30 sec - exiting 17:00:22 (704): No heartbeat from core client for 30 sec - exiting 17:00:23 (704): No heartbeat from core client for 30 sec - exiting 17:00:24 (704): No heartbeat from core client for 30 sec - exiting 17:00:25 (704): No heartbeat from core client for 30 sec - exiting 17:00:26 (704): No heartbeat from core client for 30 sec - exiting 17:00:27 (704): No heartbeat from core client for 30 sec - exiting 17:00:28 (704): No heartbeat from core client for 30 sec - exiting 17:00:29 (704): No heartbeat from core client for 30 sec - exiting 17:00:30 (704): No heartbeat from core client for 30 sec - exiting 17:00:31 (704): No heartbeat from core client for 30 sec - exiting 17:00:32 (704): No heartbeat from core client for 30 sec - exiting 17:00:33 (704): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7698B727 Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 06/08/13 17:00:34 Install Directory : C:\Program Files (x86)\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : 17:00:34 (704): No heartbeat from core client for 30 sec - exiting Loaded Library : C:\Program Files (x86)\BOINC\\dbghelp.dll Loaded Library : C:\Program Files (x86)\BOINC\\symsrv.dll Loaded Library : C:\Program Files (x86)\BOINC\\srcsrv.dll LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll CPDN Monitor - No 'heartbeat' from BOINC... 17:01:19 (5792): No heartbeat from core client for 30 sec - exiting 17:01:20 (5792): No heartbeat from core client for 30 sec - exiting 17:01:21 (5792): No heartbeat from core client for 30 sec - exiting 17:01:22 (5792): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7698B727 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 06/08/13 17:01:23 Install Directory : C:\Program Files (x86)\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : Loaded Library : C:\Program Files (x86)\BOINC\\dbghelp.dll Loaded Library : C:\Program Files (x86)\BOINC\\symsrv.dll Loaded Library : C:\Program Files (x86)\BOINC\\srcsrv.dll LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7698B727 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 06/08/13 17:02:03 Install Directory : C:\Program Files (x86)\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : LoadLibraryA( C:\Program Files (x86)\BOINC\\dbghelp.dll ): GetLastError = 1114 LoadLibraryA( dbghelp.dll ): GetLastError = 1114 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 0, Write: 0, Other 0 - I/O Transfers Counters - Read: 0, Write: 0, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0 QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0 - Virtual Memory Usage - VirtualSize: 0, PeakVirtualSize: 0 - Pagefile Usage - PagefileUsage: 0, PeakPagefileUsage: 0 - Working Set Size - WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0 * Dump of thread ID 5332 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7698B727 * Dump of thread ID 4368 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3304, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 21:01:13 (5564): No heartbeat from core client for 30 sec - exiting 21:01:14 (5564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:33:51 (2984): No heartbeat from core client for 30 sec - exiting 18:33:52 (2984): No heartbeat from core client for 30 sec - exiting 18:33:53 (2984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:24:49 (6044): No heartbeat from core client for 30 sec - exiting 21:24:50 (6044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4948, selfPID=4948, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o53fko.pji2c10 Error converting file to netcdf: dataout/o53fko.pii2c10 Error converting file to netcdf: dataout/o53fko.pfi2c10 Error converting file to netcdf: dataout/o53fka.phi2c10 Error converting file to netcdf: dataout/o53fka.pgi2c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:43:01 (4332): No heartbeat from core client for 30 sec - exiting 14:43:02 (4332): No heartbeat from core client for 30 sec - exiting 14:43:03 (4332): No heartbeat from core client for 30 sec - exiting 14:43:04 (4332): No heartbeat from core client for 30 sec - exiting 14:43:05 (4332): No heartbeat from core client for 30 sec - exiting 14:43:06 (4332): No heartbeat from core client for 30 sec - exiting 14:43:07 (4332): No heartbeat from core client for 30 sec - exiting 14:43:08 (4332): No heartbeat from core client for 30 sec - exiting 14:43:09 (4332): No heartbeat from core client for 30 sec - exiting 14:43:10 (4332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:44:13 (9464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77E13072 read attempt to address 0x4030ACC9 Engaging BOINC Windows Runtime Debugger... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7788, selfPID=7788, iMonCtr=1 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77E13072 read attempt to address 0x4030ACC9 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Sep 2013 12:44:10	1060699	15823966	hadcm3n_o53f_1980_40_008387993_0	259,200	367,215	1.4167
12 Sep 2013 16:17:00	1060699	15823966	hadcm3n_o53f_1980_40_008387993_0	233,280	330,001	1.4146
10 Sep 2013 20:52:37	1060699	15823966	hadcm3n_o53f_1980_40_008387993_0	207,360	293,159	1.4138
31 Aug 2013 12:16:24	1060699	15823966	hadcm3n_o53f_1980_40_008387993_0	181,440	256,420	1.4132
25 Aug 2013 15:29:28	1060699	15823966	hadcm3n_o53f_1980_40_008387993_0	155,520	220,044	1.4149
02 Jul 2013 18:02:07	1060699	15823966	hadcm3n_o53f_1980_40_008387993_0	129,600	183,228	1.4138
02 Jul 2013 10:40:13	1060699	15823966	hadcm3n_o53f_1980_40_008387993_0	103,680	146,390	1.4119
27 Jun 2013 17:38:36	1060699	15823966	hadcm3n_o53f_1980_40_008387993_0	77,760	109,049	1.4024
17 Jun 2013 11:01:53	1060699	15823966	hadcm3n_o53f_1980_40_008387993_0	51,840	71,896	1.3869
08 Jun 2013 07:28:05	1060699	15823966	hadcm3n_o53f_1980_40_008387993_0	25,920	35,583	1.3728