Task 18329508

Name	hadcm3s_8zmo_1984_2_009756473_0
Workunit	9827471
Created	20 Apr 2015, 11:09:26 UTC
Sent	27 May 2015, 4:06:39 UTC
Report deadline	26 Aug 2015, 11:33:50 UTC
Received	30 May 2015, 1:11:41 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1328863
Run time	1 days 15 hours 29 min 43 sec
CPU time	1 days 14 hours 10 min 55 sec
Validate state	Invalid
Credit	311.04
Device peak FLOPS	1.25 GFLOPS
Application version	UK Met Office HadCM3 short v7.24 windows_intelx86
Stderr	<core_client_version>7.4.27</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C81EB33 Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 7.2.4 Dump Timestamp : 05/29/15 20:09:29 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 126 Loaded Library : dbghelp.dll LoadLibraryA( C:\Program Files\BOINC\\symsrv.dll ): GetLastError = 126 LoadLibraryA( symsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files\BOINC\\srcsrv.dll ): GetLastError = 126 LoadLibraryA( srcsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3s_8zmo_1984_2_009756473;C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : * Dump of the Process Statistics: * - I/O Operations Counters - Read: 37, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 196, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 32824, QuotaPeakPagedPoolUsage: 32824 QuotaNonPagedPoolUsage: 1840, QuotaPeakNonPagedPoolUsage: 2480 - Virtual Memory Usage - VirtualSize: 871133184, PeakVirtualSize: 871133184 - Pagefile Usage - PagefileUsage: 32813056, PeakPagefileUsage: 106168320 - Working Set Size - WorkingSetSize: 1904640, PeakWorkingSetSize: 1904640, PageFaultCount: 464 * Dump of thread ID 4004 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 937500.000000, User Time: 156250.000000, Wait Time: 90954976.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C81EB33 - Registers - eax=1143fa54 ebx=7ffde000 ecx=00000000 edx=009e8ef8 esi=1143fadc edi=009e849c eip=7c81eb33 esp=1143fa50 ebp=1143faa4 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 1143faa4 007b2d27 e06d7363 00000001 00000003 1143fad0 !RaiseException+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c81eb33' 1143fadc 007afae1 1143faec 0083a2cc 0080e158 00832824 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007b2d27' 1143fafc 0040643c 04600000 01000000 00500000 00407892 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007afae1' 1143ff30 007b1c2a 00400000 00000000 00052391 0000000a !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '0040643c' 1143ffc0 7c816d4f 001540c4 00000000 7ffde000 e06d7363 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007b1c2a' 1143fff0 00000000 007b1c7d 00000000 00000000 00000000 !RegisterWaitForInputIdle+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c816d4f' * Dump of thread ID 1908 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 90954968.000000 - Registers - eax=0023c346 ebx=00000000 ecx=00000000 edx=325cfd9c esi=00000000 edi=325cfd4c eip=7c90eb94 esp=325cfd1c ebp=325cfd74 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202 - Callstack - ChildEBP RetAddr Args to Child 325cfd74 7c802451 00000064 00000000 325cffb4 007352f4 !KiFastSystemCallRet+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c90eb94' 325cfd84 007352f4 00000064 00736380 00346ad8 00000000 !Sleep+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c802451' 325cffb4 7c80b50b 00000000 00736380 00346ad8 00000000 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007352f4' 325cffec 00000000 007352d0 00000000 00000000 32ae0000 !GetModuleFileNameA+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c80b50b' * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=848, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C81EB33 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 7.2.4 Dump Timestamp : 05/29/15 20:10:12 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 126 Loaded Library : dbghelp.dll LoadLibraryA( C:\Program Files\BOINC\\symsrv.dll ): GetLastError = 126 LoadLibraryA( symsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files\BOINC\\srcsrv.dll ): GetLastError = 126 LoadLibraryA( srcsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3s_8zmo_1984_2_009756473;C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : * Dump of the Process Statistics: * - I/O Operations Counters - Read: 6, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 190, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 32824, QuotaPeakPagedPoolUsage: 32824 QuotaNonPagedPoolUsage: 1840, QuotaPeakNonPagedPoolUsage: 2480 - Virtual Memory Usage - VirtualSize: 871133184, PeakVirtualSize: 871133184 - Pagefile Usage - PagefileUsage: 32813056, PeakPagefileUsage: 106168320 - Working Set Size - WorkingSetSize: 1908736, PeakWorkingSetSize: 1908736, PageFaultCount: 465 * Dump of thread ID 3440 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 781250.000000, User Time: 0.000000, Wait Time: 90957712.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C81EB33 - Registers - eax=1143fa54 ebx=7ffd6000 ecx=00000000 edx=009e8ef8 esi=1143fadc edi=009e849c eip=7c81eb33 esp=1143fa50 ebp=1143faa4 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 1143faa4 007b2d27 e06d7363 00000001 00000003 1143fad0 !RaiseException+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c81eb33' 1143fadc 007afae1 1143faec 0083a2cc 0080e158 00832824 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007b2d27' 1143fafc 0040643c 04600000 01000000 00500000 00407892 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007afae1' 1143ff30 007b1c2a 00400000 00000000 00052391 0000000a !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '0040643c' 1143ffc0 7c816d4f 001540c4 00000000 7ffd6000 e06d7363 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007b1c2a' 1143fff0 00000000 007b1c7d 00000000 00000000 00000000 !RegisterWaitForInputIdle+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c816d4f' * Dump of thread ID 992 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 90957712.000000 - Registers - eax=0036d616 ebx=00000000 ecx=00000000 edx=325cfd9c esi=00000000 edi=325cfd4c eip=7c90eb94 esp=325cfd1c ebp=325cfd74 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202 - Callstack - ChildEBP RetAddr Args to Child 325cfd74 7c802451 00000064 00000000 325cffb4 007352f4 !KiFastSystemCallRet+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c90eb94' 325cfd84 007352f4 00000064 00736380 00346ad8 00000000 !Sleep+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c802451' 325cffb4 7c80b50b 00000000 00736380 00346ad8 00000000 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007352f4' 325cffec 00000000 007352d0 00000000 00000000 32ae0000 !GetModuleFileNameA+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c80b50b' * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=848, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C81EB33 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 7.2.4 Dump Timestamp : 05/29/15 20:10:50 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 126 Loaded Library : dbghelp.dll LoadLibraryA( C:\Program Files\BOINC\\symsrv.dll ): GetLastError = 126 LoadLibraryA( symsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files\BOINC\\srcsrv.dll ): GetLastError = 126 LoadLibraryA( srcsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3s_8zmo_1984_2_009756473;C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 190, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 32824, QuotaPeakPagedPoolUsage: 32824 QuotaNonPagedPoolUsage: 1840, QuotaPeakNonPagedPoolUsage: 2480 - Virtual Memory Usage - VirtualSize: 871133184, PeakVirtualSize: 871133184 - Pagefile Usage - PagefileUsage: 32813056, PeakPagefileUsage: 106168320 - Working Set Size - WorkingSetSize: 1908736, PeakWorkingSetSize: 1908736, PageFaultCount: 465 * Dump of thread ID 396 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 625000.000000, User Time: 0.000000, Wait Time: 90960184.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C81EB33 - Registers - eax=1143fa54 ebx=7ffdf000 ecx=00000000 edx=009e8ef8 esi=1143fadc edi=009e849c eip=7c81eb33 esp=1143fa50 ebp=1143faa4 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 1143faa4 007b2d27 e06d7363 00000001 00000003 1143fad0 !RaiseException+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c81eb33' 1143fadc 007afae1 1143faec 0083a2cc 0080e158 00832824 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007b2d27' 1143fafc 0040643c 04600000 01000000 00500000 00407892 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007afae1' 1143ff30 007b1c2a 00400000 00000000 00052391 0000000a !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '0040643c' 1143ffc0 7c816d4f 001540c4 00000000 7ffdf000 e06d7363 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007b1c2a' 1143fff0 00000000 007b1c7d 00000000 00000000 00000000 !RegisterWaitForInputIdle+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c816d4f' * Dump of thread ID 2076 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 90960176.000000 - Registers - eax=0015752a ebx=00000000 ecx=00000000 edx=325cfd9c esi=00000000 edi=325cfd4c eip=7c90eb94 esp=325cfd1c ebp=325cfd74 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202 - Callstack - ChildEBP RetAddr Args to Child 325cfd74 7c802451 00000064 00000000 325cffb4 007352f4 !KiFastSystemCallRet+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c90eb94' 325cfd84 007352f4 00000064 00736380 00346ad8 00000000 !Sleep+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c802451' 325cffb4 7c80b50b 00000000 00736380 00346ad8 00000000 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007352f4' 325cffec 00000000 007352d0 00000000 00000000 32ae0000 !GetModuleFileNameA+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c80b50b' * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=848, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C81EB33 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 7.2.4 Dump Timestamp : 05/29/15 20:11:24 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 126 Loaded Library : dbghelp.dll LoadLibraryA( C:\Program Files\BOINC\\symsrv.dll ): GetLastError = 126 LoadLibraryA( symsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files\BOINC\\srcsrv.dll ): GetLastError = 126 LoadLibraryA( srcsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3s_8zmo_1984_2_009756473;C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 190, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 32824, QuotaPeakPagedPoolUsage: 32824 QuotaNonPagedPoolUsage: 1840, QuotaPeakNonPagedPoolUsage: 2480 - Virtual Memory Usage - VirtualSize: 871133184, PeakVirtualSize: 871133184 - Pagefile Usage - PagefileUsage: 32813056, PeakPagefileUsage: 106168320 - Working Set Size - WorkingSetSize: 1904640, PeakWorkingSetSize: 1904640, PageFaultCount: 464 * Dump of thread ID 416 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 468750.000000, User Time: 0.000000, Wait Time: 90962328.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C81EB33 - Registers - eax=1143fa54 ebx=7ffd7000 ecx=00000000 edx=009e8ef8 esi=1143fadc edi=009e849c eip=7c81eb33 esp=1143fa50 ebp=1143faa4 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 1143faa4 007b2d27 e06d7363 00000001 00000003 1143fad0 !RaiseException+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c81eb33' 1143fadc 007afae1 1143faec 0083a2cc 0080e158 00832824 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007b2d27' 1143fafc 0040643c 04600000 01000000 00500000 00407892 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007afae1' 1143ff30 007b1c2a 00400000 00000000 00052391 0000000a !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '0040643c' 1143ffc0 7c816d4f 001540c4 00000000 7ffd7000 e06d7363 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007b1c2a' 1143fff0 00000000 007b1c7d 00000000 00000000 00000000 !RegisterWaitForInputIdle+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c816d4f' * Dump of thread ID 2356 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 90962328.000000 - Registers - eax=00000000 ebx=00000000 ecx=7c80240f edx=7c90eb94 esi=00000000 edi=325cfd4c eip=7c90eb94 esp=325cfd1c ebp=325cfd74 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202 - Callstack - ChildEBP RetAddr Args to Child 325cfd74 7c802451 00000064 00000000 325cffb4 007352f4 !KiFastSystemCallRet+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c90eb94' 325cfd84 007352f4 00000064 00736380 00346ad8 00000000 !Sleep+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c802451' 325cffb4 7c80b50b 00000000 00736380 00346ad8 00000000 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007352f4' 325cffec 00000000 007352d0 00000000 00000000 32ae0000 !GetModuleFileNameA+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c80b50b' * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=848, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C81EB33 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 7.2.4 Dump Timestamp : 05/29/15 20:11:57 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 126 Loaded Library : dbghelp.dll LoadLibraryA( C:\Program Files\BOINC\\symsrv.dll ): GetLastError = 126 LoadLibraryA( symsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files\BOINC\\srcsrv.dll ): GetLastError = 126 LoadLibraryA( srcsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3s_8zmo_1984_2_009756473;C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 190, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 32824, QuotaPeakPagedPoolUsage: 32824 QuotaNonPagedPoolUsage: 1840, QuotaPeakNonPagedPoolUsage: 2480 - Virtual Memory Usage - VirtualSize: 871133184, PeakVirtualSize: 871133184 - Pagefile Usage - PagefileUsage: 32813056, PeakPagefileUsage: 106168320 - Working Set Size - WorkingSetSize: 1908736, PeakWorkingSetSize: 1908736, PageFaultCount: 465 * Dump of thread ID 3168 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 156250.000000, User Time: 0.000000, Wait Time: 90964440.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C81EB33 - Registers - eax=1143fa54 ebx=7ffdc000 ecx=00000000 edx=009e8ef8 esi=1143fadc edi=009e849c eip=7c81eb33 esp=1143fa50 ebp=1143faa4 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 1143faa4 007b2d27 e06d7363 00000001 00000003 1143fad0 !RaiseException+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c81eb33' 1143fadc 007afae1 1143faec 0083a2cc 0080e158 00832824 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007b2d27' 1143fafc 0040643c 04600000 01000000 00500000 00407892 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007afae1' 1143ff30 007b1c2a 00400000 00000000 00052391 0000000a !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '0040643c' 1143ffc0 7c816d4f 001540c4 00000000 7ffdc000 e06d7363 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007b1c2a' 1143fff0 00000000 007b1c7d 00000000 00000000 00000000 !RegisterWaitForInputIdle+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c816d4f' * Dump of thread ID 1696 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 90964440.000000 - Registers - eax=0010b076 ebx=00000000 ecx=00000000 edx=325cfd9c esi=00000000 edi=325cfd4c eip=7c90eb94 esp=325cfd1c ebp=325cfd74 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202 - Callstack - ChildEBP RetAddr Args to Child 325cfd74 7c802451 00000064 00000000 325cffb4 007352f4 !KiFastSystemCallRet+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c90eb94' 325cfd84 007352f4 00000064 00736380 00346ad8 00000000 !Sleep+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c802451' 325cffb4 7c80b50b 00000000 00736380 00346ad8 00000000 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007352f4' 325cffec 00000000 007352d0 00000000 00000000 32ae0000 !GetModuleFileNameA+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c80b50b' * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=848, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C81EB33 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 7.2.4 Dump Timestamp : 05/29/15 20:12:34 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 126 Loaded Library : dbghelp.dll LoadLibraryA( C:\Program Files\BOINC\\symsrv.dll ): GetLastError = 126 LoadLibraryA( symsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files\BOINC\\srcsrv.dll ): GetLastError = 126 LoadLibraryA( srcsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3s_8zmo_1984_2_009756473;C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : SymGetModuleInfo(): GetLastError = 87 ModLoad: 0000000000000000 0000000000000000 ( Symbols Loaded) Linked PDB Filename : * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 190, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 32824, QuotaPeakPagedPoolUsage: 32824 QuotaNonPagedPoolUsage: 1840, QuotaPeakNonPagedPoolUsage: 2480 - Virtual Memory Usage - VirtualSize: 871133184, PeakVirtualSize: 871133184 - Pagefile Usage - PagefileUsage: 32813056, PeakPagefileUsage: 106168320 - Working Set Size - WorkingSetSize: 1904640, PeakWorkingSetSize: 1904640, PageFaultCount: 464 * Dump of thread ID 3004 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 468750.000000, User Time: 312500.000000, Wait Time: 90966792.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C81EB33 - Registers - eax=1143fa54 ebx=7ffda000 ecx=00000000 edx=009e8ef8 esi=1143fadc edi=009e849c eip=7c81eb33 esp=1143fa50 ebp=1143faa4 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 1143faa4 007b2d27 e06d7363 00000001 00000003 1143fad0 !RaiseException+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c81eb33' 1143fadc 007afae1 1143faec 0083a2cc 0080e158 00832824 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007b2d27' 1143fafc 0040643c 04600000 01000000 00500000 00407892 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007afae1' 1143ff30 007b1c2a 00400000 00000000 00052391 0000000a !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '0040643c' 1143ffc0 7c816d4f 001540c4 00000000 7ffda000 e06d7363 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007b1c2a' 1143fff0 00000000 007b1c7d 00000000 00000000 00000000 !RegisterWaitForInputIdle+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c816d4f' * Dump of thread ID 1152 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 90966792.000000 - Registers - eax=0023c346 ebx=00000000 ecx=00000000 edx=325cfd9c esi=00000000 edi=325cfd4c eip=7c90eb94 esp=325cfd1c ebp=325cfd74 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202 - Callstack - ChildEBP RetAddr Args to Child 325cfd74 7c802451 00000064 00000000 325cffb4 007352f4 !KiFastSystemCallRet+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c90eb94' 325cfd84 007352f4 00000064 00736380 00346ad8 00000000 !Sleep+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c802451' 325cffb4 7c80b50b 00000000 00736380 00346ad8 00000000 !+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '007352f4' 325cffec 00000000 007352d0 00000000 00000000 32ae0000 !GetModuleFileNameA+0x0 SymGetModuleInfo(): GetLastError = '87' Address = '7c80b50b' * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=848, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( 20:12:44 (848): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 May 2015 18:26:32	1328863	18329508	hadcm3s_8zmo_1984_2_009756473_0	25,920	127,765	4.9292