Task 15901475

Name	hadcm3n_n6bf_1960_40_008402473_1
Workunit	8553329
Created	22 Jul 2013, 18:30:24 UTC
Sent	22 Jul 2013, 18:34:59 UTC
Report deadline	22 Oct 2013, 2:02:10 UTC
Received	29 Jul 2013, 13:39:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	969347
Run time	5 days 23 hours 18 min 44 sec
CPU time	5 days 10 hours 26 min 38 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.31 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:26:59 (10868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:27:00 (10868): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:50:44 (66424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:50:45 (66424): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:00:16 (69044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:27:31 (68864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:27:32 (68864): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7633FC16 Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 07/27/13 07:12:04 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : Loaded Library : C:\Program Files\BOINC\\dbghelp.dll Loaded Library : C:\Program Files\BOINC\\symsrv.dll Loaded Library : C:\Program Files\BOINC\\srcsrv.dll LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_n6bf_1960_40_008402473;C:\ProgramData\BOINC\projects\climateprediction.net ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : ModLoad: 77350000 00128000 C:\Windows\system32\ntdll.dll (6.0.6002.18541) (-exported- Symbols Loaded) Linked PDB Filename : ntdll.pdb File Version : 6.0.6002.18541 (vistasp2_gdr.111116-0305) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18541 ModLoad: 76300000 000dc000 C:\Windows\system32\kernel32.dll (6.0.6002.18704) (-nosymbols- Symbols Loaded) Linked PDB Filename : kernel32.pdb File Version : 6.0.6002.18704 (vistasp2_gdr.120928-0336) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18704 ModLoad: 76690000 0009d000 C:\Windows\system32\USER32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : user32.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 751e0000 0004b000 C:\Windows\system32\GDI32.dll (6.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : gdi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 ModLoad: 753f0000 000c6000 C:\Windows\system32\ADVAPI32.dll (6.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 ModLoad: 76870000 000c3000 C:\Windows\system32\RPCRT4.dll (6.0.6002.18024) (-nosymbols- Symbols Loaded) Linked PDB Filename : rpcrt4.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 5d2c0000 000cb000 C:\Windows\system32\OPENGL32.dll (6.0.6000.16386) (-nosymbols- Symbols Loaded) Linked PDB Filename : opengl32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 767c0000 000aa000 C:\Windows\system32\msvcrt.dll (7.0.6002.18551) (-nosymbols- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.6002.18551 (vistasp2_gdr.111214-0301) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 7.0.6002.18551 ModLoad: 5d260000 00023000 C:\Windows\system32\GLU32.dll (6.0.6000.16386) (-nosymbols- Symbols Loaded) Linked PDB Filename : glu32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 5d170000 000e5000 C:\Windows\system32\DDRAW.dll (6.0.6001.18000) (-nosymbols- Symbols Loaded) Linked PDB Filename : ddraw.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 5d160000 00006000 C:\Windows\system32\DCIMAN32.dll (6.0.6002.18051) (-nosymbols- Symbols Loaded) Linked PDB Filename : dciman32.pdb File Version : 6.0.6002.18051 (vistasp2_gdr.090615-0258) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18051 ModLoad: 763e0000 0018a000 C:\Windows\system32\SETUPAPI.dll (6.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : setupapi.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 4, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 221, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 106312, QuotaPeakPagedPoolUsage: 106368 QuotaNonPagedPoolUsage: 2880, QuotaPeakNonPagedPoolUsage: 2880 - Virtual Memory Usage - VirtualSize: 895770624, PeakVirtualSize: 895770624 - Pagefile Usage - PagefileUsage: 33894400, PeakPagefileUsage: 107106304 - Working Set Size - WorkingSetSize: 4009984, PeakWorkingSetSize: 4009984, PageFaultCount: 1001 * Dump of thread ID 85752 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 156001.000000, User Time: 0.000000, Wait Time: 77552936.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7633FC16 - Registers - eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe eip=7633fc16 esp=1132f9c4 ebp=1132fa14 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000212 - Callstack - ChildEBP RetAddr Args to Child 1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 kernel32!+0x0 1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 1132fad8 00408130 1132faec 00202a41 1132faec 002029c9 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff88 7634d2e9 7ffdf000 1132ffd4 77391603 7ffdf000 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff94 77391603 7ffdf000 2e6f3bd0 00000000 00000000 kernel32!+0x0 1132ffd4 773915d6 007bde7f 7ffdf000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 1132ffec 00000000 007bde7f 7ffdf000 00000000 00000030 ntdll!RtlInitializeExceptionChain+0x0 * Dump of thread ID 84836 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 77552928.000000 - Registers - eax=00749910 ebx=00000000 ecx=00000000 edx=00000000 esi=3376ff48 edi=00000000 eip=773b5cd4 esp=3376ff04 ebp=3376ff6c cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 3376ff6c 76301c6c 00000064 00000000 3376ff88 00749924 ntdll!KiFastSystemCallRet+0x0 3376ff7c 00749924 00000064 3376ff94 7634d2e9 00000000 kernel32!+0x0 3376ff88 7634d2e9 00000000 3376ffd4 77391603 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 3376ff94 77391603 00000000 0c2b3bd0 00000000 00000000 kernel32!+0x0 3376ffd4 773915d6 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 3376ffec 00000000 00749910 00000000 00000000 33c80000 ntdll!RtlInitializeExceptionChain+0x0 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=84728, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7633FC16 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 07/27/13 07:12:36 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : Loaded Library : C:\Program Files\BOINC\\dbghelp.dll Loaded Library : C:\Program Files\BOINC\\symsrv.dll Loaded Library : C:\Program Files\BOINC\\srcsrv.dll LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_n6bf_1960_40_008402473;C:\ProgramData\BOINC\projects\climateprediction.net ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : ModLoad: 77350000 00128000 C:\Windows\system32\ntdll.dll (6.0.6002.18541) (-nosymbols- Symbols Loaded) Linked PDB Filename : ntdll.pdb File Version : 6.0.6002.18541 (vistasp2_gdr.111116-0305) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18541 ModLoad: 76300000 000dc000 C:\Windows\system32\kernel32.dll (6.0.6002.18704) (-nosymbols- Symbols Loaded) Linked PDB Filename : kernel32.pdb File Version : 6.0.6002.18704 (vistasp2_gdr.120928-0336) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18704 ModLoad: 76690000 0009d000 C:\Windows\system32\USER32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : user32.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 751e0000 0004b000 C:\Windows\system32\GDI32.dll (6.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : gdi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 ModLoad: 753f0000 000c6000 C:\Windows\system32\ADVAPI32.dll (6.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 ModLoad: 76870000 000c3000 C:\Windows\system32\RPCRT4.dll (6.0.6002.18024) (-nosymbols- Symbols Loaded) Linked PDB Filename : rpcrt4.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 5d2c0000 000cb000 C:\Windows\system32\OPENGL32.dll (6.0.6000.16386) (-nosymbols- Symbols Loaded) Linked PDB Filename : opengl32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 767c0000 000aa000 C:\Windows\system32\msvcrt.dll (7.0.6002.18551) (-nosymbols- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.6002.18551 (vistasp2_gdr.111214-0301) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 7.0.6002.18551 ModLoad: 5d260000 00023000 C:\Windows\system32\GLU32.dll (6.0.6000.16386) (-nosymbols- Symbols Loaded) Linked PDB Filename : glu32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 5d170000 000e5000 C:\Windows\system32\DDRAW.dll (6.0.6001.18000) (-nosymbols- Symbols Loaded) Linked PDB Filename : ddraw.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 5d160000 00006000 C:\Windows\system32\DCIMAN32.dll (6.0.6002.18051) (-nosymbols- Symbols Loaded) Linked PDB Filename : dciman32.pdb File Version : 6.0.6002.18051 (vistasp2_gdr.090615-0258) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18051 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 218, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 106312, QuotaPeakPagedPoolUsage: 106368 QuotaNonPagedPoolUsage: 2880, QuotaPeakNonPagedPoolUsage: 2880 - Virtual Memory Usage - VirtualSize: 895770624, PeakVirtualSize: 895770624 - Pagefile Usage - PagefileUsage: 33894400, PeakPagefileUsage: 107106304 - Working Set Size - WorkingSetSize: 4018176, PeakWorkingSetSize: 4018176, PageFaultCount: 1002 * Dump of thread ID 85656 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 312002.000000, User Time: 0.000000, Wait Time: 77554968.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7633FC16 - Registers - eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe eip=7633fc16 esp=1132f9c4 ebp=1132fa14 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000212 - Callstack - ChildEBP RetAddr Args to Child 1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 kernel32!+0x0 1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 1132fad8 00408130 1132faec 00112a41 1132faec 001129c9 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff88 7634d2e9 7ffd3000 1132ffd4 77391603 7ffd3000 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff94 77391603 7ffd3000 2e6ebdfc 00000000 00000000 kernel32!+0x0 1132ffd4 773915d6 007bde7f 7ffd3000 00000000 00000000 ntdll!+0x0 1132ffec 00000000 007bde7f 7ffd3000 00000000 00000030 ntdll!+0x0 * Dump of thread ID 82188 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 77554968.000000 - Registers - eax=00749910 ebx=00000000 ecx=00000000 edx=00000000 esi=3375ff48 edi=00000000 eip=773b5cd4 esp=3375ff04 ebp=3375ff6c cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 3375ff6c 76301c6c 00000064 00000000 3375ff88 00749924 ntdll!+0x0 3375ff7c 00749924 00000064 3375ff94 7634d2e9 00000000 kernel32!+0x0 3375ff88 7634d2e9 00000000 3375ffd4 77391603 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 3375ff94 77391603 00000000 0c29bdfc 00000000 00000000 kernel32!+0x0 3375ffd4 773915d6 00749910 00000000 00000000 00000000 ntdll!+0x0 3375ffec 00000000 00749910 00000000 00000000 33c70000 ntdll!+0x0 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=84728, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7633FC16 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 07/27/13 07:13:07 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : Loaded Library : C:\Program Files\BOINC\\dbghelp.dll Loaded Library : C:\Program Files\BOINC\\symsrv.dll Loaded Library : C:\Program Files\BOINC\\srcsrv.dll LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_n6bf_1960_40_008402473;C:\ProgramData\BOINC\projects\climateprediction.net ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : ModLoad: 77350000 00128000 C:\Windows\system32\ntdll.dll (6.0.6002.18541) (-exported- Symbols Loaded) Linked PDB Filename : ntdll.pdb File Version : 6.0.6002.18541 (vistasp2_gdr.111116-0305) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18541 ModLoad: 76300000 000dc000 C:\Windows\system32\kernel32.dll (6.0.6002.18704) (-nosymbols- Symbols Loaded) Linked PDB Filename : kernel32.pdb File Version : 6.0.6002.18704 (vistasp2_gdr.120928-0336) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18704 ModLoad: 76690000 0009d000 C:\Windows\system32\USER32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : user32.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 751e0000 0004b000 C:\Windows\system32\GDI32.dll (6.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : gdi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 ModLoad: 753f0000 000c6000 C:\Windows\system32\ADVAPI32.dll (6.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 ModLoad: 76870000 000c3000 C:\Windows\system32\RPCRT4.dll (6.0.6002.18024) (-nosymbols- Symbols Loaded) Linked PDB Filename : rpcrt4.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 5d2c0000 000cb000 C:\Windows\system32\OPENGL32.dll (6.0.6000.16386) (-nosymbols- Symbols Loaded) Linked PDB Filename : opengl32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 767c0000 000aa000 C:\Windows\system32\msvcrt.dll (7.0.6002.18551) (-nosymbols- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.6002.18551 (vistasp2_gdr.111214-0301) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 7.0.6002.18551 ModLoad: 5d260000 00023000 C:\Windows\system32\GLU32.dll (6.0.6000.16386) (-nosymbols- Symbols Loaded) Linked PDB Filename : glu32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 5d170000 000e5000 C:\Windows\system32\DDRAW.dll (6.0.6001.18000) (-nosymbols- Symbols Loaded) Linked PDB Filename : ddraw.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 5d160000 00006000 C:\Windows\system32\DCIMAN32.dll (6.0.6002.18051) (-nosymbols- Symbols Loaded) Linked PDB Filename : dciman32.pdb File Version : 6.0.6002.18051 (vistasp2_gdr.090615-0258) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18051 ModLoad: 763e0000 0018a000 C:\Windows\system32\SETUPAPI.dll (6.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : setupapi.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 Get Product Name Failed. ModLoad: 76730000 0008d000 C:\Windows\system32\OLEAUT32.dll (6.0.6002.18508) (-nosymbols- Symbols Loaded) Linked PDB Filename : oleaut32.pdb File Version : 6.0.6002.18508 Company Name : Microsoft Corporation Product Name : Product Version : 6.0.6002.18508 ModLoad: 76060000 00145000 C:\Windows\system32\ole32.dll (6.0.6002.18277) (-nosymbols- Symbols Loaded) Linked PDB Filename : ole32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 198, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 106312, QuotaPeakPagedPoolUsage: 106368 QuotaNonPagedPoolUsage: 2880, QuotaPeakNonPagedPoolUsage: 2880 - Virtual Memory Usage - VirtualSize: 895770624, PeakVirtualSize: 895770624 - Pagefile Usage - PagefileUsage: 33890304, PeakPagefileUsage: 107102208 - Working Set Size - WorkingSetSize: 4014080, PeakWorkingSetSize: 4014080, PageFaultCount: 1001 * Dump of thread ID 85608 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 312002.000000, User Time: 0.000000, Wait Time: 77556960.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7633FC16 - Registers - eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe eip=7633fc16 esp=1132f9c4 ebp=1132fa14 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000212 - Callstack - ChildEBP RetAddr Args to Child 1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 kernel32!+0x0 1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 1132fad8 00408130 1132faec 00072a41 1132faec 000729c9 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff88 7634d2e9 7ffdd000 1132ffd4 77391603 7ffdd000 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff94 77391603 7ffdd000 2e6e22e9 00000000 00000000 kernel32!+0x0 1132ffd4 773915d6 007bde7f 7ffdd000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 1132ffec 00000000 007bde7f 7ffdd000 00000000 00000030 ntdll!RtlInitializeExceptionChain+0x0 * Dump of thread ID 85920 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 77556960.000000 - Registers - eax=00749910 ebx=00000000 ecx=00000000 edx=00000000 esi=3365ff48 edi=00000000 eip=773b5cd4 esp=3365ff04 ebp=3365ff6c cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 3365ff6c 76301c6c 00000064 00000000 3365ff88 00749924 ntdll!KiFastSystemCallRet+0x0 3365ff7c 00749924 00000064 3365ff94 7634d2e9 00000000 kernel32!+0x0 3365ff88 7634d2e9 00000000 3365ffd4 77391603 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 3365ff94 77391603 00000000 0c3922e9 00000000 00000000 kernel32!+0x0 3365ffd4 773915d6 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 3365ffec 00000000 00749910 00000000 00000000 33b70000 ntdll!RtlInitializeExceptionChain+0x0 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=84728, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7633FC16 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 07/27/13 07:13:40 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : Loaded Library : C:\Program Files\BOINC\\dbghelp.dll Loaded Library : C:\Program Files\BOINC\\symsrv.dll Loaded Library : C:\Program Files\BOINC\\srcsrv.dll LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_n6bf_1960_40_008402473;C:\ProgramData\BOINC\projects\climateprediction.net ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : ModLoad: 77350000 00128000 C:\Windows\system32\ntdll.dll (6.0.6002.18541) (-exported- Symbols Loaded) Linked PDB Filename : ntdll.pdb File Version : 6.0.6002.18541 (vistasp2_gdr.111116-0305) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18541 ModLoad: 76300000 000dc000 C:\Windows\system32\kernel32.dll (6.0.6002.18704) (-nosymbols- Symbols Loaded) Linked PDB Filename : kernel32.pdb File Version : 6.0.6002.18704 (vistasp2_gdr.120928-0336) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18704 ModLoad: 76690000 0009d000 C:\Windows\system32\USER32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : user32.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 751e0000 0004b000 C:\Windows\system32\GDI32.dll (6.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : gdi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 ModLoad: 753f0000 000c6000 C:\Windows\system32\ADVAPI32.dll (6.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 ModLoad: 76870000 000c3000 C:\Windows\system32\RPCRT4.dll (6.0.6002.18024) (-nosymbols- Symbols Loaded) Linked PDB Filename : rpcrt4.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 5d2c0000 000cb000 C:\Windows\system32\OPENGL32.dll (6.0.6000.16386) (-nosymbols- Symbols Loaded) Linked PDB Filename : opengl32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 767c0000 000aa000 C:\Windows\system32\msvcrt.dll (7.0.6002.18551) (-nosymbols- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.6002.18551 (vistasp2_gdr.111214-0301) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 7.0.6002.18551 ModLoad: 5d260000 00023000 C:\Windows\system32\GLU32.dll (6.0.6000.16386) (-nosymbols- Symbols Loaded) Linked PDB Filename : glu32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 5d170000 000e5000 C:\Windows\system32\DDRAW.dll (6.0.6001.18000) (-nosymbols- Symbols Loaded) Linked PDB Filename : ddraw.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 5d160000 00006000 C:\Windows\system32\DCIMAN32.dll (6.0.6002.18051) (-nosymbols- Symbols Loaded) Linked PDB Filename : dciman32.pdb File Version : 6.0.6002.18051 (vistasp2_gdr.090615-0258) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18051 ModLoad: 763e0000 0018a000 C:\Windows\system32\SETUPAPI.dll (6.0.6002.18005) (-nosymbols- Symbols Loaded) Linked PDB Filename : setupapi.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 Get Product Name Failed. ModLoad: 76730000 0008d000 C:\Windows\system32\OLEAUT32.dll (6.0.6002.18508) (-nosymbols- Symbols Loaded) Linked PDB Filename : oleaut32.pdb File Version : 6.0.6002.18508 Company Name : Microsoft Corporation Product Name : Product Version : 6.0.6002.18508 ModLoad: 76060000 00145000 C:\Windows\system32\ole32.dll (6.0.6002.18277) (-nosymbols- Symbols Loaded) Linked PDB Filename : ole32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 73a10000 0000c000 C:\Windows\system32\dwmapi.dll (6.0.6001.18000) (-exported- Symbols Loaded) Linked PDB Filename : dwmapi.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 4, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 221, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 106312, QuotaPeakPagedPoolUsage: 106368 QuotaNonPagedPoolUsage: 2880, QuotaPeakNonPagedPoolUsage: 2880 - Virtual Memory Usage - VirtualSize: 895770624, PeakVirtualSize: 895770624 - Pagefile Usage - PagefileUsage: 33894400, PeakPagefileUsage: 107102208 - Working Set Size - WorkingSetSize: 4046848, PeakWorkingSetSize: 4046848, PageFaultCount: 1009 * Dump of thread ID 85832 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 312002.000000, Wait Time: 77559088.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7633FC16 - Registers - eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe eip=7633fc16 esp=1132f9c4 ebp=1132fa14 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000212 - Callstack - ChildEBP RetAddr Args to Child 1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 kernel32!+0x0 1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 1132fad8 00408130 1132faec 001e2a41 1132faec 001e29c9 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff88 7634d2e9 7ffda000 1132ffd4 77391603 7ffda000 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff94 77391603 7ffda000 2e6da03a 00000000 00000000 kernel32!+0x0 1132ffd4 773915d6 007bde7f 7ffda000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 1132ffec 00000000 007bde7f 7ffda000 00000000 00000030 ntdll!RtlInitializeExceptionChain+0x0 * Dump of thread ID 85160 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 77559088.000000 - Registers - eax=00749910 ebx=00000000 ecx=00000000 edx=00000000 esi=3367ff48 edi=00000000 eip=773b5cd4 esp=3367ff04 ebp=3367ff6c cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 3367ff6c 76301c6c 00000064 00000000 3367ff88 00749924 ntdll!KiFastSystemCallRet+0x0 3367ff7c 00749924 00000064 3367ff94 7634d2e9 00000000 kernel32!+0x0 3367ff88 7634d2e9 00000000 3367ffd4 77391603 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 3367ff94 77391603 00000000 0c38a03a 00000000 00000000 kernel32!+0x0 3367ffd4 773915d6 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 3367ffec 00000000 00749910 00000000 00000000 33b90000 ntdll!RtlInitializeExceptionChain+0x0 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22764, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22764, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22764, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22764, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22764, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22764, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Jul 2013 13:51:05	969347	15901475	hadcm3n_n6bf_1960_40_008402473_1	233,280	436,301	1.8703
29 Jul 2013 13:51:03	969347	15901475	hadcm3n_n6bf_1960_40_008402473_1	207,360	386,100	1.8620
29 Jul 2013 13:51:03	969347	15901475	hadcm3n_n6bf_1960_40_008402473_1	181,440	337,696	1.8612
26 Jul 2013 09:40:25	969347	15901475	hadcm3n_n6bf_1960_40_008402473_1	155,520	289,037	1.8585
25 Jul 2013 18:03:23	969347	15901475	hadcm3n_n6bf_1960_40_008402473_1	129,600	241,530	1.8637
25 Jul 2013 03:47:44	969347	15901475	hadcm3n_n6bf_1960_40_008402473_1	103,680	193,835	1.8696
24 Jul 2013 14:33:11	969347	15901475	hadcm3n_n6bf_1960_40_008402473_1	77,760	145,812	1.8752
24 Jul 2013 00:10:03	969347	15901475	hadcm3n_n6bf_1960_40_008402473_1	51,840	96,305	1.8577
23 Jul 2013 22:06:58	969347	15901475	hadcm3n_n6bf_1960_40_008402473_1	25,920	47,979	1.8510