climateprediction.net home page
Task 18852531

Task 18852531

Name hadam3p_pnw_xaiz_2001_1_010141299_1
Workunit 10102699
Created 21 Aug 2015, 22:36:32 UTC
Sent 22 Aug 2015, 0:33:52 UTC
Report deadline 3 Aug 2016, 5:53:52 UTC
Received 13 Sep 2015, 23:17:48 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1373228
Run time 12 days 5 hours 54 min 36 sec
CPU time 8 days 17 hours 22 min 14 sec
Validate state Workunit error - check skipped
Credit 3,010.29
Device peak FLOPS 2.74 GFLOPS
Application version UK Met Office HadAM3P-HadRM3P Pacific North West v7.27
windows_intelx86
Stderr 
<core_client_version>7.6.6</core_client_version>
<![CDATA[
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x00462B87 read attempt to address 0x011033AC

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.5.1


Dump Timestamp    : 08/24/15 00:38:31
Install Directory : 
Data Directory    : F:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( F:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( F:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( F:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( F:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: F:\ProgramData\BOINC\slots\13;F:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 000000000018e000 F:\ProgramData\BOINC\projects\climateprediction.net\hadam3p_pnw_7.27_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000077a20000 0000000000179000 C:\Windows\SYSTEM32\ntdll.dll (6.2.10240.16430) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.10240.16384

ModLoad: 00000000775e0000 00000000000f0000 C:\Windows\SYSTEM32\KERNEL32.DLL (6.2.10240.16384) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.10240.16384

ModLoad: 0000000076740000 0000000000176000 C:\Windows\SYSTEM32\KERNELBASE.dll (6.2.10240.16384) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.10240.16384

ModLoad: 0000000072900000 0000000000091000 C:\Windows\system32\apphelp.dll (6.2.10240.16384) (-exported- Symbols Loaded)
    Linked PDB Filename   : apphelp.pdb
    File Version          : 10.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.10240.16384

ModLoad: 0000000076be0000 0000000000140000 C:\Windows\SYSTEM32\USER32.dll (6.2.10240.16384) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.10240.16384

ModLoad: 0000000076920000 000000000014d000 C:\Windows\SYSTEM32\GDI32.dll (6.2.10240.16390) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.10240.16390 (th1_st1.150714-1601)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10240.16390

ModLoad: 00000000776f0000 000000000007b000 C:\Windows\SYSTEM32\ADVAPI32.dll (6.2.10240.16384) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.10240.16384

ModLoad: 0000000076db0000 00000000000be000 C:\Windows\SYSTEM32\msvcrt.dll (7.0.10240.16384) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.10240.16384

ModLoad: 0000000077920000 0000000000043000 C:\Windows\SYSTEM32\sechost.dll (6.2.10240.16384) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10240.16384

ModLoad: 0000000077970000 00000000000ac000 C:\Windows\SYSTEM32\RPCRT4.dll (6.2.10240.16412) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.10240.16384

ModLoad: 0000000074b30000 000000000001e000 C:\Windows\SYSTEM32\SspiCli.dll (6.2.10240.16384) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10240.16384

ModLoad: 0000000074b20000 000000000000a000 C:\Windows\SYSTEM32\CRYPTBASE.dll (6.2.10240.16384) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10240.16384

ModLoad: 0000000074ac0000 0000000000059000 C:\Windows\SYSTEM32\bcryptPrimitives.dll (6.2.10240.16384) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10240.16384

ModLoad: 00000000722b0000 0000000000028000 C:\Windows\SYSTEM32\ntmarta.dll (6.2.10240.16384) (-exported- Symbols Loaded)
    Linked PDB Filename   : ntmarta.pdb
    File Version          : 10.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.10240.16384

ModLoad: 0000000074980000 000000000013f000 C:\Windows\SYSTEM32\dbghelp.dll (6.2.10240.16384) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10240.16384

ModLoad: 0000000074970000 0000000000008000 C:\Windows\SYSTEM32\version.dll (6.2.10240.16384) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.10240.16384 (th1.150709-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10240.16384



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 15065, Write: 0, Other 52503

- I/O Transfers Counters -
Read: 0, Write: 2943, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 95056, QuotaPeakPagedPoolUsage: 110552
QuotaNonPagedPoolUsage: 8280, QuotaPeakNonPagedPoolUsage: 9232

- Virtual Memory Usage -
VirtualSize: 53788672, PeakVirtualSize: 61997056

- Pagefile Usage -
PagefileUsage: 2822144, PeakPagefileUsage: 3268608

- Working Set Size -
WorkingSetSize: 25288704, PeakWorkingSetSize: 31608832, PageFaultCount: 10863

*** Dump of thread ID 6852 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 156250.000000, User Time: 0.000000, Wait Time: 10661360.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x00462B87 read attempt to address 0x011033AC

- Registers -
eax=00000015 ebx=fffffffe ecx=00000000 edx=00000000 esi=01033020 edi=00000000
eip=00462b87 esp=0095fe7c ebp=0095fe80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010216

- Callstack -
ChildEBP RetAddr  Args to Child
0095fe80 00447d6f 01033020 00000000 01033020 fffffffe hadam3p_pnw_7.27_windows_intelx!+0x0 
0095fe98 0044f568 01033020 00000000 005597a4 0095fee8 hadam3p_pnw_7.27_windows_intelx!+0x0 
0095feac 0048374c 00000015 415cd407 00000006 004836bc hadam3p_pnw_7.27_windows_intelx!+0x0 
0095fee8 76852b7f 00000006 3799355c 768529c0 768529c0 hadam3p_pnw_7.27_windows_intelx!+0x0 
0095ff80 775f3744 80000006 775f3720 36793654 0095ffdc KERNELBASE!CtrlRoutine+0x0 
0095ff94 77a7a064 80000006 36a2e086 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0095ffdc 77a7a02f ffffffff 77a9d7eb 00000000 00000000 ntdll!RtlSetCurrentTransaction+0x0 
0095ffec 00000000 768529c0 80000006 00000000 00000000 ntdll!RtlSetCurrentTransaction+0x0 

*** Dump of thread ID 12292 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 22031250.000000, User Time: 90156248.000000, Wait Time: 10661350.000000

- Registers -
eax=00000000 ebx=0019f988 ecx=00000000 edx=00000000 esi=00000000 edi=000000cd
eip=77a88f2c esp=0019f948 ebp=0019f9ac
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
0019f9ac 767f131f 000000c8 00000000 0019f9d0 004689c1 ntdll!NtDelayExecution+0x0 
0019f9bc 004689c1 000000c8 000000c8 00000000 0019fa1c KERNELBASE!Sleep+0x0 
0019f9d0 00403970 a0000000 027f9999 00000000 004cf850 hadam3p_pnw_7.27_windows_intelx!+0x0 
0019fa1c 00403dcd 00000001 004cf858 004cfb88 00000000 hadam3p_pnw_7.27_windows_intelx!+0x0 
0019fa50 00404d76 00000000 41d0d7af 00484e6e 00000000 hadam3p_pnw_7.27_windows_intelx!+0x0 
0019fd40 00404f67 0019fef0 00404fe5 004d02a8 0019fd60 hadam3p_pnw_7.27_windows_intelx!+0x0 
0019fd48 00404fe5 004d02a8 0019fd60 00662818 0019fd60 hadam3p_pnw_7.27_windows_intelx!+0x0 
0019fef0 00484e1b 00400000 00000000 0066285d 0000000a hadam3p_pnw_7.27_windows_intelx!+0x0 
0019ff80 775f3744 7ffde000 775f3720 36f53654 0019ffdc hadam3p_pnw_7.27_windows_intelx!+0x0 
0019ff94 77a7a064 7ffde000 362ee086 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 77a7a02f ffffffff 77a9d7eb 00000000 00000000 ntdll!RtlSetCurrentTransaction+0x0 
0019ffec 00000000 00484e6e 7ffde000 00000000 78746341 ntdll!RtlSetCurrentTransaction+0x0 

*** Dump of thread ID 12316 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 156250.000000, Wait Time: 10661354.000000

- Registers -
eax=c0000022 ebx=0102fd1c ecx=00000000 edx=00000000 esi=00000000 edi=00000001
eip=77a88f2c esp=0102fcdc ebp=0102fd40
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0102fd40 767f131f 00000064 00000000 0102ff80 00477e94 ntdll!NtDelayExecution+0x0 
0102fd50 00477e94 00000064 00477e70 00477e70 00000000 KERNELBASE!Sleep+0x0 
0102ff80 775f3744 00000000 775f3720 37ee3654 0102ffdc hadam3p_pnw_7.27_windows_intelx!+0x0 
0102ff94 77a7a064 00000000 3735e086 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0102ffdc 77a7a02f ffffffff 77a9d7eb 00000000 00000000 ntdll!RtlSetCurrentTransaction+0x0 
0102ffec 00000000 00477e70 00000000 00000000 00000000 ntdll!RtlSetCurrentTransaction+0x0 

*** Dump of thread ID 12240 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 10661351.000000

- Registers -
eax=00000000 ebx=0066d870 ecx=00000000 edx=00000000 esi=0066d6b0 edi=00662ff0
eip=77a8a74c esp=0085fdcc ebp=0085ff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
0085ff80 775f3744 00662ff0 775f3720 36693654 0085ffdc ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
0085ff94 77a7a064 00662ff0 36b2e086 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0085ffdc 77a7a02f ffffffff 77a9d7eb 00000000 00000000 ntdll!RtlSetCurrentTransaction+0x0 
0085ffec 00000000 77a545b0 00662ff0 00000000 00000000 ntdll!RtlSetCurrentTransaction+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
00:52:39 (9536): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
20:35:29 (7088): start_timer_thread(): CreateThread() failed, errno 0
20:35:30 (9200): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
21:06:52 (6680): start_timer_thread(): CreateThread() failed, errno 0
21:06:53 (7188): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
05:35:35 (7244): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
05:46:45 (3900): start_timer_thread(): CreateThread() failed, errno 0
05:46:46 (6176): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
06:09:50 (5504): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
13:58:53 (19056): start_timer_thread(): CreateThread() failed, errno 0
13:58:55 (13752): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
22:51:53 (9484): start_timer_thread(): CreateThread() failed, errno 0
22:51:55 (9588): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9484, selfPID=9484, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9484, selfPID=9588, iMonCtr=1
11:37:48 (6088): start_timer_thread(): CreateThread() failed, errno 0
11:37:49 (8980): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
03:01:35 (5944): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
19:53:54 (12964): start_timer_thread(): CreateThread() failed, errno 0
19:53:56 (5376): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14676, selfPID=14660, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14676, selfPID=14676, iMonCtr=1
21:55:36 (6660): start_timer_thread(): CreateThread() failed, errno 0
21:55:38 (12592): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
03:05:13 (2596): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12312, selfPID=12312, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12312, selfPID=2596, iMonCtr=1
diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12
WARNING: BOINC Windows Runtime Debugger has been disabled.
09:42:02 (10860): start_timer_thread(): CreateThread() failed, errno 0
diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12
WARNING: BOINC Windows Runtime Debugger has been disabled.
09:42:04 (10960): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
07:31:43 (11308): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
08:36:30 (3916): start_timer_thread(): CreateThread() failed, errno 0
08:36:31 (980): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3916, selfPID=980, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3916, selfPID=3916, iMonCtr=1
04:43:36 (1780): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1780, selfPID=1780, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1780, selfPID=7248, iMonCtr=1
11:35:51 (4836): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
16:13:33 (8120): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Leaving CPDN_Main::Monitor...
00:15:50 (5744): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
13 Sep 2015 22:08:07 1373228 18852531 hadam3p_pnw_xaiz_2001_1_010141299_1 138,539 753,302 5.4375
11 Sep 2015 21:28:42 1373228 18852531 hadam3p_pnw_xaiz_2001_1_010141299_1 127,019 692,393 5.4511
09 Sep 2015 11:14:50 1373228 18852531 hadam3p_pnw_xaiz_2001_1_010141299_1 115,499 634,587 5.4943
07 Sep 2015 20:27:50 1373228 18852531 hadam3p_pnw_xaiz_2001_1_010141299_1 103,979 569,673 5.4787
05 Sep 2015 22:21:13 1373228 18852531 hadam3p_pnw_xaiz_2001_1_010141299_1 92,459 508,953 5.5046
04 Sep 2015 05:13:13 1373228 18852531 hadam3p_pnw_xaiz_2001_1_010141299_1 80,939 452,548 5.5912
03 Sep 2015 00:08:34 1373228 18852531 hadam3p_pnw_xaiz_2001_1_010141299_1 69,419 398,417 5.7393
01 Sep 2015 14:18:50 1373228 18852531 hadam3p_pnw_xaiz_2001_1_010141299_1 57,899 341,391 5.8963
28 Aug 2015 13:09:35 1373228 18852531 hadam3p_pnw_xaiz_2001_1_010141299_1 46,379 277,392 5.9810
27 Aug 2015 16:07:48 1373228 18852531 hadam3p_pnw_xaiz_2001_1_010141299_1 34,859 213,269 6.1180
26 Aug 2015 00:17:45 1373228 18852531 hadam3p_pnw_xaiz_2001_1_010141299_1 23,339 142,406 6.1016
25 Aug 2015 03:03:58 1373228 18852531 hadam3p_pnw_xaiz_2001_1_010141299_1 11,819 71,615 6.0593


©2024 climateprediction.net