climateprediction.net home page
Task 20793998

Task 20793998

Name wah2_pnw25_rn1p_200412_24_666_011291488_0
Workunit 11291488
Created 1 Oct 2017, 13:39:04 UTC
Sent 2 Oct 2017, 6:41:03 UTC
Report deadline 14 Sep 2018, 12:01:03 UTC
Received 13 Oct 2017, 4:02:47 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1436093
Run time 2 days 16 hours 44 min 19 sec
CPU time 2 days 11 hours 25 min 2 sec
Validate state Invalid
Credit 15,978.39
Device peak FLOPS 5.17 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 187.88 MB
Peak swap size 165.93 MB
Peak disk usage 1.64 MB
Stderr 
<core_client_version>7.8.2</core_client_version>
<![CDATA[
<message>
finish file present too long
</message>
<stderr_txt>
19:24:37 (11100): start_timer_thread(): CreateThread() failed, errno 0
19:24:39 (14648): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11100, selfPID=11100, iMonCtr=1
No Process Handle
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11100, selfPID=14648, iMonCtr=1
17:05:35 (1428): start_timer_thread(): CreateThread() failed, errno 0
17:05:38 (18776): start_timer_thread(): CreateThread() failed, errno 0
17:09:52 (19064): start_timer_thread(): CreateThread() failed, errno 0
17:09:54 (18756): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
16:28:14 (16912): start_timer_thread(): CreateThread() failed, errno 0
16:28:15 (12964): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=16912, selfPID=12964, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=16912, selfPID=16912, iMonCtr=1
19:11:25 (7036): start_timer_thread(): CreateThread() failed, errno 0
19:11:27 (16392): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
12:31:06 (11820): start_timer_thread(): CreateThread() failed, errno 0
12:31:07 (17480): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11820, selfPID=11820, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
18:40:35 (7560): start_timer_thread(): CreateThread() failed, errno 0
18:40:37 (15032): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11828, iMonCtr=2
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
04:41:24 (11828): called boinc_finish(0)
CPDN Monitor - Abort request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0268E12C

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 10/12/17 18:54:55
Install Directory : 
Data Directory    : F:\Boinc projects
Project Symstore  : 
LoadLibraryA( F:\Boinc projects\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( F:\Boinc projects\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( F:\Boinc projects\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( F:\Boinc projects\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: F:\Boinc projects\projects\climateprediction.net;F:\Boinc projects\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 F:\Boinc projects\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000077930000 000000000018e000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.15063.608) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.608

ModLoad: 00000000764d0000 00000000000d0000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.15063.296) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.608

ModLoad: 00000000755b0000 00000000001c2000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.608

ModLoad: 0000000076390000 000000000013c000 C:\WINDOWS\System32\USER32.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000074960000 0000000000016000 C:\WINDOWS\System32\win32u.dll (6.2.15063.608) (-exported- Symbols Loaded)
    Linked PDB Filename   : wwin32u.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.608

ModLoad: 00000000752e0000 0000000000021000 C:\WINDOWS\System32\GDI32.dll (6.2.15063.483) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.15063.483 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.483

ModLoad: 0000000074c80000 0000000000158000 C:\WINDOWS\System32\gdi32full.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32full.pdb
    File Version          : 10.0.15063.674 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.674

ModLoad: 0000000075e20000 0000000000079000 C:\WINDOWS\System32\msvcp_win.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000075430000 0000000000118000 C:\WINDOWS\System32\ucrtbase.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.15063.674 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.674

ModLoad: 0000000074660000 0000000000077000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000075fd0000 00000000000bd000 C:\WINDOWS\System32\msvcrt.dll (7.0.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.15063.0

ModLoad: 0000000075f70000 0000000000041000 C:\WINDOWS\System32\sechost.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000075ea0000 00000000000c0000 C:\WINDOWS\System32\RPCRT4.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000074310000 0000000000020000 C:\WINDOWS\System32\SspiCli.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.15063.674 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.674

ModLoad: 0000000074300000 000000000000a000 C:\WINDOWS\System32\CRYPTBASE.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 00000000749e0000 0000000000057000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.15063.138) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.15063.138 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.138

ModLoad: 0000000075400000 0000000000025000 C:\WINDOWS\System32\IMM32.DLL (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 000000006f040000 000000000015d000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.15063.447) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.15063.447 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.447

ModLoad: 00000000742f0000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 5297, Write: 0, Other 60042

- I/O Transfers Counters -
Read: 0, Write: 1695, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 90512, QuotaPeakPagedPoolUsage: 90640
QuotaNonPagedPoolUsage: 8624, QuotaPeakNonPagedPoolUsage: 8912

- Virtual Memory Usage -
VirtualSize: 55590912, PeakVirtualSize: 55590912

- Pagefile Usage -
PagefileUsage: 3932160, PeakPagefileUsage: 3932160

- Working Set Size -
WorkingSetSize: 6443008, PeakWorkingSetSize: 6443008, PageFaultCount: 1893

*** Dump of thread ID 7260 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 2968750.000000, User Time: 12187500.000000, Wait Time: 6122254.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0268E12C

- Registers -
eax=00000065 ebx=004b63b9 ecx=0019d868 edx=0268e12c esi=0019d620 edi=0019d898
eip=0048ee46 esp=0019d5b4 ebp=0019d844
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

- Callstack -
ChildEBP RetAddr  Args to Child
0019d844 0048161c 0019d868 004b63b8 00000000 0019d898 wah2_8.24_windows_intelx86!+0x0 
0019d888 0044fa52 0268e12c 004b63b8 004b6484 00000421 wah2_8.24_windows_intelx86!+0x0 
0019d8a8 00496e90 0000000b 00000000 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0019d8c0 00485ca1 c0000005 0019d8f4 00490c54 00000000 wah2_8.24_windows_intelx86!+0x0 
0019ff80 764e8744 0036a000 764e8720 0d3d947f 0019ffdc wah2_8.24_windows_intelx86!+0x0 
0019ff94 7799582d 0036a000 b27da328 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 779957fd ffffffff 779b636d 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0019ffec 00000000 00485cce 0036a000 00000000 78746341 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 13552 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 6120778.000000

- Registers -
eax=00000000 ebx=00564598 ecx=00000000 edx=00000000 esi=00564a60 edi=00564c20
eip=779a396c esp=0089fd7c ebp=0089ff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0089ff80 764e8744 00564598 764e8720 0dad947f 0089ffdc ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
0089ff94 7799582d 00564598 b2eda328 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0089ffdc 779957fd ffffffff 779b636d 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0089ffec 00000000 7796a470 00564598 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 18736 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 6120778.000000

- Registers -
eax=00000000 ebx=00564598 ecx=00000000 edx=00000000 esi=005655f0 edi=005657b0
eip=779a396c esp=009dfd7c ebp=009dff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
009dff80 764e8744 00564598 764e8720 0db9947f 009dffdc ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
009dff94 7799582d 00564598 b2f9a328 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
009dffdc 779957fd ffffffff 779b636d 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
009dffec 00000000 7796a470 00564598 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 404 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 6120778.000000

- Registers -
eax=00000000 ebx=00564598 ecx=00000000 edx=00000000 esi=00565a18 edi=00565bd8
eip=779a396c esp=00b1fd7c ebp=00b1ff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
00b1ff80 764e8744 00564598 764e8720 0d95947f 00b1ffdc ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
00b1ff94 7799582d 00564598 b2d5a328 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00b1ffdc 779957fd ffffffff 779b636d 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
00b1ffec 00000000 7796a470 00564598 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 11104 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 6122251.000000

- Registers -
eax=00000000 ebx=025bfd1c ecx=00000000 edx=00000000 esi=00000000 edi=00000001
eip=779a205c esp=025bfcdc ebp=025bfd40
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
025bfd40 756a358f 00000064 00000000 025bff80 00478b44 ntdll!NtDelayExecution+0x0 
025bfd50 00478b44 00000064 00478b20 00478b20 00000000 KERNELBASE!Sleep+0x0 
025bff80 764e8744 00000000 764e8720 0f7f947f 025bffdc wah2_8.24_windows_intelx86!+0x0 
025bff94 7799582d 00000000 b03fa328 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
025bffdc 779957fd ffffffff 779b636d 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
025bffec 00000000 00478b20 00000000 00000000 027a9000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
17 Oct 2017 11:48:01 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 242,219 212,031 0.8754
17 Oct 2017 11:34:52 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 230,699 201,334 0.8727
17 Oct 2017 11:20:01 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 219,179 191,184 0.8723
17 Oct 2017 11:15:31 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 207,659 181,221 0.8727
10 Oct 2017 04:48:08 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 196,139 171,242 0.8731
10 Oct 2017 01:49:54 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 184,619 161,220 0.8733
09 Oct 2017 07:29:16 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 173,099 151,124 0.8730
09 Oct 2017 04:11:02 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 161,579 141,197 0.8739
09 Oct 2017 00:37:40 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 150,059 130,996 0.8730
08 Oct 2017 08:42:08 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 138,539 121,213 0.8749
08 Oct 2017 05:58:47 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 127,019 111,321 0.8764
08 Oct 2017 00:32:09 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 115,499 101,459 0.8784
07 Oct 2017 21:46:03 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 103,979 91,705 0.8820
07 Oct 2017 09:16:00 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 92,459 81,903 0.8858
07 Oct 2017 00:32:56 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 80,939 72,017 0.8898
06 Oct 2017 06:34:38 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 69,419 62,176 0.8957
05 Oct 2017 10:17:32 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 57,899 52,291 0.9031
05 Oct 2017 07:05:24 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 46,379 42,361 0.9134
04 Oct 2017 23:52:22 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 34,859 32,121 0.9215
03 Oct 2017 10:53:16 1436093 20793998 wah2_pnw25_rn1p_200412_24_666_011291488_0 23,339 21,556 0.9236


©2024 climateprediction.net