climateprediction.net home page
Task 20774998

Task 20774998

Name wah2_pnw25_rdxx_200012_24_664_011274184_0
Workunit 11274184
Created 28 Sep 2017, 22:49:53 UTC
Sent 5 Nov 2017, 3:09:41 UTC
Report deadline 18 Oct 2018, 8:29:41 UTC
Received 20 Nov 2017, 1:30:37 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1436093
Run time 2 days 16 hours 30 min 2 sec
CPU time 2 days 11 hours 1 min 25 sec
Validate state Invalid
Credit 15,218.45
Device peak FLOPS 5.13 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 501.25 MB
Peak swap size 463.21 MB
Peak disk usage 0.02 MB
Stderr 
<core_client_version>7.8.3</core_client_version>
<![CDATA[
<message>
finish file present too long</message>
<stderr_txt>
20:32:53 (8244): start_timer_thread(): CreateThread() failed, errno 0
20:32:54 (11916): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
17:06:15 (19700): start_timer_thread(): CreateThread() failed, errno 0
17:06:16 (19692): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
22:49:01 (14712): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14712, selfPID=14712, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14712, selfPID=18928, iMonCtr=1
17:30:00 (9584): start_timer_thread(): CreateThread() failed, errno 0
17:30:01 (17776): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Geglonol Wo Wokrr:: :: CPDN cpss ie ss is not r,uexnning, exitialg,  1, tVhel = 1, chec8k selfPID=, sel8,PIiMo12t3=88
, iMonCtr=1
16:36:34 (19868): Can't acquire lockfile (32) - waiting 35s
16:37:09 (19868): Can't acquire lockfile (32) - exiting
16:37:09 (19868): Error: The process cannot access the file because it is being used by another process.

 (0x20)
16:37:45 (11340): Can't acquire lockfile (32) - waiting 35s
16:38:20 (11340): Can't acquire lockfile (32) - exiting
16:38:20 (11340): Error: The process cannot access the file because it is being used by another process.

 (0x20)
21:00:11 (14852): BOINC client no longer exists - exiting
21:00:11 (14852): timer handler: client dead, exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11604, selfPID=11604, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11604, selfPID=5968, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=20048, selfPID=20048, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=20048, selfPID=16408, iMonCtr=1
01:33:52 (11348): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
19:08:10 (2012): start_timer_thread(): CreateThread() failed, errno 0
19:08:11 (2836): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
04:16:01 (15500): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
18:24:06 (2984): start_timer_thread(): CreateThread() failed, errno 0
18:24:07 (9672): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9672, selfPID=2960, iMonCtr=1
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
04:39:15 (2960): called boinc_finish(0)
CPDN Monitor - Abort request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0264B12C

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 11/19/17 16:30:32
Install Directory : 
Data Directory    : F:\Boinc projects
Project Symstore  : 
LoadLibraryA( F:\Boinc projects\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( F:\Boinc projects\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( F:\Boinc projects\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( F:\Boinc projects\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: F:\Boinc projects\projects\climateprediction.net;F:\Boinc projects\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 F:\Boinc projects\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000077130000 000000000018e000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.15063.608) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.608

ModLoad: 00000000767b0000 00000000000d0000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.15063.296) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.608

ModLoad: 0000000076580000 00000000001c2000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.15063.726) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.608

ModLoad: 0000000076270000 000000000013c000 C:\WINDOWS\System32\USER32.dll (6.2.15063.726) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 00000000759d0000 0000000000016000 C:\WINDOWS\System32\win32u.dll (6.2.15063.608) (-exported- Symbols Loaded)
    Linked PDB Filename   : wwin32u.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.608

ModLoad: 0000000073e70000 0000000000021000 C:\WINDOWS\System32\GDI32.dll (6.2.15063.483) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.15063.483 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.483

ModLoad: 0000000076ca0000 0000000000158000 C:\WINDOWS\System32\gdi32full.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32full.pdb
    File Version          : 10.0.15063.674 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.674

ModLoad: 00000000761f0000 0000000000079000 C:\WINDOWS\System32\msvcp_win.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000075c90000 0000000000118000 C:\WINDOWS\System32\ucrtbase.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.15063.674 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.674

ModLoad: 0000000073f50000 0000000000077000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000076e00000 00000000000bd000 C:\WINDOWS\System32\msvcrt.dll (7.0.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.15063.0

ModLoad: 00000000763b0000 0000000000041000 C:\WINDOWS\System32\sechost.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000075910000 00000000000c0000 C:\WINDOWS\System32\RPCRT4.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000073c00000 0000000000020000 C:\WINDOWS\System32\SspiCli.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.15063.674 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.674

ModLoad: 0000000073bf0000 000000000000a000 C:\WINDOWS\System32\CRYPTBASE.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000076750000 0000000000057000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.15063.726) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.15063.726 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.726

ModLoad: 0000000076ec0000 0000000000025000 C:\WINDOWS\System32\IMM32.DLL (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 000000006dbb0000 000000000015d000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.15063.447) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.15063.447 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.447

ModLoad: 0000000073be0000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 3948, Write: 0, Other 50178

- I/O Transfers Counters -
Read: 0, Write: 1684, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 90512, QuotaPeakPagedPoolUsage: 90640
QuotaNonPagedPoolUsage: 8624, QuotaPeakNonPagedPoolUsage: 8912

- Virtual Memory Usage -
VirtualSize: 55513088, PeakVirtualSize: 55513088

- Pagefile Usage -
PagefileUsage: 3796992, PeakPagefileUsage: 3796992

- Working Set Size -
WorkingSetSize: 6385664, PeakWorkingSetSize: 6385664, PageFaultCount: 1881

*** Dump of thread ID 2720 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 2656250.000000, User Time: 9218750.000000, Wait Time: 5662250.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0264B12C

- Registers -
eax=00000065 ebx=004b63b9 ecx=0019d868 edx=0264b12c esi=0019d620 edi=0019d898
eip=0048ee46 esp=0019d5b4 ebp=0019d844
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

- Callstack -
ChildEBP RetAddr  Args to Child
0019d844 0048161c 0019d868 004b63b8 00000000 0019d898 wah2_8.24_windows_intelx86!+0x0 
0019d888 0044fa52 0264b12c 004b63b8 004b6484 00000421 wah2_8.24_windows_intelx86!+0x0 
0019d8a8 00496e90 0000000b 00000000 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0019d8c0 00485ca1 c0000005 0019d8f4 00490c54 00000000 wah2_8.24_windows_intelx86!+0x0 
0019ff80 767c8744 00307000 767c8720 fa34e440 0019ffdc wah2_8.24_windows_intelx86!+0x0 
0019ff94 7719582d 00307000 4041f716 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 771957fd ffffffff 771b6386 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0019ffec 00000000 00485cce 00307000 00000000 78746341 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 14980 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 5661054.000000

- Registers -
eax=00000000 ebx=00604598 ecx=00000000 edx=00000000 esi=00605388 edi=00605548
eip=771a396c esp=008cfd7c ebp=008cff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
008cff80 767c8744 00604598 767c8720 faa1e440 008cffdc ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
008cff94 7719582d 00604598 40d4f716 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
008cffdc 771957fd ffffffff 771b6386 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
008cffec 00000000 7716a470 00604598 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 16840 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 5661054.000000

- Registers -
eax=00000000 ebx=00604598 ecx=00000000 edx=00000000 esi=00609d28 edi=00609ee8
eip=771a396c esp=009cfd7c ebp=009cff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
009cff80 767c8744 00604598 767c8720 fab1e440 009cffdc ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
009cff94 7719582d 00604598 40c4f716 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
009cffdc 771957fd ffffffff 771b6386 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
009cffec 00000000 7716a470 00604598 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 8260 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 5661054.000000

- Registers -
eax=00000000 ebx=00604598 ecx=00000000 edx=00000000 esi=00609f68 edi=00000000
eip=771a396c esp=00b0fd7c ebp=00b0ff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
00b0ff80 767c8744 00604598 767c8720 fa9de440 00b0ffdc ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
00b0ff94 7719582d 00604598 40e8f716 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00b0ffdc 771957fd ffffffff 771b6386 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
00b0ffec 00000000 7716a470 00604598 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 20240 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 5662244.000000

- Registers -
eax=00000000 ebx=0257fd1c ecx=00000000 edx=00000000 esi=00000000 edi=00000001
eip=771a205c esp=0257fcdc ebp=0257fd40
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0257fd40 766748cf 00000064 00000000 0257ff80 00478b44 ntdll!NtDelayExecution+0x0 
0257fd50 00478b44 00000064 00478b20 00478b20 00000000 KERNELBASE!Sleep+0x0 
0257ff80 767c8744 00000000 767c8720 f87ae440 0257ffdc wah2_8.24_windows_intelx86!+0x0 
0257ff94 7719582d 00000000 420ff716 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0257ffdc 771957fd ffffffff 771b6386 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0257ffec 00000000 00478b20 00000000 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
19 Nov 2017 11:15:08 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 230,699 207,107 0.8977
19 Nov 2017 08:09:29 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 219,179 196,581 0.8969
19 Nov 2017 05:07:36 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 207,659 186,022 0.8958
14 Nov 2017 04:47:31 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 196,139 175,806 0.8963
13 Nov 2017 07:19:42 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 184,619 165,396 0.8959
12 Nov 2017 06:50:39 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 173,099 154,714 0.8938
11 Nov 2017 07:49:12 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 161,579 144,232 0.8926
10 Nov 2017 09:50:16 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 127,019 115,368 0.9083
10 Nov 2017 06:18:51 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 115,499 105,029 0.9093
10 Nov 2017 02:03:04 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 103,979 94,578 0.9096
09 Nov 2017 11:30:10 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 92,459 84,212 0.9108
09 Nov 2017 08:26:35 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 80,939 73,955 0.9137
09 Nov 2017 04:50:23 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 69,419 63,659 0.9170
09 Nov 2017 01:33:52 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 57,899 53,258 0.9198
09 Nov 2017 01:33:52 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 46,379 42,365 0.9135
07 Nov 2017 09:10:56 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 34,859 31,520 0.9042
07 Nov 2017 06:17:08 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 23,339 20,962 0.8982
07 Nov 2017 03:08:01 1436093 20774998 wah2_pnw25_rdxx_200012_24_664_011274184_0 11,819 10,560 0.8935


©2024 climateprediction.net