climateprediction.net home page
Task 20892420

Task 20892420

Name wah2_pnw25_s271_200312_24_683_011355576_0
Workunit 11355576
Created 17 Nov 2017, 17:44:58 UTC
Sent 27 Nov 2017, 20:00:17 UTC
Report deadline 10 Nov 2018, 1:20:17 UTC
Received 2 Dec 2017, 3:23:16 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1439952
Run time 3 days 15 hours 43 min 59 sec
CPU time 3 days 13 hours 17 min 15 sec
Validate state Invalid
Credit 16,738.33
Device peak FLOPS 3.51 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 187.90 MB
Peak swap size 165.96 MB
Peak disk usage 47.89 MB
Stderr
<core_client_version>7.8.3</core_client_version>
<![CDATA[
<message>
finish file present too long</message>
<stderr_txt>
06:01:37 (9808): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
08:21:03 (9388): start_timer_thread(): CreateThread() failed, errno 0
08:21:05 (8460): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
09:47:48 (1904): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
09:54:26 (7652): start_timer_thread(): CreateThread() failed, errno 0
09:54:28 (11716): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
12:17:13 (7956): start_timer_thread(): CreateThread() failed, errno 0
12:17:15 (11828): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
13:14:18 (6840): start_timer_thread(): CreateThread() failed, errno 0
13:14:19 (5088): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
15:36:53 (3044): start_timer_thread(): CreateThread() failed, errno 0
15:36:55 (12576): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
16:36:45 (7556): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
17:17:32 (6580): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
17:47:08 (12404): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
09:10:44 (2212): start_timer_thread(): CreateThread() failed, errno 0
09:10:46 (13144): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
10:11:14 (2060): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
10:51:56 (10688): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
11:36:11 (8280): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
12:03:13 (7132): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
12:12:18 (8696): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
12:31:34 (8548): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
17:46:27 (13116): start_timer_thread(): CreateThread() failed, errno 0
08:59:40 (12400): Can't acquire lockfile (32) - waiting 35s
09:00:15 (12400): Can't acquire lockfile (32) - exiting
09:00:15 (12400): Error: The process cannot access the file because it is being used by another process.

 (0x20)
09:10:17 (17196): Can't acquire lockfile (32) - waiting 35s
09:10:52 (17196): Can't acquire lockfile (32) - exiting
09:10:52 (17196): Error: The process cannot access the file because it is being used by another process.

 (0x20)
09:20:54 (14508): Can't acquire lockfile (32) - waiting 35s
09:21:29 (14508): Can't acquire lockfile (32) - exiting
09:21:29 (14508): Error: The process cannot access the file because it is being used by another process.

 (0x20)
09:31:50 (16236): Can't acquire lockfile (32) - waiting 35s
09:32:25 (16236): Can't acquire lockfile (32) - exiting
09:32:25 (16236): Error: The process cannot access the file because it is being used by another process.

 (0x20)
09:42:28 (3604): Can't acquire lockfile (32) - waiting 35s
09:43:03 (3604): Can't acquire lockfile (32) - exiting
09:43:03 (3604): Error: The process cannot access the file because it is being used by another process.

 (0x20)
09:53:04 (13320): Can't acquire lockfile (32) - waiting 35s
09:53:39 (13320): Can't acquire lockfile (32) - exiting
09:53:39 (13320): Error: The process cannot access the file because it is being used by another process.

 (0x20)
10:04:29 (7844): Can't acquire lockfile (32) - waiting 35s
10:05:04 (7844): Can't acquire lockfile (32) - exiting
10:05:04 (7844): Error: The process cannot access the file because it is being used by another process.

 (0x20)
10:15:07 (12280): Can't acquire lockfile (32) - waiting 35s
10:15:42 (12280): Can't acquire lockfile (32) - exiting
10:15:42 (12280): Error: The process cannot access the file because it is being used by another process.

 (0x20)
10:25:44 (7948): Can't acquire lockfile (32) - waiting 35s
10:26:19 (7948): Can't acquire lockfile (32) - exiting
10:26:19 (7948): Error: The process cannot access the file because it is being used by another process.

 (0x20)
10:36:20 (12640): Can't acquire lockfile (32) - waiting 35s
10:36:55 (12640): Can't acquire lockfile (32) - exiting
10:36:55 (12640): Error: The process cannot access the file because it is being used by another process.

 (0x20)
10:46:57 (2232): Can't acquire lockfile (32) - waiting 35s
10:47:32 (2232): Can't acquire lockfile (32) - exiting
10:47:32 (2232): Error: The process cannot access the file because it is being used by another process.

 (0x20)
10:57:33 (636): Can't acquire lockfile (32) - waiting 35s
10:58:08 (636): Can't acquire lockfile (32) - exiting
10:58:08 (636): Error: The process cannot access the file because it is being used by another process.

 (0x20)
11:08:10 (9276): Can't acquire lockfile (32) - waiting 35s
11:08:45 (9276): Can't acquire lockfile (32) - exiting
11:08:45 (9276): Error: The process cannot access the file because it is being used by another process.

 (0x20)
11:18:46 (12272): Can't acquire lockfile (32) - waiting 35s
11:19:21 (12272): Can't acquire lockfile (32) - exiting
11:19:21 (12272): Error: The process cannot access the file because it is being used by another process.

 (0x20)
11:29:22 (5064): Can't acquire lockfile (32) - waiting 35s
11:29:57 (5064): Can't acquire lockfile (32) - exiting
11:29:57 (5064): Error: The process cannot access the file because it is being used by another process.

 (0x20)
11:40:00 (9484): Can't acquire lockfile (32) - waiting 35s
11:40:35 (9484): Can't acquire lockfile (32) - exiting
11:40:35 (9484): Error: The process cannot access the file because it is being used by another process.

 (0x20)
11:50:36 (2620): Can't acquire lockfile (32) - waiting 35s
11:51:11 (2620): Can't acquire lockfile (32) - exiting
11:51:11 (2620): Error: The process cannot access the file because it is being used by another process.

 (0x20)
12:01:12 (7548): Can't acquire lockfile (32) - waiting 35s
12:01:47 (7548): Can't acquire lockfile (32) - exiting
12:01:47 (7548): Error: The process cannot access the file because it is being used by another process.

 (0x20)
12:11:49 (16544): Can't acquire lockfile (32) - waiting 35s
12:12:24 (16544): Can't acquire lockfile (32) - exiting
12:12:24 (16544): Error: The process cannot access the file because it is being used by another process.

 (0x20)
12:22:26 (16068): Can't acquire lockfile (32) - waiting 35s
12:23:01 (16068): Can't acquire lockfile (32) - exiting
12:23:01 (16068): Error: The process cannot access the file because it is being used by another process.

 (0x20)
12:33:03 (10100): Can't acquire lockfile (32) - waiting 35s
12:33:38 (10100): Can't acquire lockfile (32) - exiting
12:33:38 (10100): Error: The process cannot access the file because it is being used by another process.

 (0x20)
12:43:41 (15308): Can't acquire lockfile (32) - waiting 35s
12:44:16 (15308): Can't acquire lockfile (32) - exiting
12:44:16 (15308): Error: The process cannot access the file because it is being used by another process.

 (0x20)
12:54:17 (5876): Can't acquire lockfile (32) - waiting 35s
12:54:52 (5876): Can't acquire lockfile (32) - exiting
12:54:52 (5876): Error: The process cannot access the file because it is being used by another process.

 (0x20)
13:04:54 (15156): Can't acquire lockfile (32) - waiting 35s
Leaving CPDN_ain::Monitor...
13:05:04 (11152): called boinc_finish(0)
CPDN Monitor - Abort request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x02A4612C

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 12/02/17 13:05:30
Install Directory : 
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 C:\ProgramData\BOINC\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000077bc0000 000000000018d000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.16299.64) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 00000000771a0000 00000000000d0000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 0000000074ac0000 00000000001d7000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 00000000773b0000 0000000000175000 C:\WINDOWS\System32\USER32.dll (6.2.16299.64) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 0000000077a10000 0000000000016000 C:\WINDOWS\System32\win32u.dll (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : wwin32u.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 0000000076590000 0000000000022000 C:\WINDOWS\System32\GDI32.dll (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 0000000074880000 000000000015f000 C:\WINDOWS\System32\gdi32full.dll (6.2.16299.19) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32full.pdb
    File Version          : 10.0.16299.19 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.19

ModLoad: 00000000749e0000 000000000007c000 C:\WINDOWS\System32\msvcp_win.dll (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 0000000077290000 0000000000117000 C:\WINDOWS\System32\ucrtbase.dll (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 0000000077530000 0000000000078000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 0000000076790000 00000000000bd000 C:\WINDOWS\System32\msvcrt.dll (7.0.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.16299.15

ModLoad: 0000000076fb0000 0000000000043000 C:\WINDOWS\System32\sechost.dll (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 0000000074680000 00000000000be000 C:\WINDOWS\System32\RPCRT4.dll (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 00000000745e0000 0000000000020000 C:\WINDOWS\System32\SspiCli.dll (6.2.16299.19) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.16299.19 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.19

ModLoad: 00000000745d0000 000000000000a000 C:\WINDOWS\System32\CRYPTBASE.dll (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 0000000074740000 0000000000057000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 00000000775b0000 0000000000025000 C:\WINDOWS\System32\IMM32.DLL (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 0000000076040000 0000000000246000 C:\WINDOWS\System32\combase.dll (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : combase.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 000000006eee0000 0000000000181000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15

ModLoad: 00000000740d0000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.16299.15) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.16299.15 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.16299.15



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 27, Write: 0, Other 351

- I/O Transfers Counters -
Read: 0, Write: 2185, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 96728, QuotaPeakPagedPoolUsage: 96824
QuotaNonPagedPoolUsage: 7536, QuotaPeakNonPagedPoolUsage: 7960

- Virtual Memory Usage -
VirtualSize: 86978560, PeakVirtualSize: 87973888

- Pagefile Usage -
PagefileUsage: 2867200, PeakPagefileUsage: 2867200

- Working Set Size -
WorkingSetSize: 6594560, PeakWorkingSetSize: 6594560, PageFaultCount: 1937

*** Dump of thread ID 15576 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 312500.000000, User Time: 0.000000, Wait Time: 28798044.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x02A4612C

- Registers -
eax=00000065 ebx=004b63b9 ecx=0019dc10 edx=02a4612c esi=0019d9c8 edi=0019dc40
eip=0048ee46 esp=0019d95c ebp=0019dbec
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

- Callstack -
ChildEBP RetAddr  Args to Child
0019dbec 0048161c 0019dc10 004b63b8 00000000 0019dc40 wah2_8.24_windows_intelx86!+0x0 
0019dc30 0044fa52 02a4612c 004b63b8 004b6484 00000421 wah2_8.24_windows_intelx86!+0x0 
0019dc50 00496e90 0000000b 00000000 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0019dc68 00485ca1 c0000005 0019dc9c 00490c54 00000000 wah2_8.24_windows_intelx86!+0x0 
0019ff80 771b8654 00290000 771b8630 cc8e028f 0019ffdc wah2_8.24_windows_intelx86!+0x0 
0019ff94 77c24a47 00290000 be21ecd6 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 77c24a17 ffffffff 77c49eb5 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0019ffec 00000000 00485cce 00290000 00000000 78746341 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 13320 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 28798040.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=00000001
eip=77c2eaac esp=0297fcdc ebp=0297fd40
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0297fd40 74bc285f 00000064 00000000 0297ff80 00478b44 ntdll!ZwDelayExecution+0x0 
0297fd50 00478b44 00000064 00478b20 00478b20 00000000 KERNELBASE!Sleep+0x0 
0297ff80 771b8654 00000000 771b8630 ce00028f 0297ffdc wah2_8.24_windows_intelx86!+0x0 
0297ff94 77c24a47 00000000 bcafecd6 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0297ffdc 77c24a17 ffffffff 77c49eb5 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0297ffec 00000000 00478b20 00000000 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
01 Dec 2017 20:34:56 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 253,739 298,130 1.1749
01 Dec 2017 16:33:57 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 242,219 284,294 1.1737
01 Dec 2017 12:30:11 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 230,699 271,341 1.1762
01 Dec 2017 10:31:40 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 219,179 261,018 1.1909
01 Dec 2017 07:30:41 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 207,659 250,912 1.2083
01 Dec 2017 04:25:21 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 196,139 240,752 1.2275
01 Dec 2017 01:22:08 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 184,619 229,853 1.2450
30 Nov 2017 22:14:14 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 173,099 217,594 1.2570
30 Nov 2017 18:11:48 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 161,579 205,179 1.2698
30 Nov 2017 15:06:23 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 150,059 192,750 1.2845
30 Nov 2017 11:03:43 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 138,539 180,036 1.2995
30 Nov 2017 06:46:58 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 127,019 165,010 1.2991
29 Nov 2017 23:12:39 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 115,499 150,546 1.3034
29 Nov 2017 18:28:12 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 103,979 135,921 1.3072
29 Nov 2017 14:15:30 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 92,459 121,308 1.3120
29 Nov 2017 10:37:04 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 80,939 106,724 1.3186
29 Nov 2017 01:01:29 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 69,419 91,894 1.3238
28 Nov 2017 20:57:40 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 57,899 76,788 1.3262
28 Nov 2017 15:56:33 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 46,379 61,240 1.3204
28 Nov 2017 11:55:06 1439952 20892420 wah2_pnw25_s271_200312_24_683_011355576_0 34,859 45,674 1.3102


©2024 climateprediction.net