climateprediction.net home page
Task 21373875

Task 21373875

Name wah2_global_e03r_208812_145_767_011657007_2
Workunit 11657007
Created 10 Nov 2018, 11:40:15 UTC
Sent 10 Nov 2018, 11:59:36 UTC
Report deadline 23 Oct 2019, 17:19:36 UTC
Received 15 Feb 2019, 19:13:55 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1170173
Run time 2 days 11 hours 35 min 21 sec
CPU time 2 days 2 hours 39 min 46 sec
Validate state Invalid
Credit 4,944.01
Device peak FLOPS 4.20 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 245.63 MB
Peak swap size 174.14 MB
Peak disk usage 100.80 MB
Stderr 
<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
finish file present too long</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
No Process Handle
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1144, selfPID=1144, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
No Process Handle
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=16292, selfPID=16292, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
No Process Handle
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=16004, selfPID=16352, iMonCtr=1
No Process Handle
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=16004, selfPID=16004, iMonCtr=1
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8880, selfPID=15748, iMonCtr=1
Model crash detected, will try to restart...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14484, selfPID=15164, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14484, selfPID=14484, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16
BUFFIN: C I/O Error feof - Unit 62 - Return code = 16
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16
BUFFIN: C I/O Error feof - Unit 62 - Return code = 16
Suspended CPDN Monitor - Suspend request from BOINC...
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16
BUFFIN: C I/O Error feof - Unit 62 - Return code = 16
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7844, selfPID=15748, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2720, selfPID=17184, iMonCtr=1
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
19:18:09 (17184): called boinc_finish(0)
CPDN Monitor - Abort request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0254F12C

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 02/14/19 17:59:11
Install Directory : C:\Program Files (x86)\BOINC\
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\Program Files (x86)\BOINC\\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\Program Files (x86)\BOINC\\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files (x86)\BOINC\\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 C:\ProgramData\BOINC\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000077140000 000000000019c000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.17763.292) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.17763.289 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.17763.289

ModLoad: 0000000076d80000 00000000000e0000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.17763.292) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.17763.289 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.17763.289

ModLoad: 00000000748d0000 00000000001fa000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.17763.292) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.17763.289 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.17763.289

ModLoad: 0000000075030000 0000000000199000 C:\WINDOWS\System32\USER32.dll (6.2.17763.168) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.17763.289 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.17763.289

ModLoad: 00000000761b0000 0000000000017000 C:\WINDOWS\System32\win32u.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wwin32u.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000076ec0000 0000000000023000 C:\WINDOWS\System32\GDI32.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000075730000 0000000000167000 C:\WINDOWS\System32\gdi32full.dll (6.2.17763.316) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32full.pdb
    File Version          : 10.0.17763.316 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.316

ModLoad: 00000000747d0000 0000000000080000 C:\WINDOWS\System32\msvcp_win.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000076a30000 0000000000123000 C:\WINDOWS\System32\ucrtbase.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 00000000770a0000 000000000007e000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.17763.289 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.17763.289

ModLoad: 0000000076b60000 00000000000c0000 C:\WINDOWS\System32\msvcrt.dll (7.0.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.17763.1

ModLoad: 00000000755f0000 0000000000079000 C:\WINDOWS\System32\sechost.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000075670000 00000000000bf000 C:\WINDOWS\System32\RPCRT4.dll (6.2.17763.194) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.17763.289 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.17763.289

ModLoad: 00000000747b0000 0000000000020000 C:\WINDOWS\System32\SspiCli.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 00000000747a0000 000000000000a000 C:\WINDOWS\System32\CRYPTBASE.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000075a40000 0000000000062000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000075ab0000 0000000000025000 C:\WINDOWS\System32\IMM32.DLL (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 000000006cbe0000 000000000018f000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1

ModLoad: 0000000074530000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.17763.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.17763.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17763.1



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 3731, Write: 0, Other 42131

- I/O Transfers Counters -
Read: 0, Write: 2286, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 93112, QuotaPeakPagedPoolUsage: 93272
QuotaNonPagedPoolUsage: 8760, QuotaPeakNonPagedPoolUsage: 9184

- Virtual Memory Usage -
VirtualSize: 54759424, PeakVirtualSize: 54759424

- Pagefile Usage -
PagefileUsage: 3874816, PeakPagefileUsage: 3874816

- Working Set Size -
WorkingSetSize: 6664192, PeakWorkingSetSize: 6668288, PageFaultCount: 1957

*** Dump of thread ID 16680 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 7031250.000000, User Time: 13125000.000000, Wait Time: 5206.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0254F12C

- Registers -
eax=00000065 ebx=004b63b9 ecx=0019dc00 edx=0254f12c esi=0019d9b8 edi=0019dc30
eip=0048ee46 esp=0019d94c ebp=0019dbdc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

- Callstack -
ChildEBP RetAddr  Args to Child
0019dbdc 0048161c 0019dc00 004b63b8 00000000 0019dc30 wah2_8.24_windows_intelx86!+0x0 
0019dc20 0044fa52 0254f12c 004b63b8 004b6484 00000421 wah2_8.24_windows_intelx86!+0x0 
0019dc40 00496e90 0000000b 00000000 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0019dc58 00485ca1 c0000005 0019dc8c 00490c54 00000000 wah2_8.24_windows_intelx86!+0x0 
0019ff70 76da0179 0025c000 76da0160 0019ffdc 771a662d wah2_8.24_windows_intelx86!+0x0 
0019ff80 771a662d 0025c000 aa656eb0 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 771a65fd ffffffff 771c51aa 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0019ffec 00000000 00485cce 0025c000 00000000 78746341 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 17188 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 3990.000000

- Registers -
eax=00000000 ebx=0084c458 ecx=00000000 edx=00000000 esi=0084c298 edi=00845050
eip=771b216c esp=0081fdb4 ebp=0081ff70
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
0081ff70 76da0179 00845050 76da0160 0081ffdc 771a662d ntdll!NtWaitForWorkViaWorkerFactory+0x0 
0081ff80 771a662d 00845050 aafd6eb0 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0081ffdc 771a65fd ffffffff 771c51aa 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0081ffec 00000000 7718e230 00845050 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 12996 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 3990.000000

- Registers -
eax=7718e230 ebx=00000000 ecx=00000000 edx=00000000 esi=0084ccd8 edi=00845050
eip=771b216c esp=00a3fdb4 ebp=00a3ff70
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
00a3ff70 76da0179 00845050 76da0160 00a3ffdc 771a662d ntdll!NtWaitForWorkViaWorkerFactory+0x0 
00a3ff80 771a662d 00845050 aadf6eb0 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00a3ffdc 771a65fd ffffffff 771c51aa 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
00a3ffec 00000000 7718e230 00845050 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 12936 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 5205.000000

- Registers -
eax=00000000 ebx=0247fd0c ecx=00000000 edx=00000000 esi=00000000 edi=0247fd0c
eip=771b07ec esp=0247fccc ebp=0247fd30
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
0247fd30 749c8f8f 00000064 00000000 0247ff70 00478b44 ntdll!ZwDelayExecution+0x0 
0247fd40 00478b44 00000064 00478b20 00478b20 00000000 KERNELBASE!Sleep+0x0 
0247ff70 76da0179 00000000 76da0160 0247ffdc 771a662d wah2_8.24_windows_intelx86!+0x0 
0247ff80 771a662d 00000000 a83b6eb0 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0247ffdc 771a65fd ffffffff 771c51aa 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0247ffec 00000000 00478b20 00000000 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
12 Feb 2019 18:06:40 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 74,947 181,781 2.4255
12 Jan 2019 16:40:34 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 72,067 173,524 2.4078
04 Jan 2019 12:48:20 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 69,187 164,717 2.3808
04 Jan 2019 09:53:09 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 66,307 155,360 2.3430
03 Jan 2019 10:02:33 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 63,427 146,675 2.3125
31 Dec 2018 15:43:17 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 60,547 138,365 2.2852
30 Dec 2018 19:11:48 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 57,667 130,826 2.2686
28 Dec 2018 18:08:39 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 54,787 123,563 2.2553
27 Dec 2018 18:53:44 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 51,907 116,352 2.2415
26 Dec 2018 15:03:50 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 49,027 109,117 2.2257
26 Dec 2018 13:56:21 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 46,147 102,228 2.2153
26 Dec 2018 10:48:27 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 43,267 95,473 2.2066
22 Dec 2018 17:13:32 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 40,387 89,097 2.2061
21 Dec 2018 16:19:04 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 37,507 82,871 2.2095
16 Dec 2018 08:07:41 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 34,627 76,687 2.2147
15 Dec 2018 15:13:19 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 31,747 69,840 2.1999
14 Dec 2018 15:52:34 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 28,867 63,094 2.1857
14 Dec 2018 13:42:02 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 25,987 56,259 2.1649
14 Dec 2018 11:51:17 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 23,107 49,240 2.1310
12 Dec 2018 16:04:14 1170173 21373875 wah2_global_e03r_208812_145_767_011657007_2 20,227 42,918 2.1218


©2024 climateprediction.net