climateprediction.net home page
Task 21373606

Task 21373606

Name wah2_global_e1gw_208812_145_767_011658776_1
Workunit 11658776
Created 9 Nov 2018, 22:48:33 UTC
Sent 10 Nov 2018, 20:04:57 UTC
Report deadline 24 Oct 2019, 1:24:57 UTC
Received 6 Jan 2019, 18:19:48 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1449156
Run time 33 days 18 hours 15 min 18 sec
CPU time 6 days 8 hours 35 min 36 sec
Validate state Invalid
Credit 8,933.67
Device peak FLOPS 1.69 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 238.99 MB
Peak swap size 175.40 MB
Peak disk usage 1,617.17 MB
Stderr 
<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
finish file present too long</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14356, selfPID=13240, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14356, selfPID=14356, iMonCtr=2
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=21416, iMonCtr=2
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
06:09:06 (21416): called boinc_finish(0)
CPDN Monitor - Quit request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=13456, selfPID=9116, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2856, selfPID=13180, iMonCtr=1
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3328, iMonCtr=2
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
CPDN Monitor - Quit request from BOINC...
Regional Worker:: CPDN process is not running,Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14148, selfPID=4144, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
00:41:31 (57964): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Cegional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2868, selfPID=2868, iMonCtr=2
PDN Monitor - Quit request from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13104, iMonCtr=2
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
09:40:57 (13104): called boinc_finish(0)
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10748, selfPID=13344, iMonCtr=1
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11272, iMonCtr=2
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12572, iMonCtr=2
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=34640, selfPID=15916, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9536, selfPID=9536, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9536, selfPID=8948, iMonCtr=1
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
08:10:49 (8948): called boinc_finish(0)
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12088, selfPID=12088, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12088, selfPID=3996, iMonCtr=1
Model crash detected, will try to restart...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12260, iMonCtr=2
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
12:09:52 (12260): called boinc_finish(0)
CPDN Monitor - Abort request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x027B112C

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 01/06/19 12:15:47
Install Directory : 
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 C:\ProgramData\BOINC\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000077b90000 0000000000190000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.17134.471) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.17134.228 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.228

ModLoad: 0000000075800000 00000000000e0000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.17134.376 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.376

ModLoad: 0000000074470000 00000000001e4000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.17134.441) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.17134.376 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.376

ModLoad: 00000000778b0000 000000000018d000 C:\WINDOWS\System32\USER32.dll (6.2.17134.376) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 00000000746f0000 0000000000017000 C:\WINDOWS\System32\win32u.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wwin32u.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000077080000 0000000000022000 C:\WINDOWS\System32\GDI32.dll (6.2.17134.285) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.17134.285 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.285

ModLoad: 00000000776e0000 0000000000164000 C:\WINDOWS\System32\gdi32full.dll (6.2.17134.471) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32full.pdb
    File Version          : 10.0.17134.471 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.471

ModLoad: 0000000074a80000 000000000007d000 C:\WINDOWS\System32\msvcp_win.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 00000000775c0000 000000000011e000 C:\WINDOWS\System32\ucrtbase.dll (6.2.17134.319) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.17134.319 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.319

ModLoad: 00000000774f0000 0000000000078000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.17134.471) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 00000000770b0000 00000000000bf000 C:\WINDOWS\System32\msvcrt.dll (7.0.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.17134.1

ModLoad: 00000000757b0000 0000000000044000 C:\WINDOWS\System32\sechost.dll (6.2.17134.319) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000074cc0000 00000000000c0000 C:\WINDOWS\System32\RPCRT4.dll (6.2.17134.471) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000074450000 0000000000020000 C:\WINDOWS\System32\SspiCli.dll (6.2.17134.376) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.17134.376 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.376

ModLoad: 0000000074440000 000000000000a000 C:\WINDOWS\System32\CRYPTBASE.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000077a40000 0000000000058000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.17134.471) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.17134.471 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.471

ModLoad: 0000000075770000 0000000000026000 C:\WINDOWS\System32\IMM32.DLL (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 000000006cf80000 0000000000182000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000074220000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 999, Write: 0, Other 8004

- I/O Transfers Counters -
Read: 0, Write: 2606, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 92232, QuotaPeakPagedPoolUsage: 92392
QuotaNonPagedPoolUsage: 9848, QuotaPeakNonPagedPoolUsage: 10136

- Virtual Memory Usage -
VirtualSize: 57393152, PeakVirtualSize: 57393152

- Pagefile Usage -
PagefileUsage: 3944448, PeakPagefileUsage: 3944448

- Working Set Size -
WorkingSetSize: 6438912, PeakWorkingSetSize: 6438912, PageFaultCount: 1870

*** Dump of thread ID 8392 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 4687500.000000, User Time: 7500000.000000, Wait Time: 15049.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x027B112C

- Registers -
eax=00000065 ebx=004b63b9 ecx=0019dc88 edx=027b112c esi=0019da40 edi=0019dcb8
eip=0048ee46 esp=0019d9d4 ebp=0019dc64
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

- Callstack -
ChildEBP RetAddr  Args to Child
0019dc64 0048161c 0019dc88 004b63b8 00000000 0019dcb8 wah2_8.24_windows_intelx86!+0x0 
0019dca8 0044fa52 027b112c 004b63b8 004b6484 00000421 wah2_8.24_windows_intelx86!+0x0 
0019dcc8 00496e90 0000000b 00000000 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0019dce0 00485ca1 c0000005 0019dd14 00490c54 00000000 wah2_8.24_windows_intelx86!+0x0 
0019ff80 75818484 00238000 75818460 6cdf601b 0019ffdc wah2_8.24_windows_intelx86!+0x0 
0019ff94 77bf3ab8 00238000 1205ec24 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 77bf3a88 ffffffff 77c0f304 00000000 00000000 ntdll!RtlAreBitsSet+0x0 
0019ffec 00000000 00485cce 00238000 00000000 00000000 ntdll!RtlAreBitsSet+0x0 

*** Dump of thread ID 3208 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 15007.000000

- Registers -
eax=00000000 ebx=00715288 ecx=00000000 edx=00000000 esi=007150c8 edi=00713948
eip=77bfc58c esp=00affdc8 ebp=00afff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
00afff80 75818484 00713948 75818460 6c69601b 00afffdc ntdll!NtWaitForWorkViaWorkerFactory+0x0 
00afff94 77bf3ab8 00713948 12b3ec24 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00afffdc 77bf3a88 ffffffff 77c0f304 00000000 00000000 ntdll!RtlAreBitsSet+0x0 
00afffec 00000000 77be1440 00713948 00000000 00000000 ntdll!RtlAreBitsSet+0x0 

*** Dump of thread ID 11672 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 15007.000000

- Registers -
eax=00000000 ebx=00718ce8 ecx=00000000 edx=00000000 esi=00718b28 edi=00713948
eip=77bfc58c esp=00bffdc8 ebp=00bfff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
00bfff80 75818484 00713948 75818460 6c79601b 00bfffdc ntdll!NtWaitForWorkViaWorkerFactory+0x0 
00bfff94 77bf3ab8 00713948 12a3ec24 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00bfffdc 77bf3a88 ffffffff 77c0f304 00000000 00000000 ntdll!RtlAreBitsSet+0x0 
00bfffec 00000000 77be1440 00713948 00000000 00000000 ntdll!RtlAreBitsSet+0x0 

*** Dump of thread ID 11028 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 15007.000000

- Registers -
eax=77be1440 ebx=00000000 ecx=00000000 edx=00000000 esi=0071f0b0 edi=00713948
eip=77bfc58c esp=00cffdc8 ebp=00cfff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
00cfff80 75818484 00713948 75818460 6c09601b 00cfffdc ntdll!NtWaitForWorkViaWorkerFactory+0x0 
00cfff94 77bf3ab8 00713948 12d3ec24 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00cfffdc 77bf3a88 ffffffff 77c0f304 00000000 00000000 ntdll!RtlAreBitsSet+0x0 
00cfffec 00000000 77be1440 00713948 00000000 00000000 ntdll!RtlAreBitsSet+0x0 

*** Dump of thread ID 1552 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 15046.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=00000001
eip=77bfac1c esp=026efcdc ebp=026efd40
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
026efd40 745770ef 00000064 00000000 026eff80 00478b44 ntdll!NtDelayExecution+0x0 
026efd50 00478b44 00000064 00478b20 00478b20 00000000 KERNELBASE!Sleep+0x0 
026eff80 75818484 00000000 75818460 6ea8601b 026effdc wah2_8.24_windows_intelx86!+0x0 
026eff94 77bf3ab8 00000000 1072ec24 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
026effdc 77bf3a88 ffffffff 77c0f304 00000000 00000000 ntdll!RtlAreBitsSet+0x0 
026effec 00000000 00478b20 00000000 00000000 00000000 ntdll!RtlAreBitsSet+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
28 Dec 2018 09:06:45 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 135,427 791,528 5.8447
28 Dec 2018 00:53:47 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 132,547 775,014 5.8471
27 Dec 2018 19:45:48 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 129,667 758,792 5.8519
27 Dec 2018 10:53:05 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 126,787 741,780 5.8506
27 Dec 2018 05:41:05 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 123,907 725,428 5.8546
27 Dec 2018 00:38:38 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 121,027 709,231 5.8601
26 Dec 2018 18:06:07 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 118,147 692,742 5.8634
26 Dec 2018 13:03:24 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 115,267 676,428 5.8684
26 Dec 2018 06:17:20 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 112,387 659,738 5.8702
26 Dec 2018 00:23:33 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 109,507 643,380 5.8752
25 Dec 2018 17:05:29 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 106,627 626,973 5.8801
25 Dec 2018 10:49:22 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 103,747 610,550 5.8850
25 Dec 2018 05:35:15 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 100,867 594,451 5.8934
21 Dec 2018 06:04:22 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 97,987 577,882 5.8975
21 Dec 2018 00:16:36 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 95,107 561,053 5.8992
20 Dec 2018 16:13:11 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 92,227 544,192 5.9006
20 Dec 2018 11:14:34 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 89,347 528,151 5.9112
20 Dec 2018 04:20:12 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 86,467 511,781 5.9188
19 Dec 2018 22:29:28 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 83,587 495,042 5.9225
02 Dec 2018 12:23:04 1449156 21373606 wah2_global_e1gw_208812_145_767_011658776_1 80,707 486,956 6.0336


©2024 climateprediction.net