climateprediction.net home page
Task 21353755

Task 21353755

Name wah2_global_e10v_200412_145_766_011655699_0
Workunit 11655699
Created 4 Nov 2018, 20:05:45 UTC
Sent 5 Nov 2018, 3:40:45 UTC
Report deadline 18 Oct 2019, 9:00:45 UTC
Received 14 Nov 2018, 10:23:44 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1448744
Run time 7 days 17 hours 25 min 21 sec
CPU time 6 days 11 hours 32 min 54 sec
Validate state Invalid
Credit 12,163.40
Device peak FLOPS 3.88 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 238.51 MB
Peak swap size 174.46 MB
Peak disk usage 152.36 MB
Stderr 
<core_client_version>7.12.1</core_client_version>
<![CDATA[
<message>
finish file present too long</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14224, selfPID=64556, iMonCtr=1
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
02:11:07 (64556): called boinc_finish(0)
CPDN Monitor - Abort request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0262012C

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 11/14/18 02:52:22
Install Directory : 
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 C:\ProgramData\BOINC\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000077330000 0000000000190000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.17134.376) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.17134.228 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.228

ModLoad: 0000000076d60000 00000000000e0000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.17134.376 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.376

ModLoad: 0000000074e90000 00000000001e4000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.17134.407) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.17134.376 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.376

ModLoad: 0000000076f10000 000000000018d000 C:\WINDOWS\System32\USER32.dll (6.2.17134.376) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000077220000 0000000000017000 C:\WINDOWS\System32\win32u.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wwin32u.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000074a90000 0000000000022000 C:\WINDOWS\System32\GDI32.dll (6.2.17134.285) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.17134.285 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.285

ModLoad: 00000000766e0000 0000000000164000 C:\WINDOWS\System32\gdi32full.dll (6.2.17134.345) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32full.pdb
    File Version          : 10.0.17134.345 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.345

ModLoad: 00000000770a0000 000000000007d000 C:\WINDOWS\System32\msvcp_win.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000074ac0000 000000000011e000 C:\WINDOWS\System32\ucrtbase.dll (6.2.17134.319) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.17134.319 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.319

ModLoad: 0000000074bf0000 0000000000078000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.17134.319) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000074d20000 00000000000bf000 C:\WINDOWS\System32\msvcrt.dll (7.0.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.17134.1

ModLoad: 0000000076910000 0000000000044000 C:\WINDOWS\System32\sechost.dll (6.2.17134.319) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000076470000 00000000000c0000 C:\WINDOWS\System32\RPCRT4.dll (6.2.17134.407) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000073bf0000 0000000000020000 C:\WINDOWS\System32\SspiCli.dll (6.2.17134.376) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.17134.376 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.376

ModLoad: 0000000073be0000 000000000000a000 C:\WINDOWS\System32\CRYPTBASE.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000073c10000 0000000000058000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.17134.345) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.17134.345 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.345

ModLoad: 0000000076850000 0000000000026000 C:\WINDOWS\System32\IMM32.DLL (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 000000006b020000 0000000000182000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000073b60000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 2906, Write: 0, Other 36892

- I/O Transfers Counters -
Read: 0, Write: 1672, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 91944, QuotaPeakPagedPoolUsage: 92104
QuotaNonPagedPoolUsage: 8760, QuotaPeakNonPagedPoolUsage: 9184

- Virtual Memory Usage -
VirtualSize: 56143872, PeakVirtualSize: 56143872

- Pagefile Usage -
PagefileUsage: 3567616, PeakPagefileUsage: 3567616

- Working Set Size -
WorkingSetSize: 5984256, PeakWorkingSetSize: 5984256, PageFaultCount: 1745

*** Dump of thread ID 15316 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 10468750.000000, User Time: 10156250.000000, Wait Time: 147885.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0262012C

- Registers -
eax=00000065 ebx=004b63b9 ecx=0019dc88 edx=0262012c esi=0019da40 edi=0019dcb8
eip=0048ee46 esp=0019d9d4 ebp=0019dc64
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

- Callstack -
ChildEBP RetAddr  Args to Child
0019dc64 0048161c 0019dc88 004b63b8 00000000 0019dcb8 wah2_8.24_windows_intelx86!+0x0 
0019dca8 0044fa52 0262012c 004b63b8 004b6484 00000421 wah2_8.24_windows_intelx86!+0x0 
0019dcc8 00496e90 0000000b 00000000 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0019dce0 00485ca1 c0000005 0019dd14 00490c54 00000000 wah2_8.24_windows_intelx86!+0x0 
0019ff80 76d78484 0030a000 76d78460 102d263d 0019ffdc wah2_8.24_windows_intelx86!+0x0 
0019ff94 7739302c 0030a000 6f267e17 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 77392ffa ffffffff 773aec7d 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 
0019ffec 00000000 00485cce 0030a000 00000000 78746341 ntdll!RtlValidSecurityDescriptor+0x0 

*** Dump of thread ID 17516 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 146710.000000

- Registers -
eax=00000000 ebx=005e51f0 ecx=00000000 edx=00000000 esi=005e5030 edi=005e3d88
eip=7739beec esp=008ffdc8 ebp=008fff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
008fff80 76d78484 005e3d88 76d78460 10bb263d 008fffdc ntdll!NtWaitForWorkViaWorkerFactory+0x0 
008fff94 7739302c 005e3d88 6fb07e17 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
008fffdc 77392ffa ffffffff 773aec7d 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 
008fffec 00000000 7737c900 005e3d88 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 

*** Dump of thread ID 14296 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 146710.000000

- Registers -
eax=00000000 ebx=005e5c60 ecx=00000000 edx=00000000 esi=005e5aa0 edi=005e3d88
eip=7739beec esp=009ffdc8 ebp=009fff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
009fff80 76d78484 005e3d88 76d78460 10ab263d 009fffdc ntdll!NtWaitForWorkViaWorkerFactory+0x0 
009fff94 7739302c 005e3d88 6fa07e17 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
009fffdc 77392ffa ffffffff 773aec7d 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 
009fffec 00000000 7737c900 005e3d88 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 

*** Dump of thread ID 14952 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 146710.000000

- Registers -
eax=00000000 ebx=005e6068 ecx=00000000 edx=00000000 esi=005e5ea8 edi=005e3d88
eip=7739beec esp=00affdc8 ebp=00afff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
00afff80 76d78484 005e3d88 76d78460 109b263d 00afffdc ntdll!NtWaitForWorkViaWorkerFactory+0x0 
00afff94 7739302c 005e3d88 6f907e17 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00afffdc 77392ffa ffffffff 773aec7d 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 
00afffec 00000000 7737c900 005e3d88 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 

*** Dump of thread ID 1472 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 147880.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=00000001
eip=7739a57c esp=0255fcdc ebp=0255fd40
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0255fd40 74f970ef 00000064 00000000 0255ff80 00478b44 ntdll!NtDelayExecution+0x0 
0255fd50 00478b44 00000064 00478b20 00478b20 00000000 KERNELBASE!Sleep+0x0 
0255ff80 76d78484 00000000 76d78460 1261263d 0255ffdc wah2_8.24_windows_intelx86!+0x0 
0255ff94 7739302c 00000000 6d6a7e17 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0255ffdc 77392ffa ffffffff 773aec7d 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 
0255ffec 00000000 00478b20 00000000 00000000 00000000 ntdll!RtlValidSecurityDescriptor+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
14 Nov 2018 07:27:55 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 184,387 554,240 3.0059
14 Nov 2018 01:19:30 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 181,507 545,496 3.0054
13 Nov 2018 22:15:19 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 178,627 536,803 3.0052
13 Nov 2018 19:15:21 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 175,747 528,015 3.0044
13 Nov 2018 17:09:49 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 172,867 518,856 3.0015
13 Nov 2018 13:44:42 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 169,987 509,893 2.9996
13 Nov 2018 10:40:00 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 167,107 501,097 2.9987
13 Nov 2018 07:29:55 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 164,227 492,164 2.9969
13 Nov 2018 02:21:14 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 161,347 483,547 2.9969
12 Nov 2018 22:17:46 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 158,467 475,102 2.9981
12 Nov 2018 19:18:18 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 155,587 466,626 2.9991
12 Nov 2018 16:08:41 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 152,707 457,970 2.9990
12 Nov 2018 13:03:28 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 149,827 449,556 3.0005
12 Nov 2018 12:04:53 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 146,947 440,911 3.0005
12 Nov 2018 12:04:53 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 144,067 432,445 3.0017
12 Nov 2018 12:04:53 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 141,187 423,984 3.0030
11 Nov 2018 22:50:39 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 138,307 415,404 3.0035
11 Nov 2018 19:44:27 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 135,427 406,933 3.0048
11 Nov 2018 14:52:00 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 132,547 398,386 3.0056
11 Nov 2018 11:46:53 1448744 21353755 wah2_global_e10v_200412_145_766_011655699_0 129,667 389,630 3.0049


©2024 climateprediction.net