climateprediction.net home page
Task 21866052

Task 21866052

Name wah2_sam50_a0ot_201312_28_858_011968218_1
Workunit 11968218
Created 25 Dec 2019, 7:37:08 UTC
Sent 25 Dec 2019, 7:37:33 UTC
Report deadline 6 Dec 2020, 12:57:33 UTC
Received 31 Dec 2019, 3:03:23 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1495347
Run time 4 days 2 hours 57 min 37 sec
CPU time 4 days 2 hours 36 min 25 sec
Validate state Invalid
Credit 19,778.07
Device peak FLOPS 5.44 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 225.30 MB
Peak swap size 188.22 MB
Peak disk usage 140.67 MB
Stderr 
<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
finish file present too long</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16224, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=676, selfPID=16124, iMonCtr=1
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
02:07:30 (16124): called boinc_finish(0)
CPDN Monitor - Abort request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0268612C

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 12/30/19 12:08:30
Install Directory : 
Data Directory    : A:\BOINC
Project Symstore  : 
LoadLibraryA( A:\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( A:\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( A:\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( A:\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: A:\BOINC\projects\climateprediction.net;A:\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 A:\BOINC\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000076f40000 000000000019a000 C:\Windows\SYSTEM32\ntdll.dll (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.18362.329 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.329

ModLoad: 00000000756e0000 00000000000e0000 C:\Windows\System32\KERNEL32.DLL (6.2.18362.329) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.18362.329 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.329

ModLoad: 0000000076960000 00000000001fc000 C:\Windows\System32\KERNELBASE.dll (6.2.18362.535) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.18362.329 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.329

ModLoad: 0000000076c90000 0000000000197000 C:\Windows\System32\USER32.dll (6.2.18362.535) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000074880000 0000000000017000 C:\Windows\System32\win32u.dll (6.2.18362.535) (-exported- Symbols Loaded)
    Linked PDB Filename   : wwin32u.pdb
    File Version          : 10.0.18362.535 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.535

ModLoad: 00000000762d0000 0000000000021000 C:\Windows\System32\GDI32.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000076300000 000000000015a000 C:\Windows\System32\gdi32full.dll (6.2.18362.535) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32full.pdb
    File Version          : 10.0.18362.535 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.535

ModLoad: 0000000075eb0000 000000000007c000 C:\Windows\System32\msvcp_win.dll (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.18362.387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.387

ModLoad: 0000000076060000 000000000011f000 C:\Windows\System32\ucrtbase.dll (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.18362.387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.387

ModLoad: 0000000074740000 0000000000079000 C:\Windows\System32\ADVAPI32.dll (6.2.18362.329) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000075de0000 00000000000bf000 C:\Windows\System32\msvcrt.dll (7.0.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.18362.1

ModLoad: 00000000757e0000 0000000000076000 C:\Windows\System32\sechost.dll (6.2.18362.267) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 00000000749f0000 00000000000bb000 C:\Windows\System32\RPCRT4.dll (6.2.18362.476) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000074710000 0000000000020000 C:\Windows\System32\SspiCli.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000074700000 000000000000a000 C:\Windows\System32\CRYPTBASE.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000076000000 000000000005f000 C:\Windows\System32\bcryptPrimitives.dll (6.2.18362.295) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.18362.295 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.295

ModLoad: 00000000766b0000 0000000000025000 C:\Windows\System32\IMM32.DLL (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 10.0.18362.387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.387

ModLoad: 000000006bf40000 000000000018f000 C:\Windows\SYSTEM32\dbghelp.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000073fb0000 0000000000008000 C:\Windows\SYSTEM32\version.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 4366, Write: 0, Other 95925

- I/O Transfers Counters -
Read: 0, Write: 1684, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 92824, QuotaPeakPagedPoolUsage: 92984
QuotaNonPagedPoolUsage: 9032, QuotaPeakNonPagedPoolUsage: 9320

- Virtual Memory Usage -
VirtualSize: 56029184, PeakVirtualSize: 56029184

- Pagefile Usage -
PagefileUsage: 3862528, PeakPagefileUsage: 3862528

- Working Set Size -
WorkingSetSize: 6574080, PeakWorkingSetSize: 6578176, PageFaultCount: 1959

*** Dump of thread ID 15536 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 6718750.000000, User Time: 12500000.000000, Wait Time: 3703338.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0268612C

- Registers -
eax=00000065 ebx=004b63b9 ecx=0019dc80 edx=0268612c esi=0019da38 edi=0019dcb0
eip=0048ee46 esp=0019d9cc ebp=0019dc5c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

- Callstack -
ChildEBP RetAddr  Args to Child
0019dc5c 0048161c 0019dc80 004b63b8 00000000 0019dcb0 wah2_8.24_windows_intelx86!+0x0 
0019dca0 0044fa52 0268612c 004b63b8 004b6484 00000421 wah2_8.24_windows_intelx86!+0x0 
0019dcc0 00496e90 0000000b 00000000 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0019dcd8 00485ca1 c0000005 0019dd0c 00490c54 00000000 wah2_8.24_windows_intelx86!+0x0 
0019ff70 756f6359 003d2000 756f6340 0019ffdc 76fa7b74 wah2_8.24_windows_intelx86!+0x0 
0019ff80 76fa7b74 003d2000 f54a0483 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 76fa7b44 ffffffff 76fc8f09 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0019ffec 00000000 00485cce 003d2000 00000000 78746341 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 6172 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 3701981.000000

- Registers -
eax=00000000 ebx=00614e48 ecx=00000000 edx=00000000 esi=00614e48 edi=006138f0
eip=76fb3a4c esp=008dfdb8 ebp=008dff70
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
008dff70 756f6359 006138f0 756f6340 008dffdc 76fa7b74 ntdll!NtWaitForWorkViaWorkerFactory+0x0 
008dff80 76fa7b74 006138f0 f5de0483 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
008dffdc 76fa7b44 ffffffff 76fc8f09 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
008dffec 00000000 76f963b0 006138f0 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 6412 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 3701981.000000

- Registers -
eax=00000000 ebx=00615a98 ecx=00000000 edx=00000000 esi=00615a98 edi=006138f0
eip=76fb3a4c esp=009dfdb8 ebp=009dff70
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
009dff70 756f6359 006138f0 756f6340 009dffdc 76fa7b74 ntdll!NtWaitForWorkViaWorkerFactory+0x0 
009dff80 76fa7b74 006138f0 f5ce0483 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
009dffdc 76fa7b44 ffffffff 76fc8f09 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
009dffec 00000000 76f963b0 006138f0 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 1804 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 3701981.000000

- Registers -
eax=00000000 ebx=00618d90 ecx=00000000 edx=00000000 esi=00618d90 edi=006138f0
eip=76fb3a4c esp=00b1fdb8 ebp=00b1ff70
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
00b1ff70 756f6359 006138f0 756f6340 00b1ffdc 76fa7b74 ntdll!NtWaitForWorkViaWorkerFactory+0x0 
00b1ff80 76fa7b74 006138f0 f5e20483 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00b1ffdc 76fa7b44 ffffffff 76fc8f09 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
00b1ffec 00000000 76f963b0 006138f0 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 2348 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 3703336.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=025bfd0c
eip=76fb20bc esp=025bfccc ebp=025bfd30
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
025bfd30 76a6e82f 00000064 00000000 025bff70 00478b44 ntdll!NtDelayExecution+0x0 
025bfd40 00478b44 00000064 00478b20 00478b20 00000000 KERNELBASE!Sleep+0x0 
025bff70 756f6359 00000000 756f6340 025bffdc 76fa7b74 wah2_8.24_windows_intelx86!+0x0 
025bff80 76fa7b74 00000000 f7080483 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
025bffdc 76fa7b44 ffffffff 76fc8f09 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
025bffec 00000000 00478b20 00000000 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
29 Dec 2019 23:35:49 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 299,819 344,624 1.1494
29 Dec 2019 20:03:28 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 288,299 331,936 1.1514
29 Dec 2019 16:30:49 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 276,779 319,126 1.1530
29 Dec 2019 12:58:09 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 265,259 306,414 1.1552
29 Dec 2019 09:10:11 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 253,739 293,436 1.1564
29 Dec 2019 01:08:18 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 242,219 280,342 1.1574
28 Dec 2019 20:02:27 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 230,699 267,189 1.1582
28 Dec 2019 16:19:25 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 219,179 253,848 1.1582
28 Dec 2019 12:36:37 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 207,659 240,518 1.1582
28 Dec 2019 08:48:36 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 196,139 227,036 1.1575
28 Dec 2019 05:00:38 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 184,619 213,702 1.1575
28 Dec 2019 01:17:22 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 173,099 200,371 1.1576
27 Dec 2019 19:14:38 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 161,579 187,130 1.1581
27 Dec 2019 15:05:48 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 150,059 174,043 1.1598
27 Dec 2019 11:27:52 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 138,539 160,934 1.1617
27 Dec 2019 07:45:04 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 127,019 147,615 1.1621
27 Dec 2019 01:37:03 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 115,499 134,641 1.1657
26 Dec 2019 21:54:13 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 103,979 121,180 1.1654
26 Dec 2019 18:06:10 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 92,459 107,728 1.1651
26 Dec 2019 14:23:13 1495347 21866052 wah2_sam50_a0ot_201312_28_858_011968218_1 80,939 94,295 1.1650


©2024 cpdn.org