climateprediction.net home page
Task 20902097

Task 20902097

Name wah2_pnw25_s2jx_200312_24_683_011356040_1
Workunit 11356040
Created 21 Nov 2017, 22:39:14 UTC
Sent 21 Nov 2017, 22:51:28 UTC
Report deadline 4 Nov 2018, 4:11:28 UTC
Received 26 Nov 2017, 8:26:24 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1401333
Run time 3 days 18 hours 39 min 31 sec
CPU time 3 days 18 hours 22 min 3 sec
Validate state Invalid
Credit 14,458.52
Device peak FLOPS 4.17 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 186.60 MB
Peak swap size 165.86 MB
Peak disk usage 1.62 MB
Stderr 
<core_client_version>7.8.3</core_client_version>
<![CDATA[
<message>
finish file present too long</message>
<stderr_txt>
14:51:43 (8612): start_timer_thread(): CreateThread() failed, errno 0
14:51:45 (8572): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
11:04:56 (9196): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
11:55:03 (1360): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
12:30:25 (6360): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
12:35:38 (7348): start_timer_thread(): CreateThread() failed, errno 0
12:35:39 (5008): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
13:19:07 (6252): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
13:24:22 (9956): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
15:25:24 (3752): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
15:30:35 (5420): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
21:47:18 (4048): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
21:52:31 (4900): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
11:52:23 (2020): start_timer_thread(): CreateThread() failed, errno 0
Global Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=2020, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=2
CPDN Monitor - Quit request from BOINC...
11:57:35 (7872): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
12:37:04 (7316): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
12:42:17 (6128): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
13:56:08 (1036): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
14:01:21 (2644): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
01:21:35 (9432): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
15:54:28 (10184): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
17:39:02 (6920): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
17:44:15 (7200): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
18:28:28 (9876): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
18:40:53 (8124): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
18:46:09 (3584): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
19:14:10 (5156): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
20:05:42 (6284): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
20:20:04 (4904): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
20:25:21 (10016): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
22:16:07 (5304): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
00:05:59 (7908): start_timer_thread(): CreateThread() failed, errno 0
Global Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=1
00:06:07 (6444): start_timer_thread(): CreateThread() failed, errno 0
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6444, selfPID=7896, iMonCtr=1
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
00:06:10 (7896): called boinc_finish(0)
CPDN Monitor - Abort request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0273A12C

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 11/26/17 00:11:22
Install Directory : C:\Program Files (x86)\BOINC\
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\Program Files (x86)\BOINC\\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\Program Files (x86)\BOINC\\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files (x86)\BOINC\\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 C:\ProgramData\BOINC\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000077a00000 000000000018e000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.15063.608) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.608

ModLoad: 0000000077120000 00000000000d0000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.15063.296) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.608

ModLoad: 00000000771f0000 00000000001c2000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.15063.726) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.608

ModLoad: 0000000076cd0000 000000000013c000 C:\WINDOWS\System32\USER32.dll (6.2.15063.726) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000077810000 0000000000016000 C:\WINDOWS\System32\win32u.dll (6.2.15063.608) (-exported- Symbols Loaded)
    Linked PDB Filename   : wwin32u.pdb
    File Version          : 10.0.15063.608 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.608

ModLoad: 0000000076e20000 0000000000021000 C:\WINDOWS\System32\GDI32.dll (6.2.15063.483) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.15063.483 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.483

ModLoad: 0000000076a80000 0000000000158000 C:\WINDOWS\System32\gdi32full.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32full.pdb
    File Version          : 10.0.15063.674 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.674

ModLoad: 00000000776e0000 0000000000079000 C:\WINDOWS\System32\msvcp_win.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 00000000763e0000 0000000000118000 C:\WINDOWS\System32\ucrtbase.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.15063.674 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.674

ModLoad: 0000000077830000 0000000000077000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 00000000778b0000 00000000000bd000 C:\WINDOWS\System32\msvcrt.dll (7.0.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.15063.0

ModLoad: 00000000773c0000 0000000000041000 C:\WINDOWS\System32\sechost.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000075f70000 00000000000c0000 C:\WINDOWS\System32\RPCRT4.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 00000000744d0000 0000000000020000 C:\WINDOWS\System32\SspiCli.dll (6.2.15063.674) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.15063.674 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.674

ModLoad: 00000000744c0000 000000000000a000 C:\WINDOWS\System32\CRYPTBASE.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 0000000076270000 0000000000057000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.15063.726) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.15063.726 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.726

ModLoad: 0000000077760000 0000000000025000 C:\WINDOWS\System32\IMM32.DLL (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0

ModLoad: 00000000686a0000 000000000015d000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.15063.447) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.15063.447 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.447

ModLoad: 00000000743a0000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.15063.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.15063.0 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.15063.0



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 12067, Write: 0, Other 97098

- I/O Transfers Counters -
Read: 0, Write: 2290, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 90512, QuotaPeakPagedPoolUsage: 90640
QuotaNonPagedPoolUsage: 8488, QuotaPeakNonPagedPoolUsage: 9048

- Virtual Memory Usage -
VirtualSize: 54501376, PeakVirtualSize: 54501376

- Pagefile Usage -
PagefileUsage: 2789376, PeakPagefileUsage: 2826240

- Working Set Size -
WorkingSetSize: 6316032, PeakWorkingSetSize: 6316032, PageFaultCount: 1841

*** Dump of thread ID 4452 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 12031250.000000, User Time: 37031248.000000, Wait Time: 24333392.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0273A12C

- Registers -
eax=00000065 ebx=004b63b9 ecx=0019d8e8 edx=0273a12c esi=0019d6a0 edi=0019d918
eip=0048ee46 esp=0019d634 ebp=0019d8c4
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

- Callstack -
ChildEBP RetAddr  Args to Child
0019d8c4 0048161c 0019d8e8 004b63b8 00000000 0019d918 wah2_8.24_windows_intelx86!+0x0 
0019d908 0044fa52 0273a12c 004b63b8 004b6484 00000421 wah2_8.24_windows_intelx86!+0x0 
0019d928 00496e90 0000000b 00000000 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0019d940 00485ca1 c0000005 0019d974 00490c54 00000000 wah2_8.24_windows_intelx86!+0x0 
0019ff80 77138744 00262000 77138720 b37c2861 0019ffdc wah2_8.24_windows_intelx86!+0x0 
0019ff94 77a6582d 00262000 fc36e5bd 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 77a657fd ffffffff 77a86375 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0019ffec 00000000 00485cce 00262000 00000000 78746341 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 9192 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 24332652.000000

- Registers -
eax=00000000 ebx=00854290 ecx=00000000 edx=00000000 esi=00854758 edi=00854918
eip=77a7396c esp=0083fd7c ebp=0083ff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0083ff80 77138744 00854290 77138720 b3e62861 0083ffdc ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
0083ff94 77a6582d 00854290 fcace5bd 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0083ffdc 77a657fd ffffffff 77a86375 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0083ffec 00000000 77a3a470 00854290 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 5096 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 24332652.000000

- Registers -
eax=00000000 ebx=00854290 ecx=00000000 edx=00000000 esi=008552d0 edi=00855490
eip=77a7396c esp=00a4fd7c ebp=00a4ff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
00a4ff80 77138744 00854290 77138720 b3c12861 00a4ffdc ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
00a4ff94 77a6582d 00854290 fc8be5bd 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00a4ffdc 77a657fd ffffffff 77a86375 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
00a4ffec 00000000 77a3a470 00854290 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 4804 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 24332652.000000

- Registers -
eax=00000000 ebx=00854290 ecx=00000000 edx=00000000 esi=008587f0 edi=008589b0
eip=77a7396c esp=00b4fd7c ebp=00b4ff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
00b4ff80 77138744 00854290 77138720 b3d12861 00b4ffdc ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
00b4ff94 77a6582d 00854290 fc9be5bd 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00b4ffdc 77a657fd ffffffff 77a86375 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
00b4ffec 00000000 77a3a470 00854290 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 828 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 24333386.000000

- Registers -
eax=00000000 ebx=0266fd1c ecx=00000000 edx=00000000 esi=00000000 edi=00000001
eip=77a7205c esp=0266fcdc ebp=0266fd40
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0266fd40 772e48cf 00000064 00000000 0266ff80 00478b44 ntdll!NtDelayExecution+0x0 
0266fd50 00478b44 00000064 00478b20 00478b20 00000000 KERNELBASE!Sleep+0x0 
0266ff80 77138744 00000000 77138720 b1032861 0266ffdc wah2_8.24_windows_intelx86!+0x0 
0266ff94 77a6582d 00000000 fe49e5bd 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0266ffdc 77a657fd ffffffff 77a86375 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0266ffec 00000000 00478b20 00000000 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
26 Nov 2017 05:14:40 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 219,179 322,410 1.4710
25 Nov 2017 19:53:49 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 207,659 305,402 1.4707
25 Nov 2017 15:11:05 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 196,139 288,436 1.4706
25 Nov 2017 10:24:01 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 184,619 271,530 1.4708
25 Nov 2017 04:43:18 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 173,099 254,427 1.4698
24 Nov 2017 23:59:27 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 161,579 237,382 1.4691
24 Nov 2017 16:09:09 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 150,059 220,158 1.4671
24 Nov 2017 11:30:21 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 138,539 203,356 1.4679
24 Nov 2017 06:48:06 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 127,019 186,497 1.4683
24 Nov 2017 02:05:53 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 115,499 169,667 1.4690
23 Nov 2017 21:26:39 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 103,979 152,798 1.4695
23 Nov 2017 16:42:37 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 92,459 135,863 1.4694
23 Nov 2017 12:00:44 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 80,939 119,044 1.4708
23 Nov 2017 07:17:03 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 69,419 102,148 1.4715
23 Nov 2017 01:54:56 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 57,899 85,144 1.4706
22 Nov 2017 17:49:01 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 46,379 68,009 1.4664
22 Nov 2017 13:07:34 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 34,859 51,143 1.4671
22 Nov 2017 09:27:25 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 23,339 34,247 1.4674
22 Nov 2017 04:26:16 1401333 20902097 wah2_pnw25_s2jx_200312_24_683_011356040_1 11,819 17,381 1.4706


©2024 cpdn.org