climateprediction.net home page
Task 21426366

Task 21426366

Name wah2_global_30ij_208812_145_773_011675582_1
Workunit 11675582
Created 15 Dec 2018, 8:56:30 UTC
Sent 18 Dec 2018, 13:25:43 UTC
Report deadline 30 Nov 2019, 18:45:43 UTC
Received 17 Jan 2019, 4:50:24 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1444073
Run time 1 days 19 hours 53 min 47 sec
CPU time 1 days 19 hours 0 min 22 sec
Validate state Invalid
Credit 3,804.10
Device peak FLOPS 3.48 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 238.32 MB
Peak swap size 175.63 MB
Peak disk usage 1,423.84 MB
Stderr
<core_client_version>7.8.2</core_client_version>
<![CDATA[
<message>
finish file present too long
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
23:52:39 (23808): Can't acquire lockfile (32) - waiting 35s
23:53:14 (23808): Can't acquire lockfile (32) - exiting
23:53:14 (23808): Error: The process cannot access the file because it is being used by another process.

 (0x20)
00:03:34 (28552): Can't acquire lockfile (32) - waiting 35s
00:04:09 (28552): Can't acquire lockfile (32) - exiting
00:04:09 (28552): Error: The process cannot access the file because it is being used by another process.

 (0x20)
00:14:17 (27384): Can't acquire lockfile (32) - waiting 35s
00:14:52 (27384): Can't acquire lockfile (32) - exiting
00:14:52 (27384): Error: The process cannot access the file because it is being used by another process.

 (0x20)
00:24:59 (24872): Can't acquire lockfile (32) - waiting 35s
00:25:34 (24872): Can't acquire lockfile (32) - exiting
00:25:34 (24872): Error: The process cannot access the file because it is being used by another process.

 (0x20)
00:36:09 (26916): Can't acquire lockfile (32) - waiting 35s
00:36:44 (26916): Can't acquire lockfile (32) - exiting
00:36:44 (26916): Error: The process cannot access the file because it is being used by another process.

 (0x20)
00:47:27 (24952): Can't acquire lockfile (32) - waiting 35s
00:48:02 (24952): Can't acquire lockfile (32) - exiting
00:48:02 (24952): Error: The process cannot access the file because it is being used by another process.

 (0x20)
00:58:06 (27332): Can't acquire lockfile (32) - waiting 35s
00:58:41 (27332): Can't acquire lockfile (32) - exiting
00:58:41 (27332): Error: The process cannot access the file because it is being used by another process.

 (0x20)
01:09:17 (27248): Can't acquire lockfile (32) - waiting 35s
01:09:52 (27248): Can't acquire lockfile (32) - exiting
01:09:52 (27248): Error: The process cannot access the file because it is being used by another process.

 (0x20)
01:20:26 (27380): Can't acquire lockfile (32) - waiting 35s
01:21:01 (27380): Can't acquire lockfile (32) - exiting
01:21:01 (27380): Error: The process cannot access the file because it is being used by another process.

 (0x20)
01:31:33 (27044): Can't acquire lockfile (32) - waiting 35s
01:32:08 (27044): Can't acquire lockfile (32) - exiting
01:32:08 (27044): Error: The process cannot access the file because it is being used by another process.

 (0x20)
01:42:38 (26396): Can't acquire lockfile (32) - waiting 35s
01:43:13 (26396): Can't acquire lockfile (32) - exiting
01:43:13 (26396): Error: The process cannot access the file because it is being used by another process.

 (0x20)
01:53:44 (28048): Can't acquire lockfile (32) - waiting 35s
01:54:19 (28048): Can't acquire lockfile (32) - exiting
01:54:19 (28048): Error: The process cannot access the file because it is being used by another process.

 (0x20)
02:04:52 (28092): Can't acquire lockfile (32) - waiting 35s
02:05:27 (28092): Can't acquire lockfile (32) - exiting
02:05:27 (28092): Error: The process cannot access the file because it is being used by another process.

 (0x20)
02:15:56 (25812): Can't acquire lockfile (32) - waiting 35s
02:16:31 (25812): Can't acquire lockfile (32) - exiting
02:16:31 (25812): Error: The process cannot access the file because it is being used by another process.

 (0x20)
02:26:45 (26820): Can't acquire lockfile (32) - waiting 35s
02:27:20 (26820): Can't acquire lockfile (32) - exiting
02:27:20 (26820): Error: The process cannot access the file because it is being used by another process.

 (0x20)
02:38:06 (27012): Can't acquire lockfile (32) - waiting 35s
02:38:41 (27012): Can't acquire lockfile (32) - exiting
02:38:41 (27012): Error: The process cannot access the file because it is being used by another process.

 (0x20)
02:49:11 (26668): Can't acquire lockfile (32) - waiting 35s
02:49:46 (26668): Can't acquire lockfile (32) - exiting
02:49:46 (26668): Error: The process cannot access the file because it is being used by another process.

 (0x20)
03:00:15 (25116): Can't acquire lockfile (32) - waiting 35s
03:00:50 (25116): Can't acquire lockfile (32) - exiting
03:00:50 (25116): Error: The process cannot access the file because it is being used by another process.

 (0x20)
03:11:22 (27184): Can't acquire lockfile (32) - waiting 35s
03:11:57 (27184): Can't acquire lockfile (32) - exiting
03:11:57 (27184): Error: The process cannot access the file because it is being used by another process.

 (0x20)
03:22:27 (28596): Can't acquire lockfile (32) - waiting 35s
03:23:02 (28596): Can't acquire lockfile (32) - exiting
03:23:02 (28596): Error: The process cannot access the file because it is being used by another process.

 (0x20)
03:33:31 (27868): Can't acquire lockfile (32) - waiting 35s
03:34:06 (27868): Can't acquire lockfile (32) - exiting
03:34:06 (27868): Error: The process cannot access the file because it is being used by another process.

 (0x20)
03:44:36 (26820): Can't acquire lockfile (32) - waiting 35s
03:45:11 (26820): Can't acquire lockfile (32) - exiting
03:45:11 (26820): Error: The process cannot access the file because it is being used by another process.

 (0x20)
03:55:43 (24688): Can't acquire lockfile (32) - waiting 35s
03:56:18 (24688): Can't acquire lockfile (32) - exiting
03:56:18 (24688): Error: The process cannot access the file because it is being used by another process.

 (0x20)
04:06:47 (28572): Can't acquire lockfile (32) - waiting 35s
04:07:22 (28572): Can't acquire lockfile (32) - exiting
04:07:22 (28572): Error: The process cannot access the file because it is being used by another process.

 (0x20)
04:17:54 (26784): Can't acquire lockfile (32) - waiting 35s
04:18:29 (26784): Can't acquire lockfile (32) - exiting
04:18:29 (26784): Error: The process cannot access the file because it is being used by another process.

 (0x20)
04:28:58 (28516): Can't acquire lockfile (32) - waiting 35s
04:29:33 (28516): Can't acquire lockfile (32) - exiting
04:29:33 (28516): Error: The process cannot access the file because it is being used by another process.

 (0x20)
04:40:04 (27272): Can't acquire lockfile (32) - waiting 35s
04:40:39 (27272): Can't acquire lockfile (32) - exiting
04:40:39 (27272): Error: The process cannot access the file because it is being used by another process.

 (0x20)
04:51:10 (26836): Can't acquire lockfile (32) - waiting 35s
04:51:45 (26836): Can't acquire lockfile (32) - exiting
04:51:45 (26836): Error: The process cannot access the file because it is being used by another process.

 (0x20)
05:02:15 (25064): Can't acquire lockfile (32) - waiting 35s
05:02:50 (25064): Can't acquire lockfile (32) - exiting
05:02:50 (25064): Error: The process cannot access the file because it is being used by another process.

 (0x20)
05:13:20 (27380): Can't acquire lockfile (32) - waiting 35s
05:13:55 (27380): Can't acquire lockfile (32) - exiting
05:13:55 (27380): Error: The process cannot access the file because it is being used by another process.

 (0x20)
05:24:26 (27548): Can't acquire lockfile (32) - waiting 35s
05:25:01 (27548): Can't acquire lockfile (32) - exiting
05:25:01 (27548): Error: The process cannot access the file because it is being used by another process.

 (0x20)
05:35:28 (27812): Can't acquire lockfile (32) - waiting 35s
05:36:03 (27812): Can't acquire lockfile (32) - exiting
05:36:03 (27812): Error: The process cannot access the file because it is being used by another process.

 (0x20)
05:46:35 (28364): Can't acquire lockfile (32) - waiting 35s
05:47:10 (28364): Can't acquire lockfile (32) - exiting
05:47:10 (28364): Error: The process cannot access the file because it is being used by another process.

 (0x20)
05:57:39 (28312): Can't acquire lockfile (32) - waiting 35s
05:58:14 (28312): Can't acquire lockfile (32) - exiting
05:58:14 (28312): Error: The process cannot access the file because it is being used by another process.

 (0x20)
06:08:44 (24924): Can't acquire lockfile (32) - waiting 35s
06:09:19 (24924): Can't acquire lockfile (32) - exiting
06:09:19 (24924): Error: The process cannot access the file because it is being used by another process.

 (0x20)
06:19:47 (28516): Can't acquire lockfile (32) - waiting 35s
06:20:22 (28516): Can't acquire lockfile (32) - exiting
06:20:22 (28516): Error: The process cannot access the file because it is being used by another process.

 (0x20)
06:30:34 (29968): Can't acquire lockfile (32) - waiting 35s
06:31:09 (29968): Can't acquire lockfile (32) - exiting
06:31:09 (29968): Error: The process cannot access the file because it is being used by another process.

 (0x20)
06:41:17 (30480): Can't acquire lockfile (32) - waiting 35s
06:41:52 (30480): Can't acquire lockfile (32) - exiting
06:41:52 (30480): Error: The process cannot access the file because it is being used by another process.

 (0x20)
06:49:30 (30432): Can't acquire lockfile (32) - waiting 35s
06:50:06 (30432): Can't acquire lockfile (32) - exiting
06:50:06 (30432): Error: The process cannot access the file because it is being used by another process.

 (0x20)
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=22948, selfPID=22948, iMonCtr=2
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=22948, selfPID=25352, iMonCtr=1
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
06:59:10 (25352): called boinc_finish(0)
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Abort request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0040A58A write attempt to address 0x00000130

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 01/17/19 11:47:41
Install Directory : 
Data Directory    : D:\ProgramFilesFromC\BOINC ProgramData
Project Symstore  : 
LoadLibraryA( D:\ProgramFilesFromC\BOINC ProgramData\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( D:\ProgramFilesFromC\BOINC ProgramData\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\ProgramFilesFromC\BOINC ProgramData\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\ProgramFilesFromC\BOINC ProgramData\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\ProgramFilesFromC\BOINC ProgramData\projects\climateprediction.net;D:\ProgramFilesFromC\BOINC ProgramData\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 D:\ProgramFilesFromC\BOINC ProgramData\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000077530000 0000000000180000 C:\Windows\SysWOW64\ntdll.dll (6.1.7601.23915) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 0000000076070000 0000000000110000 C:\Windows\syswow64\kernel32.dll (6.1.7601.23915) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 6.1.7601.17965 (win7sp1_gdr.121004-0333)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17965

ModLoad: 0000000075540000 0000000000047000 C:\Windows\syswow64\KERNELBASE.dll (6.1.7601.23915) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 6.1.7601.17965 (win7sp1_gdr.121004-0333)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17965

ModLoad: 0000000075610000 0000000000100000 C:\Windows\syswow64\USER32.dll (6.1.7601.23594) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 0000000076180000 0000000000090000 C:\Windows\syswow64\GDI32.dll (6.1.7601.23914) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 6.1.7601.23914 (win7sp1_ldr.170908-0600)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.23914

ModLoad: 0000000075260000 000000000000a000 C:\Windows\syswow64\LPK.dll (6.1.7601.23930) (-exported- Symbols Loaded)
    Linked PDB Filename   : wlpk.pdb
    File Version          : 6.1.7601.23930 (win7sp1_ldr.171011-1526)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.23930

ModLoad: 0000000076230000 000000000009d000 C:\Windows\syswow64\USP10.dll (1.626.7601.23894) (-exported- Symbols Loaded)
    Linked PDB Filename   : usp10.pdb
    File Version          : 1.0626.7601.23894 (win7sp1_ldr.170816-0600)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Uniscribe Unicode script processor
    Product Version       : 1.0626.7601.23894

ModLoad: 0000000077070000 00000000000ac000 C:\Windows\syswow64\msvcrt.dll (7.0.7601.17744) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.7601.17744 (win7sp1_gdr.111215-1535)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.7601.17744

ModLoad: 0000000075710000 00000000000a1000 C:\Windows\syswow64\ADVAPI32.dll (6.1.7601.23915) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 6.1.7601.23915 (win7sp1_ldr.170913-0600)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.23915

ModLoad: 0000000075200000 0000000000019000 C:\Windows\SysWOW64\sechost.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 0000000076330000 00000000000f0000 C:\Windows\syswow64\RPCRT4.dll (6.1.7601.23915) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 0000000075060000 0000000000060000 C:\Windows\syswow64\SspiCli.dll (6.1.7601.23915) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 6.1.7601.23915 (win7sp1_ldr.170913-0600)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.23915

ModLoad: 0000000075050000 000000000000c000 C:\Windows\syswow64\CRYPTBASE.dll (6.1.7601.23915) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 6.1.7601.23915 (win7sp1_ldr.170913-0600)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.23915

ModLoad: 00000000757c0000 0000000000060000 C:\Windows\system32\IMM32.DLL (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 0000000075270000 00000000000cd000 C:\Windows\syswow64\MSCTF.dll (6.1.7601.23915) (-exported- Symbols Loaded)
    Linked PDB Filename   : msctf.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 0000000072e00000 00000000000eb000 C:\Windows\system32\dbghelp.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 0000000072f30000 0000000000009000 C:\Windows\system32\version.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 2012, Write: 0, Other 32706

- I/O Transfers Counters -
Read: 0, Write: 169, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 88456, QuotaPeakPagedPoolUsage: 88456
QuotaNonPagedPoolUsage: 6720, QuotaPeakNonPagedPoolUsage: 6912

- Virtual Memory Usage -
VirtualSize: 50315264, PeakVirtualSize: 50315264

- Pagefile Usage -
PagefileUsage: 2166784, PeakPagefileUsage: 2166784

- Working Set Size -
WorkingSetSize: 4349952, PeakWorkingSetSize: 4349952, PageFaultCount: 1128

*** Dump of thread ID 4084 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 3588023.000000, User Time: 7800050.000000, Wait Time: 25487.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0040A58A write attempt to address 0x00000130

- Registers -
eax=00000000 ebx=00000000 ecx=0000000b edx=0040a560 esi=0000000b edi=00000000
eip=0040a58a esp=0018dcc4 ebp=0018dce8
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010297

- Callstack -
ChildEBP RetAddr  Args to Child
0018dce8 00496e90 0000000b 00000000 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0018dd00 00485ca1 c0000005 0018dd34 00490c54 00000000 wah2_8.24_windows_intelx86!+0x0 
0018ff88 7608336a 7efde000 0018ffd4 775698f2 7efde000 wah2_8.24_windows_intelx86!+0x0 
0018ff94 775698f2 7efde000 777e0e2d 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
0018ffd4 775698c5 00485cce 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
0018ffec 00000000 00485cce 7efde000 00000000 78746341 ntdll!RtlInitializeExceptionChain+0x0 

*** Dump of thread ID 4080 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 25481.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=0241fd24 edi=00000000
eip=7754fda1 esp=0241fce0 ebp=0241fd48
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
0241fd48 7555460c 00000064 00000000 0241ff88 00478b44 ntdll!NtDelayExecution+0x0 
0241fd58 00478b44 00000064 00000000 00000000 00000000 KERNELBASE!Sleep+0x0 
0241ff88 7608336a 00000000 0241ffd4 775698f2 00000000 wah2_8.24_windows_intelx86!+0x0 
0241ff94 775698f2 00000000 75270e2d 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
0241ffd4 775698c5 00478b20 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
0241ffec 00000000 00478b20 00000000 00000000 0b186d70 ntdll!RtlInitializeExceptionChain+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
20 Dec 2018 16:56:43 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 57,667 153,670 2.6648
20 Dec 2018 16:08:03 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 54,787 145,674 2.6589
20 Dec 2018 11:56:05 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 51,907 137,709 2.6530
20 Dec 2018 09:44:43 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 49,027 130,106 2.6538
20 Dec 2018 07:33:19 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 46,147 122,460 2.6537
20 Dec 2018 06:15:19 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 43,267 114,525 2.6469
20 Dec 2018 01:43:34 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 40,387 106,762 2.6435
19 Dec 2018 23:37:30 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 37,507 99,142 2.6433
19 Dec 2018 21:27:31 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 34,627 91,600 2.6453
19 Dec 2018 19:22:01 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 31,747 84,032 2.6469
19 Dec 2018 17:12:12 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 28,867 76,425 2.6475
19 Dec 2018 15:16:43 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 25,987 68,796 2.6473
19 Dec 2018 14:14:10 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 23,107 61,154 2.6466
19 Dec 2018 10:50:12 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 20,227 53,538 2.6469
19 Dec 2018 08:39:57 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 17,347 45,844 2.6428
19 Dec 2018 06:29:05 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 14,467 38,147 2.6368
19 Dec 2018 05:10:09 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 11,587 30,433 2.6265
19 Dec 2018 03:08:21 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 8,707 22,799 2.6185
18 Dec 2018 23:57:55 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 5,827 15,278 2.6219
18 Dec 2018 21:52:49 1444073 21426366 wah2_global_30ij_208812_145_773_011675582_1 2,947 7,731 2.6233


©2024 cpdn.org