climateprediction.net home page
Task 17839481

Task 17839481

Name hadam3pm2_k2gj_1959_10_009464414_2
Workunit 9546648
Created 28 Jan 2015, 17:51:18 UTC
Sent 28 Jan 2015, 19:26:20 UTC
Report deadline 11 Jan 2016, 0:46:20 UTC
Received 10 Feb 2015, 13:40:09 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1303626
Run time 12 days 17 hours 6 min 7 sec
CPU time 3 days 1 hours 43 min 39 sec
Validate state Invalid
Credit 9,482.20
Device peak FLOPS 2.40 GFLOPS
Application version UK Met Office HadAM3P (global only) with MOSES II landsurface scheme v7.03
i686-pc-linux-gnu
Stderr
<core_client_version>7.2.42</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
oa.pc|9dec.nc
oa.pc|0jan.nc
oa.pc|0feb.nc
oa.pc|0mar.nc
oa.pc|0apr.nc
oa.pc|0may.nc
oa.pc|0jun.nc
oa.pc|0jul.nc
oa.pc|0aug.nc
oa.pc|0sep.nc
oa.pc|0oct.nc
oa.pc|0nov.nc
oa.pe|9dec.nc
oa.pe|0jan.nc
oa.pe|0feb.nc
oa.pe|0mar.nc
oa.pe|0apr.nc
oa.pe|0may.nc
oa.pe|0jun.nc
oa.pe|0jul.nc
oa.pe|0aug.nc
oa.pe|0sep.nc
oa.pe|0oct.nc
oa.pe|0nov.nc
oa.pc|0dec.nc
oa.pc|1jan.nc
oa.pc|1feb.nc
oa.pc|1mar.nc
oa.pc|1apr.nc
oa.pc|1may.nc
oa.pc|1jun.nc
oa.pc|1jul.nc
oa.pc|1aug.nc
oa.pc|1sep.nc
oa.pc|1oct.nc
oa.pc|1nov.nc
oa.pe|0dec.nc
oa.pe|1jan.nc
oa.pe|1feb.nc
oa.pe|1mar.nc
oa.pe|1apr.nc
oa.pe|1may.nc
oa.pe|1jun.nc
oa.pe|1jul.nc
oa.pe|1aug.nc
oa.pe|1sep.nc
oa.pe|1oct.nc
oa.pe|1nov.nc
Suspended CPDN Monitor - Suspend request from BOINC...
oa.pc|1dec.nc
oa.pc|2jan.nc
oa.pc|2feb.nc
oa.pc|2mar.nc
oa.pc|2apr.nc
oa.pc|2may.nc
oa.pc|2jun.nc
oa.pc|2jul.nc
oa.pc|2aug.nc
oa.pc|2sep.nc
oa.pc|2oct.nc
oa.pc|2nov.nc
oa.pe|1dec.nc
oa.pe|2jan.nc
oa.pe|2feb.nc
oa.pe|2mar.nc
oa.pe|2apr.nc
oa.pe|2may.nc
oa.pe|2jun.nc
oa.pe|2jul.nc
oa.pe|2aug.nc
oa.pe|2sep.nc
oa.pe|2oct.nc
oa.pe|2nov.nc
oa.pc|2dec.nc
oa.pc|3jan.nc
oa.pc|3feb.nc
oa.pc|3mar.nc
oa.pc|3apr.nc
oa.pc|3may.nc
oa.pc|3jun.nc
oa.pc|3jul.nc
oa.pc|3aug.nc
oa.pc|3sep.nc
oa.pc|3oct.nc
oa.pc|3nov.nc
oa.pe|2dec.nc
oa.pe|3jan.nc
oa.pe|3feb.nc
oa.pe|3mar.nc
oa.pe|3apr.nc
oa.pe|3may.nc
oa.pe|3jun.nc
oa.pe|3jul.nc
oa.pe|3aug.nc
oa.pe|3sep.nc
oa.pe|3oct.nc
oa.pe|3nov.nc
oa.pc|3dec.nc
oa.pc|4jan.nc
oa.pc|4feb.nc
oa.pc|4mar.nc
oa.pc|4apr.nc
oa.pc|4may.nc
oa.pc|4jun.nc
oa.pc|4jul.nc
oa.pc|4aug.nc
oa.pc|4sep.nc
oa.pc|4oct.nc
oa.pc|4nov.nc
oa.pe|3dec.nc
oa.pe|4jan.nc
oa.pe|4feb.nc
oa.pe|4mar.nc
oa.pe|4apr.nc
oa.pe|4may.nc
oa.pe|4jun.nc
oa.pe|4jul.nc
oa.pe|4aug.nc
oa.pe|4sep.nc
oa.pe|4oct.nc
oa.pe|4nov.nc
oa.pc|4dec.nc
oa.pc|5jan.nc
oa.pc|5feb.nc
oa.pc|5mar.nc
oa.pc|5apr.nc
oa.pc|5may.nc
oa.pc|5jun.nc
oa.pc|5jul.nc
oa.pc|5aug.nc
oa.pc|5sep.nc
oa.pc|5oct.nc
oa.pc|5nov.nc
oa.pe|4dec.nc
oa.pe|5jan.nc
oa.pe|5feb.nc
oa.pe|5mar.nc
oa.pe|5apr.nc
oa.pe|5may.nc
oa.pe|5jun.nc
oa.pe|5jul.nc
oa.pe|5aug.nc
oa.pe|5sep.nc
oa.pe|5oct.nc
oa.pe|5nov.nc
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0fade50]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d]
/lib/libc.so.6(__libc_start_main+0xf3)[0xf0db1993]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13253, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f94e50]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d]
/lib/libc.so.6(__libc_start_main+0xf3)[0xf0d98993]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13253, iMonCtr=1
Model crash detected, will try to restart...
oa.pc|5dec.nc
oa.pc|6jan.nc
oa.pc|6feb.nc
oa.pc|6mar.nc
oa.pc|6apr.nc
oa.pc|6may.nc
oa.pc|6jun.nc
oa.pc|6jul.nc
oa.pc|6aug.nc
oa.pc|6sep.nc
oa.pc|6oct.nc
oa.pc|6nov.nc
oa.pe|5dec.nc
oa.pe|6jan.nc
oa.pe|6feb.nc
oa.pe|6mar.nc
oa.pe|6apr.nc
oa.pe|6may.nc
oa.pe|6jun.nc
oa.pe|6jul.nc
oa.pe|6aug.nc
oa.pe|6sep.nc
oa.pe|6oct.nc
oa.pe|6nov.nc
oa.pc|6dec.nc
oa.pc|7jan.nc
oa.pc|7feb.nc
oa.pc|7mar.nc
oa.pc|7apr.nc
oa.pc|7may.nc
oa.pc|7jun.nc
oa.pc|7jul.nc
oa.pc|7aug.nc
oa.pc|7sep.nc
oa.pc|7oct.nc
oa.pc|7nov.nc
oa.pe|6dec.nc
oa.pe|7jan.nc
oa.pe|7feb.nc
oa.pe|7mar.nc
oa.pe|7apr.nc
oa.pe|7may.nc
oa.pe|7jun.nc
oa.pe|7jul.nc
oa.pe|7aug.nc
oa.pe|7sep.nc
oa.pe|7oct.nc
oa.pe|7nov.nc
Signal 15 received, exiting...
Signal 1 received, exiting...
08:51:43 (10992): called boinc_finish
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f02e50]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d]
/lib/libc.so.6(__libc_start_main+0xf3)[0xf0d06993]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3133, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0fb5e50]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d]
/lib/libc.so.6(__libc_start_main+0xf3)[0xf0db9993]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3133, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f58e50]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d]
/lib/libc.so.6(__libc_start_main+0xf3)[0xf0d5c993]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3133, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f3ee50]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d]
/lib/libc.so.6(__libc_start_main+0xf3)[0xf0d42993]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3133, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f0fe50]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d]
/lib/libc.so.6(__libc_start_main+0xf3)[0xf0d13993]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3133, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation
Stack trace (13 frames):
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf]
linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f8be50]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9]
/home/martin/BOINC/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d]
/lib/libc.so.6(__libc_start_main+0xf3)[0xf0d8f993]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3133, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
13:39:31 (3133): called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
10 Feb 2015 12:17:52 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 328,388 363,849 1.1080
10 Feb 2015 09:42:10 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 325,508 354,651 1.0895
10 Feb 2015 06:56:24 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 322,628 344,836 1.0688
10 Feb 2015 04:14:36 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 319,748 335,536 1.0494
10 Feb 2015 01:38:17 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 316,868 326,145 1.0293
09 Feb 2015 22:56:58 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 313,988 316,737 1.0088
09 Feb 2015 05:47:21 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 296,708 257,024 0.8663
09 Feb 2015 02:40:12 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 293,828 246,555 0.8391
08 Feb 2015 23:12:22 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 290,948 235,446 0.8092
08 Feb 2015 20:06:39 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 288,068 224,793 0.7803
08 Feb 2015 17:15:57 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 285,188 215,281 0.7549
08 Feb 2015 14:30:21 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 282,308 206,178 0.7303
08 Feb 2015 11:39:41 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 279,428 196,602 0.7036
08 Feb 2015 05:38:14 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 273,668 177,610 0.6490
08 Feb 2015 03:17:05 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 270,788 168,318 0.6216
08 Feb 2015 00:35:28 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 267,908 158,970 0.5934
07 Feb 2015 21:49:48 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 265,028 149,815 0.5653
07 Feb 2015 18:59:11 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 262,148 140,278 0.5351
07 Feb 2015 16:08:32 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 259,268 130,489 0.5033
07 Feb 2015 13:17:55 1303626 17839481 hadam3pm2_k2gj_1959_10_009464414_2 256,388 120,893 0.4715


©2024 cpdn.org