climateprediction.net home page
Task 11922681

Task 11922681

Name famous_vn6l_1599_200_006715811_6
Workunit 6919064
Created 4 Oct 2010, 10:42:54 UTC
Sent 4 Oct 2010, 11:57:57 UTC
Report deadline 3 Jan 2011, 19:25:08 UTC
Received 25 Nov 2010, 7:14:11 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1105248
Run time 1 days 7 hours 28 min 34 sec
CPU time 1 days 7 hours 5 min 5 sec
Validate state Invalid
Credit 1,019.18
Device peak FLOPS 2.70 GFLOPS
Application version UK Met Office FAMOUS v6.11
windows_intelx86
Stderr 
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
The device does not recognize the command. (0x16) - exit code 22 (0x16)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7664, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7664, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7664, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7664, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7664, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2732, selfPID=2732, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3620, selfPID=3620, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7160, selfPID=7160, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
05:33:44 (8036): Can't acquire lockfile (32) - waiting 35s
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1720, selfPID=1720, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2572, selfPID=2572, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2948, selfPID=2948, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6192, selfPID=6192, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8288, selfPID=8288, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
05:36:16 (8828): Can't acquire lockfile (32) - waiting 35s
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8648, selfPID=8648, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8352, selfPID=8352, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8760, selfPID=8760, iMonCtr=1
05:37:49 (4616): Can't acquire lockfile (32) - waiting 35s
CPDN Monitor - Quit request from BOINC...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6772, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8656, selfPID=8656, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8880, selfPID=8880, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
05:39:56 (9096): Can't acquire lockfile (32) - waiting 35s
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9076, selfPID=9076, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8792, selfPID=8792, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
05:41:16 (6220): Can't acquire lockfile (32) - waiting 35s
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8412, selfPID=8412, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5480, selfPID=5480, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9032, selfPID=9032, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9036, selfPID=9036, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
05:44:17 (8652): Can't acquire lockfile (32) - waiting 35s
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8412, selfPID=8412, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7476, selfPID=7476, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=124, selfPID=124, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7700, selfPID=7700, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7280, selfPID=7280, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3564, selfPID=3564, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8104, selfPID=8104, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7580, selfPID=7580, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5424, selfPID=5424, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7892, selfPID=7892, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6108, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6108, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8428, selfPID=8428, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4904, selfPID=4904, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8796, selfPID=8796, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7276, selfPID=7276, iMonCtr=1
05:35:32 (8164): Can't acquire lockfile (32) - waiting 35s
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7544, selfPID=7544, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6236, selfPID=6236, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8996, selfPID=8996, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8376, selfPID=8376, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
05:38:22 (8288): Can't acquire lockfile (32) - waiting 35s
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7932, selfPID=7932, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
05:39:14 (4336): Can't acquire lockfile (32) - waiting 35s
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4460, selfPID=4460, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6236, selfPID=6236, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
05:40:34 (5448): Can't acquire lockfile (32) - waiting 35s
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9016, selfPID=9016, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
05:42:25 (4216): Can't acquire lockfile (32) - waiting 35s
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5912, selfPID=5912, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
05:43:14 (2256): Can't acquire lockfile (32) - waiting 35s
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7844, selfPID=7844, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
05:44:04 (8780): Can't acquire lockfile (32) - waiting 35s
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1692, selfPID=1692, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7536, selfPID=7536, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5568, selfPID=5568, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
05:45:45 (4540): Can't acquire lockfile (32) - waiting 35s
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4952, selfPID=4952, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7884, selfPID=7884, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7056, selfPID=7056, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6984, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6984, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6984, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6984, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6984, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vn6l_1599_200_006715811\dataout\stdout_um.txt

Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6984, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5044, selfPID=5044, iMonCtr=1
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/jobs/afyel.ihist after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/jobs/afyel.ihist after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/dataout/ocean_restart.day after 11 attempts

Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO                                                                                                                                                                                           tmp/pipe_dummy                                                                  
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/jobs/afyel.ihist after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/dataout/ocean_restart.day after 11 attempts

Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO                                                                                                                                                                                           tmp/pipe_dummy                                                                  
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/jobs/afyel.ihist after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/dataout/ocean_restart.day after 11 attempts

Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO                                                                                                                                                                                           tmp/pipe_dummy                                                                  
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/jobs/afyel.ihist after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/dataout/ocean_restart.day after 11 attempts

Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO                                                                                                                                                                                           tmp/pipe_dummy                                                                  
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/jobs/afyel.ihist after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/dataout/ocean_restart.day after 11 attempts

Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO                                                                                                                                                                                           tmp/pipe_dummy                                                                  
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/jobs/afyel.ihist after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vn6l_1599_200_006715811/dataout/ocean_restart.day after 11 attempts

Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO                                                                                                                                                                                           tmp/pipe_dummy                                                                  

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
11 Oct 2010 16:09:51 1105248 11922681 famous_vn6l_1599_200_006715811_6 308,906 110,902 0.3590
11 Oct 2010 15:13:06 1105248 11922681 famous_vn6l_1599_200_006715811_6 299,546 107,534 0.3590
11 Oct 2010 14:16:06 1105248 11922681 famous_vn6l_1599_200_006715811_6 290,186 104,192 0.3591
11 Oct 2010 13:22:36 1105248 11922681 famous_vn6l_1599_200_006715811_6 280,826 100,803 0.3590
11 Oct 2010 12:21:05 1105248 11922681 famous_vn6l_1599_200_006715811_6 271,466 97,446 0.3590
11 Oct 2010 11:28:05 1105248 11922681 famous_vn6l_1599_200_006715811_6 262,106 94,097 0.3590
11 Oct 2010 10:31:31 1105248 11922681 famous_vn6l_1599_200_006715811_6 252,746 90,713 0.3589
11 Oct 2010 09:48:25 1105248 11922681 famous_vn6l_1599_200_006715811_6 243,386 87,353 0.3589
11 Oct 2010 08:26:34 1105248 11922681 famous_vn6l_1599_200_006715811_6 234,026 84,013 0.3590
11 Oct 2010 07:25:00 1105248 11922681 famous_vn6l_1599_200_006715811_6 224,666 80,640 0.3589
11 Oct 2010 07:25:00 1105248 11922681 famous_vn6l_1599_200_006715811_6 215,306 77,212 0.3586
11 Oct 2010 06:18:32 1105248 11922681 famous_vn6l_1599_200_006715811_6 205,946 73,853 0.3586
11 Oct 2010 06:18:32 1105248 11922681 famous_vn6l_1599_200_006715811_6 196,586 70,465 0.3584
11 Oct 2010 06:18:32 1105248 11922681 famous_vn6l_1599_200_006715811_6 187,226 67,091 0.3583
11 Oct 2010 06:18:32 1105248 11922681 famous_vn6l_1599_200_006715811_6 177,866 63,743 0.3584
11 Oct 2010 06:18:30 1105248 11922681 famous_vn6l_1599_200_006715811_6 168,506 60,379 0.3583
11 Oct 2010 06:18:24 1105248 11922681 famous_vn6l_1599_200_006715811_6 159,146 56,990 0.3581
10 Oct 2010 23:48:00 1105248 11922681 famous_vn6l_1599_200_006715811_6 149,786 53,609 0.3579
10 Oct 2010 22:51:22 1105248 11922681 famous_vn6l_1599_200_006715811_6 140,426 50,213 0.3576
10 Oct 2010 21:54:28 1105248 11922681 famous_vn6l_1599_200_006715811_6 131,066 46,856 0.3575


©2024 climateprediction.net