climateprediction.net home page
Task 11599639

Task 11599639

Name famous_ulvb_1999_200_006661186_5
Workunit 6864558
Created 9 Jul 2010, 15:08:40 UTC
Sent 9 Jul 2010, 15:33:42 UTC
Report deadline 8 Oct 2010, 23:00:53 UTC
Received 5 Aug 2010, 0:00:19 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1082616
Run time 10 days 2 hours 9 min 45 sec
CPU time 9 days 10 hours 19 min 38 sec
Validate state Invalid
Credit 2,532.38
Device peak FLOPS 1.65 GFLOPS
Application version UK Met Office FAMOUS v6.11
windows_intelx86
Stderr
<core_client_version>6.10.43</core_client_version>
<![CDATA[
<message>
The device does not recognize the command. (0x16) - exit code 22 (0x16)
</message>
<stderr_txt>
error:  cannot delete old C:/ProgramData/BOINC/projects/climateprediction.net/famous_um_6.11_windows_intelx86.exe
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3924, selfPID=3924, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5836, selfPID=5836, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3916, selfPID=3916, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6092, selfPID=6092, iMonCtr=1
13:42:59 (3332): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
13:43:00 (3332): No heartbeat from core client for 30 sec - exiting
13:43:01 (3332): No heartbeat from core client for 30 sec - exiting
13:43:02 (3332): No heartbeat from core client for 30 sec - exiting
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5800, selfPID=5800, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4536, selfPID=4536, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5944, selfPID=5944, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4868, selfPID=4868, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5920, selfPID=5920, iMonCtr=1
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5656, iMonCtr=1
Model crash detected, will try to restart...
16:58:15 (3928): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:58:16 (3928): No heartbeat from core client for 30 sec - exiting
16:58:17 (3928): No heartbeat from core client for 30 sec - exiting
16:58:18 (3928): No heartbeat from core client for 30 sec - exiting
16:58:19 (3928): No heartbeat from core client for 30 sec - exiting
16:58:20 (3928): No heartbeat from core client for 30 sec - exiting
16:58:21 (3928): No heartbeat from core client for 30 sec - exiting
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4264, selfPID=4264, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5664, selfPID=5664, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1388, selfPID=1388, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4460, selfPID=4460, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5216, selfPID=5216, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5532, selfPID=5532, iMonCtr=1
22:57:44 (1160): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
22:57:45 (1160): No heartbeat from core client for 30 sec - exiting
22:57:46 (1160): No heartbeat from core client for 30 sec - exiting
22:57:47 (1160): No heartbeat from core client for 30 sec - exiting
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5304, selfPID=5304, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4580, selfPID=4580, iMonCtr=1
06:48:47 (6336): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:48:48 (6336): No heartbeat from core client for 30 sec - exiting
06:48:49 (6336): No heartbeat from core client for 30 sec - exiting
06:48:50 (6336): No heartbeat from core client for 30 sec - exiting
06:48:51 (6336): No heartbeat from core client for 30 sec - exiting
06:48:52 (6336): No heartbeat from core client for 30 sec - exiting
06:48:53 (6336): No heartbeat from core client for 30 sec - exiting
06:48:55 (6336): No heartbeat from core client for 30 sec - exiting
06:48:56 (6336): No heartbeat from core client for 30 sec - exiting
06:48:57 (6336): No heartbeat from core client for 30 sec - exiting
06:48:58 (6336): No heartbeat from core client for 30 sec - exiting
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6368, selfPID=6368, iMonCtr=1
06:49:54 (6932): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:49:55 (6932): No heartbeat from core client for 30 sec - exiting
06:49:56 (6932): No heartbeat from core client for 30 sec - exiting
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4396, selfPID=4396, iMonCtr=1
06:53:00 (6276): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:53:01 (6276): No heartbeat from core client for 30 sec - exiting
06:53:02 (6276): No heartbeat from core client for 30 sec - exiting
06:53:04 (6276): No heartbeat from core client for 30 sec - exiting
06:53:05 (6276): No heartbeat from core client for 30 sec - exiting
06:53:06 (6276): No heartbeat from core client for 30 sec - exiting
06:53:07 (6276): No heartbeat from core client for 30 sec - exiting
06:53:08 (6276): No heartbeat from core client for 30 sec - exiting
06:53:09 (6276): No heartbeat from core client for 30 sec - exiting
06:53:10 (6276): No heartbeat from core client for 30 sec - exiting
06:53:11 (6276): No heartbeat from core client for 30 sec - exiting
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6864, selfPID=6864, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4528, selfPID=4528, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3352, selfPID=3352, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5728, selfPID=5728, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3212, selfPID=3212, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2612, selfPID=2612, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3888, selfPID=3888, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5924, selfPID=5924, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5992, selfPID=5992, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6016, selfPID=6016, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4436, selfPID=4436, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5156, selfPID=5156, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=616, selfPID=616, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5872, selfPID=5872, iMonCtr=1
23:30:56 (3888): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
23:30:57 (3888): No heartbeat from core client for 30 sec - exiting
23:30:58 (3888): No heartbeat from core client for 30 sec - exiting
23:30:59 (3888): No heartbeat from core client for 30 sec - exiting
23:31:00 (3888): No heartbeat from core client for 30 sec - exiting
23:31:01 (3888): No heartbeat from core client for 30 sec - exiting
23:31:02 (3888): No heartbeat from core client for 30 sec - exiting
23:31:03 (3888): No heartbeat from core client for 30 sec - exiting
23:31:04 (3888): No heartbeat from core client for 30 sec - exiting
23:31:05 (3888): No heartbeat from core client for 30 sec - exiting
23:31:06 (3888): No heartbeat from core client for 30 sec - exiting
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5688, selfPID=5688, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1540, selfPID=1540, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6124, selfPID=6124, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3976, selfPID=3976, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5472, selfPID=5472, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=500, selfPID=500, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5192, selfPID=5192, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7160, selfPID=7160, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5972, selfPID=5972, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5640, selfPID=5640, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
CPDN Monitor - Quit request from BOINC...
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4192, selfPID=4192, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4320, selfPID=4320, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6904, selfPID=6904, iMonCtr=1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
15:46:14 (4636): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:46:15 (4636): No heartbeat from core client for 30 sec - exiting
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_ulvb_1999_200_006661186\dataout\stdout_um.txt
Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=416, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_ulvb_1999_200_006661186\dataout\stdout_um.txt
Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=416, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_ulvb_1999_200_006661186\dataout\stdout_um.txt
Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=416, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_ulvb_1999_200_006661186\dataout\stdout_um.txt
Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=416, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_ulvb_1999_200_006661186\dataout\stdout_um.txt
Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=416, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_ulvb_1999_200_006661186\dataout\stdout_um.txt
Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=416, iMonCtr=1
Model crash detected, will try to restart...
forrtl: severe (47): write to READONLY file, unit 6, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_ulvb_1999_200_006661186\dataout\stdout_um.txt
Image              PC        Routine            Line        Source             
famous_um_6.11_wi  008846D2  Unknown               Unknown  Unknown

Stack trace terminated abnormally.
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=416, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
No Process Handle
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3076, selfPID=3076, iMonCtr=1
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/jobs/afyel.ihist after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/jobs/afyel.ihist after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/dataout/ocean_restart.day after 11 attempts

Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO                                                                                                                                                                                           tmp/pipe_dummy                                                                  
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/jobs/afyel.ihist after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/dataout/ocean_restart.day after 11 attempts

Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO                                                                                                                                                                                           tmp/pipe_dummy                                                                  
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/jobs/afyel.ihist after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/dataout/ocean_restart.day after 11 attempts

Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO                                                                                                                                                                                           tmp/pipe_dummy                                                                  
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/jobs/afyel.ihist after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/dataout/ocean_restart.day after 11 attempts

Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO                                                                                                                                                                                           tmp/pipe_dummy                                                                  
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/jobs/afyel.ihist after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/dataout/ocean_restart.day after 11 attempts

Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO                                                                                                                                                                                           tmp/pipe_dummy                                                                  
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/jobs/afyel.ihist after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_ulvb_1999_200_006661186/dataout/ocean_restart.day after 11 attempts

Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO                                                                                                                                                                                           tmp/pipe_dummy                                                                  
Sorry, too many model crashes! :-(
15:59:32 (5760): called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
05 Aug 2010 00:04:40 1082616 11599639 famous_ulvb_1999_200_006661186_5 767,546 814,786 1.0615
03 Aug 2010 11:10:00 1082616 11599639 famous_ulvb_1999_200_006661186_5 758,186 805,075 1.0618
03 Aug 2010 07:48:58 1082616 11599639 famous_ulvb_1999_200_006661186_5 748,826 793,837 1.0601
03 Aug 2010 04:49:37 1082616 11599639 famous_ulvb_1999_200_006661186_5 739,466 782,890 1.0587
03 Aug 2010 03:08:43 1082616 11599639 famous_ulvb_1999_200_006661186_5 730,106 770,608 1.0555
02 Aug 2010 20:17:19 1082616 11599639 famous_ulvb_1999_200_006661186_5 720,746 759,069 1.0532
02 Aug 2010 09:23:06 1082616 11599639 famous_ulvb_1999_200_006661186_5 711,386 750,437 1.0549
02 Aug 2010 07:11:49 1082616 11599639 famous_ulvb_1999_200_006661186_5 702,026 742,686 1.0579
02 Aug 2010 04:56:58 1082616 11599639 famous_ulvb_1999_200_006661186_5 692,666 734,796 1.0608
02 Aug 2010 02:38:15 1082616 11599639 famous_ulvb_1999_200_006661186_5 683,306 726,632 1.0634
02 Aug 2010 01:56:50 1082616 11599639 famous_ulvb_1999_200_006661186_5 673,946 715,647 1.0619
01 Aug 2010 20:21:13 1082616 11599639 famous_ulvb_1999_200_006661186_5 664,586 705,804 1.0620
01 Aug 2010 17:14:12 1082616 11599639 famous_ulvb_1999_200_006661186_5 655,226 695,835 1.0620
01 Aug 2010 14:31:26 1082616 11599639 famous_ulvb_1999_200_006661186_5 645,866 686,761 1.0633
01 Aug 2010 12:01:22 1082616 11599639 famous_ulvb_1999_200_006661186_5 636,506 678,156 1.0654
01 Aug 2010 09:56:55 1082616 11599639 famous_ulvb_1999_200_006661186_5 627,146 670,822 1.0696
01 Aug 2010 07:46:07 1082616 11599639 famous_ulvb_1999_200_006661186_5 617,786 663,083 1.0733
01 Aug 2010 05:45:28 1082616 11599639 famous_ulvb_1999_200_006661186_5 608,426 655,659 1.0776
01 Aug 2010 00:54:38 1082616 11599639 famous_ulvb_1999_200_006661186_5 599,066 645,154 1.0769
31 Jul 2010 19:33:26 1082616 11599639 famous_ulvb_1999_200_006661186_5 589,706 634,690 1.0763


©2024 climateprediction.net