 |
|
| Name | famous_u893_1599_200_006643538_2 |
| Workunit | 6846910 |
| Created | 10 Jun 2010, 12:34:14 UTC |
| Sent | 4 Aug 2010, 1:35:50 UTC |
| Report deadline | 3 Nov 2010, 9:03:01 UTC |
| Received | 29 Aug 2010, 7:27:49 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 22 (0x00000016) Unknown error code |
| Computer ID | 1079389 |
| Run time | 7 days 15 hours 9 min 15 sec |
| CPU time | 7 days 11 hours 57 min 14 sec |
| Validate state | Invalid |
| Credit | 4,570.57 |
| Device peak FLOPS | 2.66 GFLOPS |
| Application version | UK Met Office FAMOUS v6.11 windows_intelx86 |
| Stderr | <core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 19:09:56 (4312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:13:06 (5900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:13:59 (5440): No heartbeat from core client for 30 sec - exiting 19:14:00 (5440): No heartbeat from core client for 30 sec - exiting 19:14:01 (5440): No heartbeat from core client for 30 sec - exiting 19:14:02 (5440): No heartbeat from core client for 30 sec - exiting 19:14:03 (5440): No heartbeat from core client for 30 sec - exiting 19:14:04 (5440): No heartbeat from core client for 30 sec - exiting 19:14:05 (5440): No heartbeat from core client for 30 sec - exiting 19:14:06 (5440): No heartbeat from core client for 30 sec - exiting 19:14:07 (5440): No heartbeat from core client for 30 sec - exiting 19:14:08 (5440): No heartbeat from core client for 30 sec - exiting 19:14:09 (5440): No heartbeat from core client for 30 sec - exiting 19:14:10 (5440): No heartbeat from core client for 30 sec - exiting 19:14:11 (5440): No heartbeat from core client for 30 sec - exiting 19:14:12 (5440): No heartbeat from core client for 30 sec - exiting 19:14:13 (5440): No heartbeat from core client for 30 sec - exiting 19:14:14 (5440): No heartbeat from core client for 30 sec - exiting 19:14:15 (5440): No heartbeat from core client for 30 sec - exiting 19:14:16 (5440): No heartbeat from core client for 30 sec - exiting 19:14:17 (5440): No heartbeat from core client for 30 sec - exiting 19:14:18 (5440): No heartbeat from core client for 30 sec - exiting 19:14:19 (5440): No heartbeat from core client for 30 sec - exiting 19:14:20 (5440): No heartbeat from core client for 30 sec - exiting 19:14:21 (5440): No heartbeat from core client for 30 sec - exiting 19:14:22 (5440): No heartbeat from core client for 30 sec - exiting 19:14:23 (5440): No heartbeat from core client for 30 sec - exiting 19:14:24 (5440): No heartbeat from core client for 30 sec - exiting 19:14:25 (5440): No heartbeat from core client for 30 sec - exiting 19:14:26 (5440): No heartbeat from core client for 30 sec - exiting 19:14:27 (5440): No heartbeat from core client for 30 sec - exiting 19:14:28 (5440): No heartbeat from core client for 30 sec - exiting 19:14:29 (5440): No heartbeat from core client for 30 sec - exiting 19:14:30 (5440): No heartbeat from core client for 30 sec - exiting 19:14:31 (5440): No heartbeat from core client for 30 sec - exiting 19:14:32 (5440): No heartbeat from core client for 30 sec - exiting 19:14:33 (5440): No heartbeat from core client for 30 sec - exiting 19:14:34 (5440): No heartbeat from core client for 30 sec - exiting 19:14:35 (5440): No heartbeat from core client for 30 sec - exiting 19:14:36 (5440): No heartbeat from core client for 30 sec - exiting 19:14:37 (5440): No heartbeat from core client for 30 sec - exiting 19:14:38 (5440): No heartbeat from core client for 30 sec - exiting 19:14:39 (5440): No heartbeat from core client for 30 sec - exiting 19:14:40 (5440): No heartbeat from core client for 30 sec - exiting 19:14:41 (5440): No heartbeat from core client for 30 sec - exiting 19:14:42 (5440): No heartbeat from core client for 30 sec - exiting 19:14:43 (5440): No heartbeat from core client for 30 sec - exiting 19:14:44 (5440): No heartbeat from core client for 30 sec - exiting 19:14:45 (5440): No heartbeat from core client for 30 sec - exiting 19:14:46 (5440): No heartbeat from core client for 30 sec - exiting 19:14:47 (5440): No heartbeat from core client for 30 sec - exiting 19:14:48 (5440): No heartbeat from core client for 30 sec - exiting 19:14:49 (5440): No heartbeat from core client for 30 sec - exiting 19:14:50 (5440): No heartbeat from core client for 30 sec - exiting 19:14:51 (5440): No heartbeat from core client for 30 sec - exiting 19:14:52 (5440): No heartbeat from core client for 30 sec - exiting 19:14:53 (5440): No heartbeat from core client for 30 sec - exiting 19:14:54 (5440): No heartbeat from core client for 30 sec - exiting 19:14:55 (5440): No heartbeat from core client for 30 sec - exiting 19:14:56 (5440): No heartbeat from core client for 30 sec - exiting 19:14:57 (5440): No heartbeat from core client for 30 sec - exiting 19:14:58 (5440): No heartbeat from core client for 30 sec - exiting 19:14:59 (5440): No heartbeat from core client for 30 sec - exiting 19:15:00 (5440): No heartbeat from core client for 30 sec - exiting 19:15:01 (5440): No heartbeat from core client for 30 sec - exiting 19:15:02 (5440): No heartbeat from core client for 30 sec - exiting 19:15:03 (5440): No heartbeat from core client for 30 sec - exiting 19:15:04 (5440): No heartbeat from core client for 30 sec - exiting 19:15:05 (5440): No heartbeat from core client for 30 sec - exiting 19:15:06 (5440): No heartbeat from core client for 30 sec - exiting 19:15:07 (5440): No heartbeat from core client for 30 sec - exiting 19:15:08 (5440): No heartbeat from core client for 30 sec - exiting 19:15:09 (5440): No heartbeat from core client for 30 sec - exiting 19:15:10 (5440): No heartbeat from core client for 30 sec - exiting 19:15:11 (5440): No heartbeat from core client for 30 sec - exiting 19:15:12 (5440): No heartbeat from core client for 30 sec - exiting 19:15:13 (5440): No heartbeat from core client for 30 sec - exiting 19:15:14 (5440): No heartbeat from core client for 30 sec - exiting 19:15:15 (5440): No heartbeat from core client for 30 sec - exiting 19:15:16 (5440): No heartbeat from core client for 30 sec - exiting 19:15:17 (5440): No heartbeat from core client for 30 sec - exiting 19:15:18 (5440): No heartbeat from core client for 30 sec - exiting 19:15:19 (5440): No heartbeat from core client for 30 sec - exiting 19:15:20 (5440): No heartbeat from core client for 30 sec - exiting 19:15:21 (5440): No heartbeat from core client for 30 sec - exiting 19:15:22 (5440): No heartbeat from core client for 30 sec - exiting 19:15:23 (5440): No heartbeat from core client for 30 sec - exiting 19:15:24 (5440): No heartbeat from core client for 30 sec - exiting 19:15:25 (5440): No heartbeat from core client for 30 sec - exiting 19:15:26 (5440): No heartbeat from core client for 30 sec - exiting 19:15:27 (5440): No heartbeat from core client for 30 sec - exiting 19:15:28 (5440): No heartbeat from core client for 30 sec - exiting 19:15:29 (5440): No heartbeat from core client for 30 sec - exiting 19:15:30 (5440): No heartbeat from core client for 30 sec - exiting 19:15:31 (5440): No heartbeat from core client for 30 sec - exiting 19:15:32 (5440): No heartbeat from core client for 30 sec - exiting 19:15:33 (5440): No heartbeat from core client for 30 sec - exiting 19:15:34 (5440): No heartbeat from core client for 30 sec - exiting 19:15:35 (5440): No heartbeat from core client for 30 sec - exiting 19:15:36 (5440): No heartbeat from core client for 30 sec - exiting 19:15:37 (5440): No heartbeat from core client for 30 sec - exiting 19:15:38 (5440): No heartbeat from core client for 30 sec - exiting 19:15:39 (5440): No heartbeat from core client for 30 sec - exiting 19:15:40 (5440): No heartbeat from core client for 30 sec - exiting 19:15:41 (5440): No heartbeat from core client for 30 sec - exiting 19:15:42 (5440): No heartbeat from core client for 30 sec - exiting 19:15:43 (5440): No heartbeat from core client for 30 sec - exiting 19:15:44 (5440): No heartbeat from core client for 30 sec - exiting 19:15:45 (5440): No heartbeat from core client for 30 sec - exiting 19:15:46 (5440): No heartbeat from core client for 30 sec - exiting 19:15:47 (5440): No heartbeat from core client for 30 sec - exiting 19:15:48 (5440): No heartbeat from core client for 30 sec - exiting 19:15:49 (5440): No heartbeat from core client for 30 sec - exiting 19:15:50 (5440): No heartbeat from core client for 30 sec - exiting 19:15:51 (5440): No heartbeat from core client for 30 sec - exiting 19:15:52 (5440): No heartbeat from core client for 30 sec - exiting 19:15:53 (5440): No heartbeat from core client for 30 sec - exiting 19:15:54 (5440): No heartbeat from core client for 30 sec - exiting 19:15:55 (5440): No heartbeat from core client for 30 sec - exiting 19:15:56 (5440): No heartbeat from core client for 30 sec - exiting 19:15:57 (5440): No heartbeat from core client for 30 sec - exiting 19:15:58 (5440): No heartbeat from core client for 30 sec - exiting 19:15:59 (5440): No heartbeat from core client for 30 sec - exiting 19:16:00 (5440): No heartbeat from core client for 30 sec - exiting 19:16:01 (5440): No heartbeat from core client for 30 sec - exiting 19:16:02 (5440): No heartbeat from core client for 30 sec - exiting 19:16:03 (5440): No heartbeat from core client for 30 sec - exiting 19:16:04 (5440): No heartbeat from core client for 30 sec - exiting 19:16:05 (5440): No heartbeat from core client for 30 sec - exiting 19:16:06 (5440): No heartbeat from core client for 30 sec - exiting 19:16:07 (5440): No heartbeat from core client for 30 sec - exiting 19:16:08 (5440): No heartbeat from core client for 30 sec - exiting 19:16:09 (5440): No heartbeat from core client for 30 sec - exiting 19:16:10 (5440): No heartbeat from core client for 30 sec - exiting 19:16:11 (5440): No heartbeat from core client for 30 sec - exiting 19:16:12 (5440): No heartbeat from core client for 30 sec - exiting 19:16:13 (5440): No heartbeat from core client for 30 sec - exiting 19:16:14 (5440): No heartbeat from core client for 30 sec - exiting 19:16:15 (5440): No heartbeat from core client for 30 sec - exiting 19:16:16 (5440): No heartbeat from core client for 30 sec - exiting 19:16:17 (5440): No heartbeat from core client for 30 sec - exiting 19:16:18 (5440): No heartbeat from core client for 30 sec - exiting 19:16:19 (5440): No heartbeat from core client for 30 sec - exiting 19:16:20 (5440): No heartbeat from core client for 30 sec - exiting 19:16:21 (5440): No heartbeat from core client for 30 sec - exiting 19:16:22 (5440): No heartbeat from core client for 30 sec - exiting 19:16:23 (5440): No heartbeat from core client for 30 sec - exiting 19:16:24 (5440): No heartbeat from core client for 30 sec - exiting 19:16:25 (5440): No heartbeat from core client for 30 sec - exiting 19:16:26 (5440): No heartbeat from core client for 30 sec - exiting 19:16:27 (5440): No heartbeat from core client for 30 sec - exiting 19:16:28 (5440): No heartbeat from core client for 30 sec - exiting 19:16:29 (5440): No heartbeat from core client for 30 sec - exiting 19:16:30 (5440): No heartbeat from core client for 30 sec - exiting 19:16:31 (5440): No heartbeat from core client for 30 sec - exiting 19:16:32 (5440): No heartbeat from core client for 30 sec - exiting 19:16:33 (5440): No heartbeat from core client for 30 sec - exiting 19:16:34 (5440): No heartbeat from core client for 30 sec - exiting 19:16:35 (5440): No heartbeat from core client for 30 sec - exiting 19:16:36 (5440): No heartbeat from core client for 30 sec - exiting 19:16:37 (5440): No heartbeat from core client for 30 sec - exiting 19:16:38 (5440): No heartbeat from core client for 30 sec - exiting 19:16:39 (5440): No heartbeat from core client for 30 sec - exiting 19:16:40 (5440): No heartbeat from core client for 30 sec - exiting 19:16:41 (5440): No heartbeat from core client for 30 sec - exiting 19:16:42 (5440): No heartbeat from core client for 30 sec - exiting 19:16:43 (5440): No heartbeat from core client for 30 sec - exiting 19:16:44 (5440): No heartbeat from core client for 30 sec - exiting 19:16:45 (5440): No heartbeat from core client for 30 sec - exiting 19:16:46 (5440): No heartbeat from core client for 30 sec - exiting 19:16:47 (5440): No heartbeat from core client for 30 sec - exiting 19:16:48 (5440): No heartbeat from core client for 30 sec - exiting 19:16:49 (5440): No heartbeat from core client for 30 sec - exiting 19:16:50 (5440): No heartbeat from core client for 30 sec - exiting 19:16:51 (5440): No heartbeat from core client for 30 sec - exiting 19:16:52 (5440): No heartbeat from core client for 30 sec - exiting 19:16:53 (5440): No heartbeat from core client for 30 sec - exiting 19:16:54 (5440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:23:21 (3364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:23:22 (3364): No heartbeat from core client for 30 sec - exiting 19:23:23 (3364): No heartbeat from core client for 30 sec - exiting 19:23:24 (3364): No heartbeat from core client for 30 sec - exiting 19:23:25 (3364): No heartbeat from core client for 30 sec - exiting 19:23:26 (3364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5008, iMonCtr=1 Model crash detected, will try to restart... 17:44:45 (5268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:45:33 (2612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:45:34 (2612): No heartbeat from core client for 30 sec - exiting 17:45:35 (2612): No heartbeat from core client for 30 sec - exiting 17:45:36 (2612): No heartbeat from core client for 30 sec - exiting 17:45:37 (2612): No heartbeat from core client for 30 sec - exiting 17:45:38 (2612): No heartbeat from core client for 30 sec - exiting 17:45:39 (2612): No heartbeat from core client for 30 sec - exiting 17:45:40 (2612): No heartbeat from core client for 30 sec - exiting 17:45:41 (2612): No heartbeat from core client for 30 sec - exiting 17:45:42 (2612): No heartbeat from core client for 30 sec - exiting 17:45:43 (2612): No heartbeat from core client for 30 sec - exiting 17:47:28 (3024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:47:29 (3024): No heartbeat from core client for 30 sec - exiting 17:47:30 (3024): No heartbeat from core client for 30 sec - exiting 17:47:31 (3024): No heartbeat from core client for 30 sec - exiting 17:47:32 (3024): No heartbeat from core client for 30 sec - exiting 17:47:33 (3024): No heartbeat from core client for 30 sec - exiting 17:47:34 (3024): No heartbeat from core client for 30 sec - exiting 17:47:35 (3024): No heartbeat from core client for 30 sec - exiting 17:47:36 (3024): No heartbeat from core client for 30 sec - exiting 17:47:37 (3024): No heartbeat from core client for 30 sec - exiting 17:47:38 (3024): No heartbeat from core client for 30 sec - exiting 17:49:23 (5704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:09:18 (6268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:35:47 (820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:36:39 (7120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:37:37 (7088): No heartbeat from core client for 30 sec - exiting 21:37:38 (7088): No heartbeat from core client for 30 sec - exiting 21:37:39 (7088): No heartbeat from core client for 30 sec - exiting 21:37:40 (7088): No heartbeat from core client for 30 sec - exiting 21:37:41 (7088): No heartbeat from core client for 30 sec - exiting 21:37:42 (7088): No heartbeat from core client for 30 sec - exiting 21:37:43 (7088): No heartbeat from core client for 30 sec - exiting 21:37:44 (7088): No heartbeat from core client for 30 sec - exiting 21:37:45 (7088): No heartbeat from core client for 30 sec - exiting 21:37:46 (7088): No heartbeat from core client for 30 sec - exiting 21:37:47 (7088): No heartbeat from core client for 30 sec - exiting 21:37:48 (7088): No heartbeat from core client for 30 sec - exiting 21:37:49 (7088): No heartbeat from core client for 30 sec - exiting 21:37:50 (7088): No heartbeat from core client for 30 sec - exiting 21:37:51 (7088): No heartbeat from core client for 30 sec - exiting 21:37:52 (7088): No heartbeat from core client for 30 sec - exiting 21:37:53 (7088): No heartbeat from core client for 30 sec - exiting 21:37:54 (7088): No heartbeat from core client for 30 sec - exiting 21:37:55 (7088): No heartbeat from core client for 30 sec - exiting 21:37:56 (7088): No heartbeat from core client for 30 sec - exiting 21:37:57 (7088): No heartbeat from core client for 30 sec - exiting 21:37:58 (7088): No heartbeat from core client for 30 sec - exiting 21:37:59 (7088): No heartbeat from core client for 30 sec - exiting 21:38:00 (7088): No heartbeat from core client for 30 sec - exiting 21:38:01 (7088): No heartbeat from core client for 30 sec - exiting 21:38:02 (7088): No heartbeat from core client for 30 sec - exiting 21:38:03 (7088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:38:38 (6448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:39:33 (7020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:15:09 (6460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:15:10 (6460): No heartbeat from core client for 30 sec - exiting 03:15:11 (6460): No heartbeat from core client for 30 sec - exiting 03:15:12 (6460): No heartbeat from core client for 30 sec - exiting 03:15:13 (6460): No heartbeat from core client for 30 sec - exiting 03:15:14 (6460): No heartbeat from core client for 30 sec - exiting 03:15:15 (6460): No heartbeat from core client for 30 sec - exiting 03:15:16 (6460): No heartbeat from core client for 30 sec - exiting 03:15:17 (6460): No heartbeat from core client for 30 sec - exiting cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_u893_1599_200_006643538/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_u893_1599_200_006643538/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_u893_1599_200_006643538/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_u893_1599_200_006643538/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_u893_1599_200_006643538/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/famous_u893_1599_200_006643538/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy Sorry, too many model crashes! :-( 03:16:33 (7148): called boinc_finish </stderr_txt> ]]> |
| Latest Trickles Received | ||||||
|---|---|---|---|---|---|---|
| Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
| 29 Aug 2010 06:39:17 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,385,306 | 645,662 | 0.4661 |
| 29 Aug 2010 05:19:56 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,375,946 | 641,271 | 0.4661 |
| 29 Aug 2010 04:06:15 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,366,586 | 636,953 | 0.4661 |
| 29 Aug 2010 02:46:47 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,357,226 | 632,492 | 0.4660 |
| 29 Aug 2010 01:35:57 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,347,866 | 628,692 | 0.4664 |
| 29 Aug 2010 01:29:23 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,338,506 | 624,095 | 0.4663 |
| 28 Aug 2010 22:48:26 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,329,146 | 619,728 | 0.4663 |
| 28 Aug 2010 21:34:51 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,319,786 | 615,386 | 0.4663 |
| 28 Aug 2010 20:20:20 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,310,426 | 611,034 | 0.4663 |
| 28 Aug 2010 19:09:32 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,301,066 | 606,705 | 0.4663 |
| 28 Aug 2010 17:57:04 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,291,706 | 602,373 | 0.4663 |
| 28 Aug 2010 16:43:14 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,282,346 | 597,977 | 0.4663 |
| 28 Aug 2010 15:28:47 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,272,986 | 593,556 | 0.4663 |
| 28 Aug 2010 14:15:01 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,263,626 | 589,273 | 0.4663 |
| 28 Aug 2010 13:00:42 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,254,266 | 584,946 | 0.4664 |
| 28 Aug 2010 11:51:31 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,244,906 | 580,580 | 0.4664 |
| 28 Aug 2010 10:36:37 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,235,546 | 576,222 | 0.4664 |
| 28 Aug 2010 07:19:09 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,226,186 | 571,854 | 0.4664 |
| 28 Aug 2010 06:06:25 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,216,826 | 567,503 | 0.4664 |
| 28 Aug 2010 04:53:06 | 1079389 | 11468513 | famous_u893_1599_200_006643538_2 | 1,207,466 | 563,160 | 0.4664 |
©2024 climateprediction.net