 |
|
| Name | famous_u8d0_1799_200_006643679_3 |
| Workunit | 6847051 |
| Created | 10 Jun 2010, 12:35:30 UTC |
| Sent | 4 Aug 2010, 8:27:26 UTC |
| Report deadline | 3 Nov 2010, 15:54:37 UTC |
| Received | 12 Aug 2010, 5:14:35 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 22 (0x00000016) Unknown error code |
| Computer ID | 1090890 |
| Run time | 7 days 19 hours 21 min 40 sec |
| CPU time | 7 days 15 hours 51 min 37 sec |
| Validate state | Invalid |
| Credit | 5,682.31 |
| Device peak FLOPS | 3.01 GFLOPS |
| Application version | UK Met Office FAMOUS v6.11 windows_intelx86 |
| Stderr | <core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 05:33:15 (4120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:35:18 (2380): No heartbeat from core client for 30 sec - exiting 05:35:19 (2380): No heartbeat from core client for 30 sec - exiting 05:35:20 (2380): No heartbeat from core client for 30 sec - exiting 05:35:21 (2380): No heartbeat from core client for 30 sec - exiting 05:35:23 (2380): No heartbeat from core client for 30 sec - exiting 05:35:24 (2380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:02:46 (3492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:29:38 (4656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:30:23 (2512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:28:31 (3960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:40:36 (3292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:42:54 (2144): No heartbeat from core client for 30 sec - exiting 20:42:55 (2144): No heartbeat from core client for 30 sec - exiting 20:42:56 (2144): No heartbeat from core client for 30 sec - exiting 20:42:57 (2144): No heartbeat from core client for 30 sec - exiting 20:42:58 (2144): No heartbeat from core client for 30 sec - exiting 20:42:59 (2144): No heartbeat from core client for 30 sec - exiting 20:43:00 (2144): No heartbeat from core client for 30 sec - exiting 20:43:01 (2144): No heartbeat from core client for 30 sec - exiting 20:43:02 (2144): No heartbeat from core client for 30 sec - exiting 20:43:03 (2144): No heartbeat from core client for 30 sec - exiting 20:43:05 (2144): No heartbeat from core client for 30 sec - exiting 20:43:06 (2144): No heartbeat from core client for 30 sec - exiting 20:43:07 (2144): No heartbeat from core client for 30 sec - exiting 20:43:08 (2144): No heartbeat from core client for 30 sec - exiting 20:43:09 (2144): No heartbeat from core client for 30 sec - exiting 20:43:10 (2144): No heartbeat from core client for 30 sec - exiting 20:43:11 (2144): No heartbeat from core client for 30 sec - exiting 20:43:12 (2144): No heartbeat from core client for 30 sec - exiting 20:43:13 (2144): No heartbeat from core client for 30 sec - exiting 20:43:14 (2144): No heartbeat from core client for 30 sec - exiting 20:43:15 (2144): No heartbeat from core client for 30 sec - exiting 20:43:17 (2144): No heartbeat from core client for 30 sec - exiting 20:43:18 (2144): No heartbeat from core client for 30 sec - exiting 20:43:19 (2144): No heartbeat from core client for 30 sec - exiting 20:43:20 (2144): No heartbeat from core client for 30 sec - exiting 20:43:21 (2144): No heartbeat from core client for 30 sec - exiting 20:43:22 (2144): No heartbeat from core client for 30 sec - exiting 20:43:23 (2144): No heartbeat from core client for 30 sec - exiting 20:43:24 (2144): No heartbeat from core client for 30 sec - exiting 20:43:25 (2144): No heartbeat from core client for 30 sec - exiting 20:43:26 (2144): No heartbeat from core client for 30 sec - exiting 20:43:27 (2144): No heartbeat from core client for 30 sec - exiting 20:43:29 (2144): No heartbeat from core client for 30 sec - exiting 20:43:30 (2144): No heartbeat from core client for 30 sec - exiting 20:43:31 (2144): No heartbeat from core client for 30 sec - exiting 20:43:32 (2144): No heartbeat from core client for 30 sec - exiting 20:43:33 (2144): No heartbeat from core client for 30 sec - exiting 20:43:34 (2144): No heartbeat from core client for 30 sec - exiting 20:43:35 (2144): No heartbeat from core client for 30 sec - exiting 20:43:36 (2144): No heartbeat from core client for 30 sec - exiting 20:43:37 (2144): No heartbeat from core client for 30 sec - exiting 20:43:38 (2144): No heartbeat from core client for 30 sec - exiting 20:43:39 (2144): No heartbeat from core client for 30 sec - exiting 20:43:41 (2144): No heartbeat from core client for 30 sec - exiting 20:43:42 (2144): No heartbeat from core client for 30 sec - exiting 20:43:43 (2144): No heartbeat from core client for 30 sec - exiting 20:43:44 (2144): No heartbeat from core client for 30 sec - exiting 20:43:45 (2144): No heartbeat from core client for 30 sec - exiting 20:43:46 (2144): No heartbeat from core client for 30 sec - exiting 20:43:47 (2144): No heartbeat from core client for 30 sec - exiting 20:43:48 (2144): No heartbeat from core client for 30 sec - exiting 20:43:49 (2144): No heartbeat from core client for 30 sec - exiting 20:43:50 (2144): No heartbeat from core client for 30 sec - exiting 20:43:51 (2144): No heartbeat from core client for 30 sec - exiting 20:43:53 (2144): No heartbeat from core client for 30 sec - exiting 20:43:54 (2144): No heartbeat from core client for 30 sec - exiting 20:43:55 (2144): No heartbeat from core client for 30 sec - exiting 20:43:56 (2144): No heartbeat from core client for 30 sec - exiting 20:43:57 (2144): No heartbeat from core client for 30 sec - exiting 20:43:58 (2144): No heartbeat from core client for 30 sec - exiting 20:43:59 (2144): No heartbeat from core client for 30 sec - exiting 20:44:00 (2144): No heartbeat from core client for 30 sec - exiting 20:44:01 (2144): No heartbeat from core client for 30 sec - exiting 20:44:02 (2144): No heartbeat from core client for 30 sec - exiting 20:44:04 (2144): No heartbeat from core client for 30 sec - exiting 20:44:05 (2144): No heartbeat from core client for 30 sec - exiting 20:44:06 (2144): No heartbeat from core client for 30 sec - exiting 20:44:07 (2144): No heartbeat from core client for 30 sec - exiting 20:44:08 (2144): No heartbeat from core client for 30 sec - exiting 20:44:09 (2144): No heartbeat from core client for 30 sec - exiting 20:44:10 (2144): No heartbeat from core client for 30 sec - exiting 20:44:11 (2144): No heartbeat from core client for 30 sec - exiting 20:44:12 (2144): No heartbeat from core client for 30 sec - exiting 20:44:13 (2144): No heartbeat from core client for 30 sec - exiting 20:44:14 (2144): No heartbeat from core client for 30 sec - exiting 20:44:16 (2144): No heartbeat from core client for 30 sec - exiting 20:44:17 (2144): No heartbeat from core client for 30 sec - exiting 20:44:18 (2144): No heartbeat from core client for 30 sec - exiting 20:44:19 (2144): No heartbeat from core client for 30 sec - exiting 20:44:20 (2144): No heartbeat from core client for 30 sec - exiting 20:44:21 (2144): No heartbeat from core client for 30 sec - exiting 20:44:22 (2144): No heartbeat from core client for 30 sec - exiting 20:44:23 (2144): No heartbeat from core client for 30 sec - exiting 20:44:24 (2144): No heartbeat from core client for 30 sec - exiting 20:44:25 (2144): No heartbeat from core client for 30 sec - exiting 20:44:26 (2144): No heartbeat from core client for 30 sec - exiting 20:44:28 (2144): No heartbeat from core client for 30 sec - exiting 20:44:29 (2144): No heartbeat from core client for 30 sec - exiting 20:44:30 (2144): No heartbeat from core client for 30 sec - exiting 20:44:31 (2144): No heartbeat from core client for 30 sec - exiting 20:44:32 (2144): No heartbeat from core client for 30 sec - exiting 20:44:33 (2144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:01:13 (4004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:01:14 (4004): No heartbeat from core client for 30 sec - exiting 10:01:15 (4004): No heartbeat from core client for 30 sec - exiting 10:01:16 (4004): No heartbeat from core client for 30 sec - exiting 10:01:17 (4004): No heartbeat from core client for 30 sec - exiting 10:01:18 (4004): No heartbeat from core client for 30 sec - exiting 10:01:19 (4004): No heartbeat from core client for 30 sec - exiting 10:01:20 (4004): No heartbeat from core client for 30 sec - exiting 10:01:21 (4004): No heartbeat from core client for 30 sec - exiting 10:01:22 (4004): No heartbeat from core client for 30 sec - exiting 10:02:16 (876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:02:17 (876): No heartbeat from core client for 30 sec - exiting 10:02:18 (876): No heartbeat from core client for 30 sec - exiting 10:02:19 (876): No heartbeat from core client for 30 sec - exiting 15:18:22 (1204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:18:56 (3804): No heartbeat from core client for 30 sec - exiting 15:18:57 (3804): No heartbeat from core client for 30 sec - exiting 15:18:58 (3804): No heartbeat from core client for 30 sec - exiting 15:19:00 (3804): No heartbeat from core client for 30 sec - exiting 15:19:01 (3804): No heartbeat from core client for 30 sec - exiting 15:19:02 (3804): No heartbeat from core client for 30 sec - exiting 15:19:03 (3804): No heartbeat from core client for 30 sec - exiting 15:19:04 (3804): No heartbeat from core client for 30 sec - exiting 15:19:05 (3804): No heartbeat from core client for 30 sec - exiting 15:19:06 (3804): No heartbeat from core client for 30 sec - exiting 15:19:07 (3804): No heartbeat from core client for 30 sec - exiting 15:19:08 (3804): No heartbeat from core client for 30 sec - exiting 15:19:09 (3804): No heartbeat from core client for 30 sec - exiting 15:19:10 (3804): No heartbeat from core client for 30 sec - exiting 15:19:12 (3804): No heartbeat from core client for 30 sec - exiting 15:19:13 (3804): No heartbeat from core client for 30 sec - exiting 15:19:14 (3804): No heartbeat from core client for 30 sec - exiting 15:19:15 (3804): No heartbeat from core client for 30 sec - exiting 15:19:16 (3804): No heartbeat from core client for 30 sec - exiting 15:19:17 (3804): No heartbeat from core client for 30 sec - exiting 15:19:18 (3804): No heartbeat from core client for 30 sec - exiting 15:19:19 (3804): No heartbeat from core client for 30 sec - exiting 15:19:20 (3804): No heartbeat from core client for 30 sec - exiting 15:19:21 (3804): No heartbeat from core client for 30 sec - exiting 15:19:22 (3804): No heartbeat from core client for 30 sec - exiting 15:19:24 (3804): No heartbeat from core client for 30 sec - exiting 15:19:25 (3804): No heartbeat from core client for 30 sec - exiting 15:19:26 (3804): No heartbeat from core client for 30 sec - exiting 15:19:27 (3804): No heartbeat from core client for 30 sec - exiting 15:19:28 (3804): No heartbeat from core client for 30 sec - exiting 15:19:29 (3804): No heartbeat from core client for 30 sec - exiting 15:19:30 (3804): No heartbeat from core client for 30 sec - exiting 15:19:31 (3804): No heartbeat from core client for 30 sec - exiting 15:19:32 (3804): No heartbeat from core client for 30 sec - exiting 15:19:33 (3804): No heartbeat from core client for 30 sec - exiting 15:19:34 (3804): No heartbeat from core client for 30 sec - exiting 15:19:36 (3804): No heartbeat from core client for 30 sec - exiting 15:19:37 (3804): No heartbeat from core client for 30 sec - exiting 15:19:38 (3804): No heartbeat from core client for 30 sec - exiting 15:19:39 (3804): No heartbeat from core client for 30 sec - exiting 15:19:40 (3804): No heartbeat from core client for 30 sec - exiting 15:19:41 (3804): No heartbeat from core client for 30 sec - exiting 15:19:42 (3804): No heartbeat from core client for 30 sec - exiting 15:19:43 (3804): No heartbeat from core client for 30 sec - exiting 15:19:44 (3804): No heartbeat from core client for 30 sec - exiting 15:19:45 (3804): No heartbeat from core client for 30 sec - exiting 15:19:46 (3804): No heartbeat from core client for 30 sec - exiting 15:19:48 (3804): No heartbeat from core client for 30 sec - exiting 15:19:49 (3804): No heartbeat from core client for 30 sec - exiting 15:19:50 (3804): No heartbeat from core client for 30 sec - exiting 15:19:51 (3804): No heartbeat from core client for 30 sec - exiting 15:19:52 (3804): No heartbeat from core client for 30 sec - exiting 15:19:53 (3804): No heartbeat from core client for 30 sec - exiting 15:19:54 (3804): No heartbeat from core client for 30 sec - exiting 15:19:55 (3804): No heartbeat from core client for 30 sec - exiting 15:19:56 (3804): No heartbeat from core client for 30 sec - exiting 15:19:57 (3804): No heartbeat from core client for 30 sec - exiting 15:19:58 (3804): No heartbeat from core client for 30 sec - exiting 15:20:00 (3804): No heartbeat from core client for 30 sec - exiting 15:20:01 (3804): No heartbeat from core client for 30 sec - exiting 15:20:02 (3804): No heartbeat from core client for 30 sec - exiting 15:20:03 (3804): No heartbeat from core client for 30 sec - exiting 15:20:04 (3804): No heartbeat from core client for 30 sec - exiting 15:20:05 (3804): No heartbeat from core client for 30 sec - exiting 15:20:06 (3804): No heartbeat from core client for 30 sec - exiting 15:20:07 (3804): No heartbeat from core client for 30 sec - exiting 15:20:08 (3804): No heartbeat from core client for 30 sec - exiting 15:20:09 (3804): No heartbeat from core client for 30 sec - exiting 15:20:10 (3804): No heartbeat from core client for 30 sec - exiting 15:20:12 (3804): No heartbeat from core client for 30 sec - exiting 15:20:13 (3804): No heartbeat from core client for 30 sec - exiting 15:20:14 (3804): No heartbeat from core client for 30 sec - exiting 15:20:15 (3804): No heartbeat from core client for 30 sec - exiting 15:20:16 (3804): No heartbeat from core client for 30 sec - exiting 15:20:17 (3804): No heartbeat from core client for 30 sec - exiting 15:20:18 (3804): No heartbeat from core client for 30 sec - exiting 15:20:19 (3804): No heartbeat from core client for 30 sec - exiting 15:20:20 (3804): No heartbeat from core client for 30 sec - exiting 15:20:21 (3804): No heartbeat from core client for 30 sec - exiting 15:20:22 (3804): No heartbeat from core client for 30 sec - exiting 15:20:24 (3804): No heartbeat from core client for 30 sec - exiting 15:20:25 (3804): No heartbeat from core client for 30 sec - exiting 15:20:26 (3804): No heartbeat from core client for 30 sec - exiting 15:20:27 (3804): No heartbeat from core client for 30 sec - exiting 15:20:28 (3804): No heartbeat from core client for 30 sec - exiting 15:20:29 (3804): No heartbeat from core client for 30 sec - exiting 15:20:30 (3804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:23:24 (3596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:49:46 (3724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:52:19 (3680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:52:20 (3680): No heartbeat from core client for 30 sec - exiting 01:52:21 (3680): No heartbeat from core client for 30 sec - exiting 01:52:22 (3680): No heartbeat from core client for 30 sec - exiting 01:52:23 (3680): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 08:31:57 (3756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:31:58 (3756): No heartbeat from core client for 30 sec - exiting 08:31:59 (3756): No heartbeat from core client for 30 sec - exiting 08:32:00 (3756): No heartbeat from core client for 30 sec - exiting 08:32:01 (3756): No heartbeat from core client for 30 sec - exiting 08:32:02 (3756): No heartbeat from core client for 30 sec - exiting 08:32:03 (3756): No heartbeat from core client for 30 sec - exiting 08:32:04 (3756): No heartbeat from core client for 30 sec - exiting 08:32:05 (3756): No heartbeat from core client for 30 sec - exiting 08:32:07 (3756): No heartbeat from core client for 30 sec - exiting 08:32:08 (3756): No heartbeat from core client for 30 sec - exiting 08:32:09 (3756): No heartbeat from core client for 30 sec - exiting 08:32:10 (3756): No heartbeat from core client for 30 sec - exiting 08:32:11 (3756): No heartbeat from core client for 30 sec - exiting 10:53:11 (3640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:53:25 (3640): No heartbeat from core client for 30 sec - exiting 10:53:27 (3640): No heartbeat from core client for 30 sec - exiting 10:53:28 (3640): No heartbeat from core client for 30 sec - exiting 10:53:29 (3640): No heartbeat from core client for 30 sec - exiting 10:53:30 (3640): No heartbeat from core client for 30 sec - exiting 10:53:31 (3640): No heartbeat from core client for 30 sec - exiting 10:53:32 (3640): No heartbeat from core client for 30 sec - exiting 10:53:33 (3640): No heartbeat from core client for 30 sec - exiting 10:53:34 (3640): No heartbeat from core client for 30 sec - exiting 10:53:35 (3640): No heartbeat from core client for 30 sec - exiting 10:53:36 (3640): No heartbeat from core client for 30 sec - exiting 10:53:38 (3640): No heartbeat from core client for 30 sec - exiting 10:53:39 (3640): No heartbeat from core client for 30 sec - exiting 10:53:40 (3640): No heartbeat from core client for 30 sec - exiting 10:53:41 (3640): No heartbeat from core client for 30 sec - exiting 10:53:42 (3640): No heartbeat from core client for 30 sec - exiting 10:53:43 (3640): No heartbeat from core client for 30 sec - exiting 10:53:44 (3640): No heartbeat from core client for 30 sec - exiting 10:53:45 (3640): No heartbeat from core client for 30 sec - exiting 10:53:46 (3640): No heartbeat from core client for 30 sec - exiting 10:53:47 (3640): No heartbeat from core client for 30 sec - exiting 10:53:48 (3640): No heartbeat from core client for 30 sec - exiting 10:53:50 (3640): No heartbeat from core client for 30 sec - exiting 10:53:51 (3640): No heartbeat from core client for 30 sec - exiting 10:53:52 (3640): No heartbeat from core client for 30 sec - exiting 10:53:53 (3640): No heartbeat from core client for 30 sec - exiting 10:53:54 (3640): No heartbeat from core client for 30 sec - exiting 10:53:55 (3640): No heartbeat from core client for 30 sec - exiting 10:53:56 (3640): No heartbeat from core client for 30 sec - exiting 10:53:57 (3640): No heartbeat from core client for 30 sec - exiting 10:53:58 (3640): No heartbeat from core client for 30 sec - exiting 10:53:59 (3640): No heartbeat from core client for 30 sec - exiting 10:54:00 (3640): No heartbeat from core client for 30 sec - exiting 10:54:02 (3640): No heartbeat from core client for 30 sec - exiting 10:54:03 (3640): No heartbeat from core client for 30 sec - exiting 10:54:04 (3640): No heartbeat from core client for 30 sec - exiting 10:54:05 (3640): No heartbeat from core client for 30 sec - exiting 10:54:06 (3640): No heartbeat from core client for 30 sec - exiting 10:54:07 (3640): No heartbeat from core client for 30 sec - exiting 10:54:08 (3640): No heartbeat from core client for 30 sec - exiting 10:54:09 (3640): No heartbeat from core client for 30 sec - exiting 10:54:10 (3640): No heartbeat from core client for 30 sec - exiting 10:54:11 (3640): No heartbeat from core client for 30 sec - exiting 10:54:12 (3640): No heartbeat from core client for 30 sec - exiting 10:54:14 (3640): No heartbeat from core client for 30 sec - exiting 10:54:15 (3640): No heartbeat from core client for 30 sec - exiting 10:54:16 (3640): No heartbeat from core client for 30 sec - exiting 10:54:17 (3640): No heartbeat from core client for 30 sec - exiting 10:54:18 (3640): No heartbeat from core client for 30 sec - exiting 10:54:19 (3640): No heartbeat from core client for 30 sec - exiting 07:12:02 (524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:03 (524): No heartbeat from core client for 30 sec - exiting 07:12:04 (524): No heartbeat from core client for 30 sec - exiting cpdnmonitor: cannot open input file D:\BOINC/projects/climateprediction.net/famous_u8d0_1799_200_006643679/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC/projects/climateprediction.net/famous_u8d0_1799_200_006643679/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC/projects/climateprediction.net/famous_u8d0_1799_200_006643679/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC/projects/climateprediction.net/famous_u8d0_1799_200_006643679/dataout/ocean_restart.day after 11 attempts 07:13:07 (1800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:13:08 (1800): No heartbeat from core client for 30 sec - exiting 07:13:09 (1800): No heartbeat from core client for 30 sec - exiting 07:13:10 (1800): No heartbeat from core client for 30 sec - exiting cpdnmonitor: cannot open input file D:\BOINC/projects/climateprediction.net/famous_u8d0_1799_200_006643679/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC/projects/climateprediction.net/famous_u8d0_1799_200_006643679/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC/projects/climateprediction.net/famous_u8d0_1799_200_006643679/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy Sorry, too many model crashes! :-( 07:13:56 (3676): called boinc_finish </stderr_txt> ]]> |
| Latest Trickles Received | ||||||
|---|---|---|---|---|---|---|
| Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
| 12 Aug 2010 04:29:16 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,722,266 | 659,554 | 0.3830 |
| 12 Aug 2010 03:49:43 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,712,906 | 655,805 | 0.3829 |
| 12 Aug 2010 02:53:42 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,703,546 | 652,057 | 0.3828 |
| 12 Aug 2010 02:49:43 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,694,186 | 648,317 | 0.3827 |
| 12 Aug 2010 02:49:43 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,684,826 | 644,572 | 0.3826 |
| 12 Aug 2010 02:49:43 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,675,466 | 640,830 | 0.3825 |
| 11 Aug 2010 22:06:08 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,666,106 | 637,211 | 0.3825 |
| 11 Aug 2010 21:43:28 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,656,746 | 633,447 | 0.3823 |
| 11 Aug 2010 20:01:22 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,647,386 | 629,675 | 0.3822 |
| 11 Aug 2010 18:53:11 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,638,026 | 625,914 | 0.3821 |
| 11 Aug 2010 17:49:30 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,628,666 | 622,174 | 0.3820 |
| 11 Aug 2010 16:50:22 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,619,306 | 618,426 | 0.3819 |
| 11 Aug 2010 15:41:31 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,609,946 | 614,677 | 0.3818 |
| 11 Aug 2010 14:37:10 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,600,586 | 610,951 | 0.3817 |
| 11 Aug 2010 13:33:14 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,591,226 | 607,226 | 0.3816 |
| 11 Aug 2010 12:28:20 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,581,866 | 603,450 | 0.3815 |
| 11 Aug 2010 11:23:51 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,572,506 | 599,669 | 0.3813 |
| 11 Aug 2010 10:20:04 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,563,146 | 595,904 | 0.3812 |
| 11 Aug 2010 09:20:15 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,553,786 | 592,108 | 0.3811 |
| 11 Aug 2010 08:12:02 | 1090890 | 11469219 | famous_u8d0_1799_200_006643679_3 | 1,544,426 | 588,401 | 0.3810 |
©2024 climateprediction.net