 |
|
| Name | hadcm3n_o4sv_2140_40_008270002_3 |
| Workunit | 8425126 |
| Created | 13 Feb 2013, 11:04:54 UTC |
| Sent | 13 Feb 2013, 11:05:06 UTC |
| Report deadline | 15 May 2013, 18:32:17 UTC |
| Received | 3 May 2013, 13:37:11 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 22 (0x00000016) Unknown error code |
| Computer ID | 456198 |
| Run time | 22 days 4 hours 10 min 57 sec |
| CPU time | 22 days 4 hours 10 min 57 sec |
| Validate state | Invalid |
| Credit | 9,331.20 |
| Device peak FLOPS | 1.53 GFLOPS |
| Application version | UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86 |
| Stderr | <core_client_version>6.2.19</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 11:21:10 (5124): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:46:50 (4432): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:52:47 (5488): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 22:08:45 (5924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:02 (9364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:03 (9364): No heartbeat from core client for 30 sec - exiting 22:10:04 (9364): No heartbeat from core client for 30 sec - exiting 22:10:05 (9364): No heartbeat from core client for 30 sec - exiting 22:10:06 (9364): No heartbeat from core client for 30 sec - exiting 22:10:07 (9364): No heartbeat from core client for 30 sec - exiting 22:10:08 (9364): No heartbeat from core client for 30 sec - exiting 22:10:09 (9364): No heartbeat from core client for 30 sec - exiting 22:10:10 (9364): No heartbeat from core client for 30 sec - exiting 22:10:11 (9364): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 23:28:14 (1740): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:32:03 (7176): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:42:21 (6104): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:52:12 (5364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:53:11 (8520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:53:12 (8520): No heartbeat from core client for 30 sec - exiting 04:09:03 (4024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:30:59 (9552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:34:09 (7308): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 21:38:41 (3344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:39:30 (8856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:49:27 (6640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:49:28 (6640): No heartbeat from core client for 30 sec - exiting 22:50:37 (6036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:49:01 (4224): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 08:49:03 (4224): No heartbeat from core client for 30 sec - exiting 05:22:14 (5844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:22:48 (9244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:54:17 (8476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:10:59 (8188): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 13:40:45 (9360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:40:46 (9360): No heartbeat from core client for 30 sec - exiting 13:40:47 (9360): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 19:03:15 (9852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:03:16 (9852): No heartbeat from core client for 30 sec - exiting 19:03:17 (9852): No heartbeat from core client for 30 sec - exiting 19:03:18 (9852): No heartbeat from core client for 30 sec - exiting 19:03:19 (9852): No heartbeat from core client for 30 sec - exiting 19:04:29 (1752): No heartbeat from core client for 30 sec - exiting 19:04:30 (1752): No heartbeat from core client for 30 sec - exiting 19:04:31 (1752): No heartbeat from core client for 30 sec - exiting 19:04:32 (1752): No heartbeat from core client for 30 sec - exiting 19:04:33 (1752): No heartbeat from core client for 30 sec - exiting 19:04:34 (1752): No heartbeat from core client for 30 sec - exiting 19:04:35 (1752): No heartbeat from core client for 30 sec - exiting 19:04:36 (1752): No heartbeat from core client for 30 sec - exiting 19:04:37 (1752): No heartbeat from core client for 30 sec - exiting 19:04:38 (1752): No heartbeat from core client for 30 sec - exiting 19:04:39 (1752): No heartbeat from core client for 30 sec - exiting 19:04:40 (1752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:06:05 (4680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:06:06 (4680): No heartbeat from core client for 30 sec - exiting 19:06:07 (4680): No heartbeat from core client for 30 sec - exiting 19:06:08 (4680): No heartbeat from core client for 30 sec - exiting 19:06:09 (4680): No heartbeat from core client for 30 sec - exiting 19:06:10 (4680): No heartbeat from core client for 30 sec - exiting 19:06:11 (4680): No heartbeat from core client for 30 sec - exiting 19:06:12 (4680): No heartbeat from core client for 30 sec - exiting 19:07:16 (2524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:37:03 (8652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:00:33 (9064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:01:21 (5912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:16:03 (2288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:26:38 (6924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:37:23 (7864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:19:23 (7244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:20:21 (8184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:24:33 (7408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:25:33 (5072): No heartbeat from core client for 30 sec - exiting 02:25:35 (5072): No heartbeat from core client for 30 sec - exiting 02:25:36 (5072): No heartbeat from core client for 30 sec - exiting 02:25:37 (5072): No heartbeat from core client for 30 sec - exiting 02:25:38 (5072): No heartbeat from core client for 30 sec - exiting 02:25:39 (5072): No heartbeat from core client for 30 sec - exiting 02:25:40 (5072): No heartbeat from core client for 30 sec - exiting 02:25:41 (5072): No heartbeat from core client for 30 sec - exiting 02:25:42 (5072): No heartbeat from core client for 30 sec - exiting 02:25:43 (5072): No heartbeat from core client for 30 sec - exiting 02:25:44 (5072): No heartbeat from core client for 30 sec - exiting 02:25:45 (5072): No heartbeat from core client for 30 sec - exiting 02:25:46 (5072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:25:47 (5072): No heartbeat from core client for 30 sec - exiting 02:25:48 (5072): No heartbeat from core client for 30 sec - exiting 02:25:49 (5072): No heartbeat from core client for 30 sec - exiting 02:25:50 (5072): No heartbeat from core client for 30 sec - exiting 02:25:51 (5072): No heartbeat from core client for 30 sec - exiting 02:25:52 (5072): No heartbeat from core client for 30 sec - exiting 02:25:53 (5072): No heartbeat from core client for 30 sec - exiting 02:25:54 (5072): No heartbeat from core client for 30 sec - exiting 02:26:47 (8128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:26:48 (8128): No heartbeat from core client for 30 sec - exiting 02:26:49 (8128): No heartbeat from core client for 30 sec - exiting 02:26:50 (8128): No heartbeat from core client for 30 sec - exiting 02:56:35 (8820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:49:17 (9464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:50:10 (5124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:50:11 (5124): No heartbeat from core client for 30 sec - exiting 08:50:12 (5124): No heartbeat from core client for 30 sec - exiting 08:50:13 (5124): No heartbeat from core client for 30 sec - exiting 08:50:14 (5124): No heartbeat from core client for 30 sec - exiting 08:51:03 (5396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:05:13 (516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:05:49 (9748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:05:50 (9748): No heartbeat from core client for 30 sec - exiting 18:05:51 (9748): No heartbeat from core client for 30 sec - exiting 18:05:52 (9748): No heartbeat from core client for 30 sec - exiting 20:18:31 (3624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:49:09 (2612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:22:37 (3488): No heartbeat from core client for 30 sec - exiting 03:22:38 (3488): No heartbeat from core client for 30 sec - exiting 03:22:39 (3488): No heartbeat from core client for 30 sec - exiting 03:22:40 (3488): No heartbeat from core client for 30 sec - exiting 03:22:41 (3488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:22:42 (3488): No heartbeat from core client for 30 sec - exiting BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o4svko.pjy4c10 Error converting file to netcdf: dataout/o4svko.piy4c10 Error converting file to netcdf: dataout/o4svko.pfy4c10 Error converting file to netcdf: dataout/o4svka.phy4c10 Error converting file to netcdf: dataout/o4svka.pgy4c10 Error converting file to netcdf: dataout/o4svka.pey4c10 Error converting file to netcdf: dataout/o4svka.pdy4c10 12:27:03 (3356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:32:25 (4288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:42:32 (5684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:43:32 (5952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:48:14 (5516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:40:45 (7500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:40:46 (7500): No heartbeat from core client for 30 sec - exiting BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:21:35 (2364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:34:33 (5592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 04:39:25 AM No files match the supplied pattern. MainError: 04:39:25 AM No files match the supplied pattern. 07:59:38 (5184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:59:39 (5184): No heartbeat from core client for 30 sec - exiting 08:01:41 (7676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:42 (7676): No heartbeat from core client for 30 sec - exiting 13:11:47 (1292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:12:40 (3044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:12:41 (3044): No heartbeat from core client for 30 sec - exiting 13:12:42 (3044): No heartbeat from core client for 30 sec - exiting MainError: 12:26:00 AM No files match the supplied pattern. MainError: 12:26:00 AM No files match the supplied pattern. MainError: 07:55:56 PM No files match the supplied pattern. MainError: 07:55:56 PM No files match the supplied pattern. MainError: 03:29:19 PM No files match the supplied pattern. MainError: 03:29:19 PM No files match the supplied pattern. 09:05:04 (5344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... MainError: 11:20:52 AM No files match the supplied pattern. MainError: 11:20:52 AM No files match the supplied pattern. MainError: 07:05:56 AM No files match the supplied pattern. MainError: 07:05:56 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 03:01:44 AM No files match the supplied pattern. MainError: 03:01:44 AM No files match the supplied pattern. MainError: 10:44:49 PM No files match the supplied pattern. MainError: 10:44:49 PM No files match the supplied pattern. MainError: 06:55:42 PM No files match the supplied pattern. MainError: 06:55:43 PM No files match the supplied pattern. MainError: 02:33:20 PM No files match the supplied pattern. MainError: 02:33:20 PM No files match the supplied pattern. BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error 18:08:53 (9208): No heartbeat from core client for 30 sec - exiting 18:08:54 (9208): No heartbeat from core client for 30 sec - exiting 18:08:55 (9208): No heartbeat from core client for 30 sec - exiting 18:08:56 (9208): No heartbeat from core client for 30 sec - exiting 18:08:57 (9208): No heartbeat from core client for 30 sec - exiting 18:08:58 (9208): No heartbeat from core client for 30 sec - exiting 18:08:59 (9208): No heartbeat from core client for 30 sec - exiting 18:09:00 (9208): No heartbeat from core client for 30 sec - exiting 18:09:01 (9208): No heartbeat from core client for 30 sec - exiting 18:09:02 (9208): No heartbeat from core client for 30 sec - exiting 18:09:03 (9208): No heartbeat from core client for 30 sec - exiting 18:09:04 (9208): No heartbeat from core client for 30 sec - exiting 18:09:05 (9208): No heartbeat from core client for 30 sec - exiting 18:09:06 (9208): No heartbeat from core client for 30 sec - exiting 18:09:07 (9208): No heartbeat from core client for 30 sec - exiting 18:09:08 (9208): No heartbeat from core client for 30 sec - exiting 18:09:09 (9208): No heartbeat from core client for 30 sec - exiting 18:09:11 (9208): No heartbeat from core client for 30 sec - exiting 18:09:12 (9208): No heartbeat from core client for 30 sec - exiting 18:09:13 (9208): No heartbeat from core client for 30 sec - exiting 18:09:14 (9208): No heartbeat from core client for 30 sec - exiting 18:09:15 (9208): No heartbeat from core client for 30 sec - exiting 18:09:16 (9208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:09:17 (9208): No heartbeat from core client for 30 sec - exiting 18:09:18 (9208): No heartbeat from core client for 30 sec - exiting BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 21:19:42 (524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Error converting file to netcdf: dataout/o4svka.ph11c10 Error converting file to netcdf: dataout/o4svka.pg11c10 Error converting file to netcdf: dataout/o4svka.pe11c10 MainError: 11:48:00 AM No files match the supplied pattern. MainError: 11:48:00 AM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]> |
| Latest Trickles Received | ||||||
|---|---|---|---|---|---|---|
| Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
| 03 May 2013 12:34:50 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 777,600 | 1,972,074 | 2.5361 |
| 01 May 2013 15:51:02 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 751,680 | 1,906,372 | 2.5361 |
| 30 Apr 2013 19:18:21 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 725,760 | 1,840,377 | 2.5358 |
| 29 Apr 2013 22:57:18 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 699,840 | 1,773,823 | 2.5346 |
| 29 Apr 2013 03:11:01 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 673,920 | 1,707,013 | 2.5330 |
| 28 Apr 2013 07:10:40 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 648,000 | 1,640,220 | 2.5312 |
| 27 Apr 2013 11:23:44 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 622,080 | 1,574,146 | 2.5305 |
| 26 Apr 2013 15:31:29 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 596,160 | 1,507,881 | 2.5293 |
| 25 Apr 2013 19:57:11 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 570,240 | 1,442,306 | 2.5293 |
| 25 Apr 2013 00:29:16 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 544,320 | 1,376,907 | 2.5296 |
| 24 Apr 2013 05:39:04 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 518,400 | 1,311,069 | 2.5291 |
| 23 Apr 2013 09:14:13 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 492,480 | 1,246,196 | 2.5304 |
| 22 Apr 2013 13:50:10 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 466,560 | 1,181,705 | 2.5328 |
| 21 Apr 2013 18:28:19 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 440,640 | 1,117,233 | 2.5355 |
| 20 Apr 2013 23:04:07 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 414,720 | 1,052,421 | 2.5377 |
| 20 Apr 2013 03:52:20 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 388,800 | 986,370 | 2.5370 |
| 19 Apr 2013 07:38:14 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 362,880 | 921,140 | 2.5384 |
| 18 Apr 2013 11:47:36 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 336,960 | 854,734 | 2.5366 |
| 17 Apr 2013 16:21:05 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 311,040 | 789,233 | 2.5374 |
| 16 Apr 2013 20:41:50 | 456198 | 15606999 | hadcm3n_o4sv_2140_40_008270002_3 | 285,120 | 723,996 | 2.5393 |
©2024 cpdn.org