Name | hadcm3n_ybg9_1900_40_007523912_1 |
Workunit | 7721387 |
Created | 28 Oct 2011, 13:24:41 UTC |
Sent | 31 Oct 2011, 5:45:49 UTC |
Report deadline | 30 Jan 2012, 13:13:00 UTC |
Received | 5 Dec 2011, 23:19:49 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 22 (0x00000016) Unknown error code |
Computer ID | 1134605 |
Run time | 20 days 20 hours 46 min 31 sec |
CPU time | 20 days 16 hours 0 min 52 sec |
Validate state | Invalid |
Credit | 11,197.44 |
Device peak FLOPS | 2.79 GFLOPS |
Application version | UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86 |
Stderr | <core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:25:44 (10924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:27:36 (11140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:28:24 (8524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:32:16 (8524): No heartbeat from core client for 30 sec - exiting 12:32:17 (8524): No heartbeat from core client for 30 sec - exiting 12:32:18 (8524): No heartbeat from core client for 30 sec - exiting 12:32:19 (8524): No heartbeat from core client for 30 sec - exiting 12:32:20 (8524): No heartbeat from core client for 30 sec - exiting 12:32:21 (8524): No heartbeat from core client for 30 sec - exiting 12:32:22 (8524): No heartbeat from core client for 30 sec - exiting 12:32:23 (8524): No heartbeat from core client for 30 sec - exiting 12:32:24 (8524): No heartbeat from core client for 30 sec - exiting 12:32:25 (8524): No heartbeat from core client for 30 sec - exiting 12:34:04 (10312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:35:07 (2264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:39:11 (2264): No heartbeat from core client for 30 sec - exiting 12:39:12 (2264): No heartbeat from core client for 30 sec - exiting 12:39:13 (2264): No heartbeat from core client for 30 sec - exiting 12:39:14 (2264): No heartbeat from core client for 30 sec - exiting 12:39:15 (2264): No heartbeat from core client for 30 sec - exiting 12:39:16 (2264): No heartbeat from core client for 30 sec - exiting 12:39:17 (2264): No heartbeat from core client for 30 sec - exiting 12:39:18 (2264): No heartbeat from core client for 30 sec - exiting 12:39:19 (2264): No heartbeat from core client for 30 sec - exiting 12:39:20 (2264): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:03:46 (9960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:03:53 (9960): No heartbeat from core client for 30 sec - exiting 15:03:54 (9960): No heartbeat from core client for 30 sec - exiting 15:03:55 (9960): No heartbeat from core client for 30 sec - exiting 15:03:56 (9960): No heartbeat from core client for 30 sec - exiting 15:03:57 (9960): No heartbeat from core client for 30 sec - exiting 15:03:58 (9960): No heartbeat from core client for 30 sec - exiting 15:03:59 (9960): No heartbeat from core client for 30 sec - exiting 15:05:52 (12656): No heartbeat from core client for 30 sec - exiting 15:05:53 (12656): No heartbeat from core client for 30 sec - exiting 15:05:54 (12656): No heartbeat from core client for 30 sec - exiting 15:05:55 (12656): No heartbeat from core client for 30 sec - exiting 15:05:56 (12656): No heartbeat from core client for 30 sec - exiting 15:05:57 (12656): No heartbeat from core client for 30 sec - exiting 15:05:58 (12656): No heartbeat from core client for 30 sec - exiting 15:05:59 (12656): No heartbeat from core client for 30 sec - exiting 15:06:00 (12656): No heartbeat from core client for 30 sec - exiting 15:06:01 (12656): No heartbeat from core client for 30 sec - exiting 15:06:02 (12656): No heartbeat from core client for 30 sec - exiting 15:06:03 (12656): No heartbeat from core client for 30 sec - exiting 15:06:04 (12656): No heartbeat from core client for 30 sec - exiting 15:06:05 (12656): No heartbeat from core client for 30 sec - exiting 15:06:06 (12656): No heartbeat from core client for 30 sec - exiting 15:06:07 (12656): No heartbeat from core client for 30 sec - exiting 15:06:08 (12656): No heartbeat from core client for 30 sec - exiting 15:06:09 (12656): No heartbeat from core client for 30 sec - exiting 15:06:10 (12656): No heartbeat from core client for 30 sec - exiting 15:06:11 (12656): No heartbeat from core client for 30 sec - exiting 15:06:12 (12656): No heartbeat from core client for 30 sec - exiting 15:06:13 (12656): No heartbeat from core client for 30 sec - exiting 15:06:14 (12656): No heartbeat from core client for 30 sec - exiting 15:06:15 (12656): No heartbeat from core client for 30 sec - exiting 15:06:16 (12656): No heartbeat from core client for 30 sec - exiting 15:06:17 (12656): No heartbeat from core client for 30 sec - exiting 15:06:18 (12656): No heartbeat from core client for 30 sec - exiting 15:06:19 (12656): No heartbeat from core client for 30 sec - exiting 15:06:20 (12656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:06:21 (12656): No heartbeat from core client for 30 sec - exiting 15:06:22 (12656): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7804, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ybg9ko.pja4c10 Error converting file to netcdf: dataout/ybg9ko.pia4c10 Error converting file to netcdf: dataout/ybg9ko.pfa4c10 Error converting file to netcdf: dataout/ybg9ka.pha4c10 Error converting file to netcdf: dataout/ybg9ka.pga4c10 Error converting file to netcdf: dataout/ybg9ka.pea4c10 Error converting file to netcdf: dataout/ybg9ka.pda4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:36:12 (6332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:36:56 (6712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:08:38 (1796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:08:41 (1796): No heartbeat from core client for 30 sec - exiting 19:08:42 (1796): No heartbeat from core client for 30 sec - exiting 19:08:43 (1796): No heartbeat from core client for 30 sec - exiting 19:08:44 (1796): No heartbeat from core client for 30 sec - exiting 19:08:45 (1796): No heartbeat from core client for 30 sec - exiting 19:08:46 (1796): No heartbeat from core client for 30 sec - exiting 19:08:47 (1796): No heartbeat from core client for 30 sec - exiting 19:08:48 (1796): No heartbeat from core client for 30 sec - exiting 19:08:49 (1796): No heartbeat from core client for 30 sec - exiting 19:08:50 (1796): No heartbeat from core client for 30 sec - exiting 19:18:44 (9284): No heartbeat from core client for 30 sec - exiting 19:18:45 (9284): No heartbeat from core client for 30 sec - exiting 19:18:46 (9284): No heartbeat from core client for 30 sec - exiting 19:18:47 (9284): No heartbeat from core client for 30 sec - exiting 19:18:48 (9284): No heartbeat from core client for 30 sec - exiting 19:18:49 (9284): No heartbeat from core client for 30 sec - exiting 19:18:50 (9284): No heartbeat from core client for 30 sec - exiting 19:18:51 (9284): No heartbeat from core client for 30 sec - exiting 19:18:52 (9284): No heartbeat from core client for 30 sec - exiting 19:18:53 (9284): No heartbeat from core client for 30 sec - exiting 19:18:54 (9284): No heartbeat from core client for 30 sec - exiting 19:18:55 (9284): No heartbeat from core client for 30 sec - exiting 19:18:56 (9284): No heartbeat from core client for 30 sec - exiting 19:18:57 (9284): No heartbeat from core client for 30 sec - exiting 19:18:58 (9284): No heartbeat from core client for 30 sec - exiting 19:18:59 (9284): No heartbeat from core client for 30 sec - exiting 19:19:00 (9284): No heartbeat from core client for 30 sec - exiting 19:19:01 (9284): No heartbeat from core client for 30 sec - exiting 19:19:02 (9284): No heartbeat from core client for 30 sec - exiting 19:19:03 (9284): No heartbeat from core client for 30 sec - exiting 19:19:04 (9284): No heartbeat from core client for 30 sec - exiting 19:19:05 (9284): No heartbeat from core client for 30 sec - exiting 19:19:06 (9284): No heartbeat from core client for 30 sec - exiting 19:19:07 (9284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:22:33 (1476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:24:22 (4120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:26:20 (6048): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 12:38:45 (4664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:38:48 (4664): No heartbeat from core client for 30 sec - exiting 12:38:49 (4664): No heartbeat from core client for 30 sec - exiting 12:38:50 (4664): No heartbeat from core client for 30 sec - exiting 12:38:51 (4664): No heartbeat from core client for 30 sec - exiting 12:38:52 (4664): No heartbeat from core client for 30 sec - exiting 12:38:53 (4664): No heartbeat from core client for 30 sec - exiting 12:38:54 (4664): No heartbeat from core client for 30 sec - exiting 12:38:55 (4664): No heartbeat from core client for 30 sec - exiting 12:38:56 (4664): No heartbeat from core client for 30 sec - exiting 12:38:57 (4664): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 12:54:42 (8528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:54:43 (8528): No heartbeat from core client for 30 sec - exiting 12:54:44 (8528): No heartbeat from core client for 30 sec - exiting 12:56:33 (8236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:45:31 (8788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:45:36 (8788): No heartbeat from core client for 30 sec - exiting 13:45:37 (8788): No heartbeat from core client for 30 sec - exiting 13:45:38 (8788): No heartbeat from core client for 30 sec - exiting 13:45:39 (8788): No heartbeat from core client for 30 sec - exiting 13:47:58 (4264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:48:47 (8204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:46:40 (9392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:46:53 (9392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1880, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
05 Dec 2011 06:40:27 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 933,120 | 1,777,975 | 1.9054 |
04 Dec 2011 03:13:38 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 907,200 | 1,727,286 | 1.9040 |
03 Dec 2011 14:07:30 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 881,280 | 1,678,180 | 1.9043 |
02 Dec 2011 23:18:46 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 855,360 | 1,628,372 | 1.9037 |
02 Dec 2011 08:11:51 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 829,440 | 1,579,652 | 1.9045 |
01 Dec 2011 09:46:34 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 803,520 | 1,529,735 | 1.9038 |
30 Nov 2011 15:40:22 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 777,600 | 1,481,782 | 1.9056 |
30 Nov 2011 01:04:12 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 751,680 | 1,433,088 | 1.9065 |
29 Nov 2011 11:16:03 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 725,760 | 1,383,492 | 1.9063 |
28 Nov 2011 20:56:41 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 699,840 | 1,332,480 | 1.9040 |
28 Nov 2011 07:05:06 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 673,920 | 1,280,727 | 1.9004 |
27 Nov 2011 15:57:36 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 648,000 | 1,229,285 | 1.8970 |
27 Nov 2011 02:03:42 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 622,080 | 1,179,709 | 1.8964 |
26 Nov 2011 12:20:28 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 596,160 | 1,130,388 | 1.8961 |
25 Nov 2011 22:21:41 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 570,240 | 1,080,467 | 1.8948 |
25 Nov 2011 08:51:00 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 544,320 | 1,031,778 | 1.8955 |
24 Nov 2011 19:25:57 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 518,400 | 982,698 | 1.8956 |
24 Nov 2011 05:15:40 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 492,480 | 933,436 | 1.8954 |
23 Nov 2011 16:39:10 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 466,560 | 884,973 | 1.8968 |
23 Nov 2011 01:00:41 | 1134605 | 13553524 | hadcm3n_ybg9_1900_40_007523912_1 | 440,640 | 836,549 | 1.8985 |
©2024 cpdn.org