Task 13553524

Name	hadcm3n_ybg9_1900_40_007523912_1
Workunit	7721387
Created	28 Oct 2011, 13:24:41 UTC
Sent	31 Oct 2011, 5:45:49 UTC
Report deadline	30 Jan 2012, 13:13:00 UTC
Received	5 Dec 2011, 23:19:49 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1134605
Run time	20 days 20 hours 46 min 31 sec
CPU time	20 days 16 hours 0 min 52 sec
Validate state	Invalid
Credit	11,197.44
Device peak FLOPS	2.79 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:25:44 (10924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:27:36 (11140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:28:24 (8524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:32:16 (8524): No heartbeat from core client for 30 sec - exiting 12:32:17 (8524): No heartbeat from core client for 30 sec - exiting 12:32:18 (8524): No heartbeat from core client for 30 sec - exiting 12:32:19 (8524): No heartbeat from core client for 30 sec - exiting 12:32:20 (8524): No heartbeat from core client for 30 sec - exiting 12:32:21 (8524): No heartbeat from core client for 30 sec - exiting 12:32:22 (8524): No heartbeat from core client for 30 sec - exiting 12:32:23 (8524): No heartbeat from core client for 30 sec - exiting 12:32:24 (8524): No heartbeat from core client for 30 sec - exiting 12:32:25 (8524): No heartbeat from core client for 30 sec - exiting 12:34:04 (10312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:35:07 (2264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:39:11 (2264): No heartbeat from core client for 30 sec - exiting 12:39:12 (2264): No heartbeat from core client for 30 sec - exiting 12:39:13 (2264): No heartbeat from core client for 30 sec - exiting 12:39:14 (2264): No heartbeat from core client for 30 sec - exiting 12:39:15 (2264): No heartbeat from core client for 30 sec - exiting 12:39:16 (2264): No heartbeat from core client for 30 sec - exiting 12:39:17 (2264): No heartbeat from core client for 30 sec - exiting 12:39:18 (2264): No heartbeat from core client for 30 sec - exiting 12:39:19 (2264): No heartbeat from core client for 30 sec - exiting 12:39:20 (2264): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:03:46 (9960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:03:53 (9960): No heartbeat from core client for 30 sec - exiting 15:03:54 (9960): No heartbeat from core client for 30 sec - exiting 15:03:55 (9960): No heartbeat from core client for 30 sec - exiting 15:03:56 (9960): No heartbeat from core client for 30 sec - exiting 15:03:57 (9960): No heartbeat from core client for 30 sec - exiting 15:03:58 (9960): No heartbeat from core client for 30 sec - exiting 15:03:59 (9960): No heartbeat from core client for 30 sec - exiting 15:05:52 (12656): No heartbeat from core client for 30 sec - exiting 15:05:53 (12656): No heartbeat from core client for 30 sec - exiting 15:05:54 (12656): No heartbeat from core client for 30 sec - exiting 15:05:55 (12656): No heartbeat from core client for 30 sec - exiting 15:05:56 (12656): No heartbeat from core client for 30 sec - exiting 15:05:57 (12656): No heartbeat from core client for 30 sec - exiting 15:05:58 (12656): No heartbeat from core client for 30 sec - exiting 15:05:59 (12656): No heartbeat from core client for 30 sec - exiting 15:06:00 (12656): No heartbeat from core client for 30 sec - exiting 15:06:01 (12656): No heartbeat from core client for 30 sec - exiting 15:06:02 (12656): No heartbeat from core client for 30 sec - exiting 15:06:03 (12656): No heartbeat from core client for 30 sec - exiting 15:06:04 (12656): No heartbeat from core client for 30 sec - exiting 15:06:05 (12656): No heartbeat from core client for 30 sec - exiting 15:06:06 (12656): No heartbeat from core client for 30 sec - exiting 15:06:07 (12656): No heartbeat from core client for 30 sec - exiting 15:06:08 (12656): No heartbeat from core client for 30 sec - exiting 15:06:09 (12656): No heartbeat from core client for 30 sec - exiting 15:06:10 (12656): No heartbeat from core client for 30 sec - exiting 15:06:11 (12656): No heartbeat from core client for 30 sec - exiting 15:06:12 (12656): No heartbeat from core client for 30 sec - exiting 15:06:13 (12656): No heartbeat from core client for 30 sec - exiting 15:06:14 (12656): No heartbeat from core client for 30 sec - exiting 15:06:15 (12656): No heartbeat from core client for 30 sec - exiting 15:06:16 (12656): No heartbeat from core client for 30 sec - exiting 15:06:17 (12656): No heartbeat from core client for 30 sec - exiting 15:06:18 (12656): No heartbeat from core client for 30 sec - exiting 15:06:19 (12656): No heartbeat from core client for 30 sec - exiting 15:06:20 (12656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:06:21 (12656): No heartbeat from core client for 30 sec - exiting 15:06:22 (12656): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7804, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ybg9ko.pja4c10 Error converting file to netcdf: dataout/ybg9ko.pia4c10 Error converting file to netcdf: dataout/ybg9ko.pfa4c10 Error converting file to netcdf: dataout/ybg9ka.pha4c10 Error converting file to netcdf: dataout/ybg9ka.pga4c10 Error converting file to netcdf: dataout/ybg9ka.pea4c10 Error converting file to netcdf: dataout/ybg9ka.pda4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:36:12 (6332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:36:56 (6712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:08:38 (1796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:08:41 (1796): No heartbeat from core client for 30 sec - exiting 19:08:42 (1796): No heartbeat from core client for 30 sec - exiting 19:08:43 (1796): No heartbeat from core client for 30 sec - exiting 19:08:44 (1796): No heartbeat from core client for 30 sec - exiting 19:08:45 (1796): No heartbeat from core client for 30 sec - exiting 19:08:46 (1796): No heartbeat from core client for 30 sec - exiting 19:08:47 (1796): No heartbeat from core client for 30 sec - exiting 19:08:48 (1796): No heartbeat from core client for 30 sec - exiting 19:08:49 (1796): No heartbeat from core client for 30 sec - exiting 19:08:50 (1796): No heartbeat from core client for 30 sec - exiting 19:18:44 (9284): No heartbeat from core client for 30 sec - exiting 19:18:45 (9284): No heartbeat from core client for 30 sec - exiting 19:18:46 (9284): No heartbeat from core client for 30 sec - exiting 19:18:47 (9284): No heartbeat from core client for 30 sec - exiting 19:18:48 (9284): No heartbeat from core client for 30 sec - exiting 19:18:49 (9284): No heartbeat from core client for 30 sec - exiting 19:18:50 (9284): No heartbeat from core client for 30 sec - exiting 19:18:51 (9284): No heartbeat from core client for 30 sec - exiting 19:18:52 (9284): No heartbeat from core client for 30 sec - exiting 19:18:53 (9284): No heartbeat from core client for 30 sec - exiting 19:18:54 (9284): No heartbeat from core client for 30 sec - exiting 19:18:55 (9284): No heartbeat from core client for 30 sec - exiting 19:18:56 (9284): No heartbeat from core client for 30 sec - exiting 19:18:57 (9284): No heartbeat from core client for 30 sec - exiting 19:18:58 (9284): No heartbeat from core client for 30 sec - exiting 19:18:59 (9284): No heartbeat from core client for 30 sec - exiting 19:19:00 (9284): No heartbeat from core client for 30 sec - exiting 19:19:01 (9284): No heartbeat from core client for 30 sec - exiting 19:19:02 (9284): No heartbeat from core client for 30 sec - exiting 19:19:03 (9284): No heartbeat from core client for 30 sec - exiting 19:19:04 (9284): No heartbeat from core client for 30 sec - exiting 19:19:05 (9284): No heartbeat from core client for 30 sec - exiting 19:19:06 (9284): No heartbeat from core client for 30 sec - exiting 19:19:07 (9284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:22:33 (1476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:24:22 (4120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:26:20 (6048): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 12:38:45 (4664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:38:48 (4664): No heartbeat from core client for 30 sec - exiting 12:38:49 (4664): No heartbeat from core client for 30 sec - exiting 12:38:50 (4664): No heartbeat from core client for 30 sec - exiting 12:38:51 (4664): No heartbeat from core client for 30 sec - exiting 12:38:52 (4664): No heartbeat from core client for 30 sec - exiting 12:38:53 (4664): No heartbeat from core client for 30 sec - exiting 12:38:54 (4664): No heartbeat from core client for 30 sec - exiting 12:38:55 (4664): No heartbeat from core client for 30 sec - exiting 12:38:56 (4664): No heartbeat from core client for 30 sec - exiting 12:38:57 (4664): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 12:54:42 (8528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:54:43 (8528): No heartbeat from core client for 30 sec - exiting 12:54:44 (8528): No heartbeat from core client for 30 sec - exiting 12:56:33 (8236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:45:31 (8788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:45:36 (8788): No heartbeat from core client for 30 sec - exiting 13:45:37 (8788): No heartbeat from core client for 30 sec - exiting 13:45:38 (8788): No heartbeat from core client for 30 sec - exiting 13:45:39 (8788): No heartbeat from core client for 30 sec - exiting 13:47:58 (4264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:48:47 (8204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:46:40 (9392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:46:53 (9392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1880, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Dec 2011 06:40:27	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	933,120	1,777,975	1.9054
04 Dec 2011 03:13:38	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	907,200	1,727,286	1.9040
03 Dec 2011 14:07:30	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	881,280	1,678,180	1.9043
02 Dec 2011 23:18:46	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	855,360	1,628,372	1.9037
02 Dec 2011 08:11:51	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	829,440	1,579,652	1.9045
01 Dec 2011 09:46:34	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	803,520	1,529,735	1.9038
30 Nov 2011 15:40:22	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	777,600	1,481,782	1.9056
30 Nov 2011 01:04:12	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	751,680	1,433,088	1.9065
29 Nov 2011 11:16:03	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	725,760	1,383,492	1.9063
28 Nov 2011 20:56:41	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	699,840	1,332,480	1.9040
28 Nov 2011 07:05:06	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	673,920	1,280,727	1.9004
27 Nov 2011 15:57:36	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	648,000	1,229,285	1.8970
27 Nov 2011 02:03:42	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	622,080	1,179,709	1.8964
26 Nov 2011 12:20:28	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	596,160	1,130,388	1.8961
25 Nov 2011 22:21:41	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	570,240	1,080,467	1.8948
25 Nov 2011 08:51:00	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	544,320	1,031,778	1.8955
24 Nov 2011 19:25:57	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	518,400	982,698	1.8956
24 Nov 2011 05:15:40	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	492,480	933,436	1.8954
23 Nov 2011 16:39:10	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	466,560	884,973	1.8968
23 Nov 2011 01:00:41	1134605	13553524	hadcm3n_ybg9_1900_40_007523912_1	440,640	836,549	1.8985