Task 13095772

Name	hadcm3n_ya3r_1900_40_007345953_1
Workunit	7543383
Created	6 Jul 2011, 13:34:36 UTC
Sent	20 Jul 2011, 1:08:18 UTC
Report deadline	19 Oct 2011, 8:35:29 UTC
Received	13 Oct 2011, 2:33:59 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1144915
Run time	18 days 21 hours 13 min 25 sec
CPU time	18 days 2 hours 18 min 45 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.78 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:00:32 (4868): No heartbeat from core client for 30 sec - exiting 16:00:33 (4868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on ya3rko.daa19m0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2728, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:58:23 (1128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=408, iMonCtr=1 Model crash detected, will try to restart... 13:29:01 (5252): No heartbeat from core client for 30 sec - exiting 13:29:02 (5252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:17:55 (7188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:28:53 (6468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:02:44 (3164): No heartbeat from core client for 30 sec - exiting 13:02:45 (3164): No heartbeat from core client for 30 sec - exiting 13:02:46 (3164): No heartbeat from core client for 30 sec - exiting 13:02:47 (3164): No heartbeat from core client for 30 sec - exiting 13:02:48 (3164): No heartbeat from core client for 30 sec - exiting 13:02:49 (3164): No heartbeat from core client for 30 sec - exiting 13:02:50 (3164): No heartbeat from core client for 30 sec - exiting 13:02:51 (3164): No heartbeat from core client for 30 sec - exiting 13:02:52 (3164): No heartbeat from core client for 30 sec - exiting 13:02:53 (3164): No heartbeat from core client for 30 sec - exiting 13:02:54 (3164): No heartbeat from core client for 30 sec - exiting 13:02:55 (3164): No heartbeat from core client for 30 sec - exiting 13:02:56 (3164): No heartbeat from core client for 30 sec - exiting 13:02:57 (3164): No heartbeat from core client for 30 sec - exiting 13:02:58 (3164): No heartbeat from core client for 30 sec - exiting 13:02:59 (3164): No heartbeat from core client for 30 sec - exiting 13:03:00 (3164): No heartbeat from core client for 30 sec - exiting 13:03:01 (3164): No heartbeat from core client for 30 sec - exiting 13:03:02 (3164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:03:49 (1888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11844, iMonCtr=1 Model crash detected, will try to restart... 14:58:42 (5980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:37:22 (5964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:31:23 (4808): No heartbeat from core client for 30 sec - exiting 12:31:25 (4808): No heartbeat from core client for 30 sec - exiting 12:31:26 (4808): No heartbeat from core client for 30 sec - exiting 12:31:27 (4808): No heartbeat from core client for 30 sec - exiting 12:31:28 (4808): No heartbeat from core client for 30 sec - exiting 12:31:29 (4808): No heartbeat from core client for 30 sec - exiting 12:31:30 (4808): No heartbeat from core client for 30 sec - exiting 12:31:31 (4808): No heartbeat from core client for 30 sec - exiting 12:31:32 (4808): No heartbeat from core client for 30 sec - exiting 12:31:33 (4808): No heartbeat from core client for 30 sec - exiting 12:31:34 (4808): No heartbeat from core client for 30 sec - exiting 12:31:35 (4808): No heartbeat from core client for 30 sec - exiting 12:31:36 (4808): No heartbeat from core client for 30 sec - exiting 12:31:37 (4808): No heartbeat from core client for 30 sec - exiting 12:31:38 (4808): No heartbeat from core client for 30 sec - exiting 12:31:39 (4808): No heartbeat from core client for 30 sec - exiting 12:31:40 (4808): No heartbeat from core client for 30 sec - exiting 12:31:41 (4808): No heartbeat from core client for 30 sec - exiting 12:31:42 (4808): No heartbeat from core client for 30 sec - exiting 12:31:43 (4808): No heartbeat from core client for 30 sec - exiting 12:31:44 (4808): No heartbeat from core client for 30 sec - exiting 12:31:45 (4808): No heartbeat from core client for 30 sec - exiting 12:31:46 (4808): No heartbeat from core client for 30 sec - exiting 12:31:47 (4808): No heartbeat from core client for 30 sec - exiting 12:31:48 (4808): No heartbeat from core client for 30 sec - exiting 12:31:49 (4808): No heartbeat from core client for 30 sec - exiting 12:31:50 (4808): No heartbeat from core client for 30 sec - exiting 12:31:51 (4808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:43:38 (5160): No heartbeat from core client for 30 sec - exiting 16:43:39 (5160): No heartbeat from core client for 30 sec - exiting 16:43:40 (5160): No heartbeat from core client for 30 sec - exiting 16:43:41 (5160): No heartbeat from core client for 30 sec - exiting 16:43:42 (5160): No heartbeat from core client for 30 sec - exiting 16:43:43 (5160): No heartbeat from core client for 30 sec - exiting 16:43:45 (5160): No heartbeat from core client for 30 sec - exiting 16:43:46 (5160): No heartbeat from core client for 30 sec - exiting 16:43:47 (5160): No heartbeat from core client for 30 sec - exiting 16:43:48 (5160): No heartbeat from core client for 30 sec - exiting 16:43:49 (5160): No heartbeat from core client for 30 sec - exiting 16:43:50 (5160): No heartbeat from core client for 30 sec - exiting 16:43:51 (5160): No heartbeat from core client for 30 sec - exiting 16:43:52 (5160): No heartbeat from core client for 30 sec - exiting 16:43:53 (5160): No heartbeat from core client for 30 sec - exiting 16:43:54 (5160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1248, iMonCtr=1 Model crash detected, will try to restart... 08:08:01 (5188): No heartbeat from core client for 30 sec - exiting 08:08:02 (5188): No heartbeat from core client for 30 sec - exiting 08:08:03 (5188): No heartbeat from core client for 30 sec - exiting 08:08:04 (5188): No heartbeat from core client for 30 sec - exiting 08:08:05 (5188): No heartbeat from core client for 30 sec - exiting 08:08:06 (5188): No heartbeat from core client for 30 sec - exiting 08:08:07 (5188): No heartbeat from core client for 30 sec - exiting 08:08:08 (5188): No heartbeat from core client for 30 sec - exiting 08:08:09 (5188): No heartbeat from core client for 30 sec - exiting 08:08:10 (5188): No heartbeat from core client for 30 sec - exiting 08:08:11 (5188): No heartbeat from core client for 30 sec - exiting 08:08:12 (5188): No heartbeat from core client for 30 sec - exiting 08:08:13 (5188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2768, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:00:19 (7988): No heartbeat from core client for 30 sec - exiting 16:00:21 (7988): No heartbeat from core client for 30 sec - exiting 16:00:22 (7988): No heartbeat from core client for 30 sec - exiting 16:00:23 (7988): No heartbeat from core client for 30 sec - exiting 16:00:24 (7988): No heartbeat from core client for 30 sec - exiting 16:00:25 (7988): No heartbeat from core client for 30 sec - exiting 16:00:26 (7988): No heartbeat from core client for 30 sec - exiting 16:00:27 (7988): No heartbeat from core client for 30 sec - exiting 16:00:28 (7988): No heartbeat from core client for 30 sec - exiting 16:00:29 (7988): No heartbeat from core client for 30 sec - exiting 16:00:30 (7988): No heartbeat from core client for 30 sec - exiting 16:00:31 (7988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:04:36 (7308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:17:36 (1444): No heartbeat from core client for 30 sec - exiting 09:17:37 (1444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4436, iMonCtr=1 Model crash detected, will try to restart... 12:22:34 (4680): No heartbeat from core client for 30 sec - exiting 12:22:35 (4680): No heartbeat from core client for 30 sec - exiting 12:22:36 (4680): No heartbeat from core client for 30 sec - exiting 12:22:38 (4680): No heartbeat from core client for 30 sec - exiting 12:22:39 (4680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:05:40 (5152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:16:18 (3844): No heartbeat from core client for 30 sec - exiting 13:16:19 (3844): No heartbeat from core client for 30 sec - exiting 13:16:20 (3844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:00:32 (2392): No heartbeat from core client for 30 sec - exiting 16:00:33 (2392): No heartbeat from core client for 30 sec - exiting 16:00:34 (2392): No heartbeat from core client for 30 sec - exiting 16:00:35 (2392): No heartbeat from core client for 30 sec - exiting 16:00:36 (2392): No heartbeat from core client for 30 sec - exiting 16:00:37 (2392): No heartbeat from core client for 30 sec - exiting 16:00:38 (2392): No heartbeat from core client for 30 sec - exiting 16:00:39 (2392): No heartbeat from core client for 30 sec - exiting 16:00:40 (2392): No heartbeat from core client for 30 sec - exiting 16:00:41 (2392): No heartbeat from core client for 30 sec - exiting 16:00:42 (2392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:52:44 (5360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5512, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:54:55 (8608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:00:35 (2532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:01:14 (8044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:08:45 (3756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:08:46 (3756): No heartbeat from core client for 30 sec - exiting 16:08:47 (3756): No heartbeat from core client for 30 sec - exiting 16:08:48 (3756): No heartbeat from core client for 30 sec - exiting 16:09:36 (6528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:54:25 (3144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:59:31 (5852): No heartbeat from core client for 30 sec - exiting 07:59:33 (5852): No heartbeat from core client for 30 sec - exiting 07:59:34 (5852): No heartbeat from core client for 30 sec - exiting 07:59:35 (5852): No heartbeat from core client for 30 sec - exiting 07:59:36 (5852): No heartbeat from core client for 30 sec - exiting 07:59:37 (5852): No heartbeat from core client for 30 sec - exiting 07:59:38 (5852): No heartbeat from core client for 30 sec - exiting 07:59:39 (5852): No heartbeat from core client for 30 sec - exiting 07:59:40 (5852): No heartbeat from core client for 30 sec - exiting 07:59:41 (5852): No heartbeat from core client for 30 sec - exiting 07:59:42 (5852): No heartbeat from core client for 30 sec - exiting 07:59:43 (5852): No heartbeat from core client for 30 sec - exiting 07:59:44 (5852): No heartbeat from core client for 30 sec - exiting 07:59:45 (5852): No heartbeat from core client for 30 sec - exiting 07:59:46 (5852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:00:36 (5540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:00:31 (6744): No heartbeat from core client for 30 sec - exiting 16:00:33 (6744): No heartbeat from core client for 30 sec - exiting 16:00:34 (6744): No heartbeat from core client for 30 sec - exiting 16:00:35 (6744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:09:44 (5032): No heartbeat from core client for 30 sec - exiting 16:09:45 (5032): No heartbeat from core client for 30 sec - exiting 16:09:46 (5032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ya3rko.pjc3c10 Error converting file to netcdf: dataout/ya3rko.pic3c10 Error converting file to netcdf: dataout/ya3rko.pfc3c10 Error converting file to netcdf: dataout/ya3rka.phc3c10 Error converting file to netcdf: dataout/ya3rka.pgc3c10 Error converting file to netcdf: dataout/ya3rka.pec3c10 Error converting file to netcdf: dataout/ya3rka.pdc3c10 CPDN Monitor - Quit request from BOINC... 16:40:05 (6064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:04:35 (7208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:51:13 (6108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:33:35 (9216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:44:27 (2676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:18:50 (1396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:41:15 (4488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:19:31 (5096): No heartbeat from core client for 30 sec - exiting 14:19:32 (5096): No heartbeat from core client for 30 sec - exiting 14:19:33 (5096): No heartbeat from core client for 30 sec - exiting 14:19:35 (5096): No heartbeat from core client for 30 sec - exiting 14:19:36 (5096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:25:20 (5192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:25:21 (5192): No heartbeat from core client for 30 sec - exiting 05:21:47 (8388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:00:31 (4156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:01:05 (1160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:18:22 (5576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:17:15 (7244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:17:17 (7244): No heartbeat from core client for 30 sec - exiting 19:17:18 (7244): No heartbeat from core client for 30 sec - exiting Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Oct 2011 18:56:59	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	777,600	1,563,520	2.0107
11 Oct 2011 20:17:56	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	751,680	1,515,259	2.0158
05 Oct 2011 21:42:43	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	725,760	1,467,118	2.0215
03 Oct 2011 23:22:57	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	699,840	1,419,736	2.0287
29 Sep 2011 01:47:19	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	673,920	1,371,144	2.0346
28 Sep 2011 07:03:45	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	648,000	1,321,643	2.0396
27 Sep 2011 04:25:13	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	622,080	1,267,942	2.0382
26 Sep 2011 07:49:10	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	596,160	1,219,527	2.0456
22 Sep 2011 05:35:45	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	570,240	1,161,624	2.0371
21 Sep 2011 15:37:07	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	544,320	1,112,093	2.0431
20 Sep 2011 17:57:29	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	518,400	1,064,319	2.0531
06 Sep 2011 03:36:48	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	492,480	1,012,503	2.0559
03 Sep 2011 11:34:08	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	466,560	957,376	2.0520
31 Aug 2011 01:20:21	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	440,640	902,793	2.0488
30 Aug 2011 07:24:59	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	414,720	846,966	2.0423
28 Aug 2011 07:24:54	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	388,800	792,743	2.0389
24 Aug 2011 22:53:25	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	362,880	741,887	2.0444
24 Aug 2011 00:30:56	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	336,960	691,141	2.0511
23 Aug 2011 02:06:50	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	311,040	640,825	2.0603
22 Aug 2011 08:39:27	1144915	13095772	hadcm3n_ya3r_1900_40_007345953_1	285,120	589,405	2.0672