Task 13345602

Name	hadcm3n_p5dz_1940_40_007421061_2
Workunit	7618696
Created	8 Sep 2011, 10:38:25 UTC
Sent	8 Sep 2011, 10:38:46 UTC
Report deadline	8 Dec 2011, 18:05:57 UTC
Received	9 Nov 2011, 23:34:01 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1068723
Run time	44 days 2 hours 55 min 22 sec
CPU time	34 days 0 hours 28 min 19 sec
Validate state	Invalid
Credit	7,464.96
Device peak FLOPS	0.65 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (16 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77cc400] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8361715] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x830801f] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x828d3b0] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8090808] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x808a417] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8299901] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x821c545] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8219da1] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838720b] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83995c6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7625bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23763, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (12 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf76f7400] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8280180] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83350c5] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x821d177] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8219da1] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838720b] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83995c6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7550bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2601, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (16 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7706400] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8361715] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x830801f] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x828d3b0] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8090808] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x808a417] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8299901] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x821c545] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8219da1] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838720b] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83995c6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf755fbd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2601, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGILL: illegal instruction Stack trace (17 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7710400] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8063638] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x81ca51e] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8309a29] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x828d3b0] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8090808] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x808a417] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8299901] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x821c545] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8219da1] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838720b] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83995c6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7569bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2477, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGILL: illegal instruction Stack trace (16 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf772c400] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x81bed4a] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x830da36] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x828d3b0] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8090808] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x808a417] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8299901] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x821c545] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8219da1] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838720b] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83995c6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7585bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2855, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:22:05 (31587): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:13:01 (1586): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:15:57 (5431): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:19:01 (5528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:34:17 (5565): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:34:36 (5565): No heartbeat from core client for 30 sec - exiting 04:34:37 (5565): No heartbeat from core client for 30 sec - exiting 04:34:38 (5565): No heartbeat from core client for 30 sec - exiting 04:39:00 (5844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:40:46 (5934): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:44:45 (5996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:48:01 (6017): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:49:40 (6127): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:50:54 (6147): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:53:44 (6192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:13:23 (6231): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:14:27 (6231): No heartbeat from core client for 30 sec - exiting 06:14:28 (6231): No heartbeat from core client for 30 sec - exiting 06:14:29 (6231): No heartbeat from core client for 30 sec - exiting 06:22:44 (7837): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:27:44 (7916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:31:15 (8029): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:36:47 (8150): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:49:22 (8230): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:54:46 (8529): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:07:19 (8646): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:11:15 (8858): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:14:35 (8928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:15:24 (8928): No heartbeat from core client for 30 sec - exiting 07:16:07 (9045): No heartbeat from core client for 30 sec - exiting 07:16:15 (9045): No heartbeat from core client for 30 sec - exiting 07:16:16 (9045): No heartbeat from core client for 30 sec - exiting 07:16:17 (9045): No heartbeat from core client for 30 sec - exiting 07:16:18 (9045): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:21:15 (9057): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:22:34 (9140): No heartbeat from core client for 30 sec - exiting 07:23:50 (9140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:31:08 (9235): No heartbeat from core client for 30 sec - exiting 07:31:10 (9235): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:31:11 (9235): No heartbeat from core client for 30 sec - exiting 07:42:54 (9345): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:43:58 (9345): No heartbeat from core client for 30 sec - exiting 07:51:09 (9540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:51:31 (9540): No heartbeat from core client for 30 sec - exiting 07:51:33 (9540): No heartbeat from core client for 30 sec - exiting 07:51:34 (9540): No heartbeat from core client for 30 sec - exiting 07:51:35 (9540): No heartbeat from core client for 30 sec - exiting 07:51:36 (9540): No heartbeat from core client for 30 sec - exiting 08:04:11 (9723): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:04:54 (9723): No heartbeat from core client for 30 sec - exiting 08:09:40 (9960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:32 (10037): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:20:43 (10114): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:35:15 (10296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:48:36 (10551): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:50:13 (10799): No heartbeat from core client for 30 sec - exiting 08:50:22 (10799): No heartbeat from core client for 30 sec - exiting 08:50:23 (10799): No heartbeat from core client for 30 sec - exiting 08:50:24 (10799): No heartbeat from core client for 30 sec - exiting 08:50:25 (10799): No heartbeat from core client for 30 sec - exiting 08:50:26 (10799): No heartbeat from core client for 30 sec - exiting 08:50:28 (10799): No heartbeat from core client for 30 sec - exiting 08:50:29 (10799): No heartbeat from core client for 30 sec - exiting 08:50:30 (10799): No heartbeat from core client for 30 sec - exiting 08:50:31 (10799): No heartbeat from core client for 30 sec - exiting 08:50:32 (10799): No heartbeat from core client for 30 sec - exiting 08:50:33 (10799): No heartbeat from core client for 30 sec - exiting 08:50:34 (10799): No heartbeat from core client for 30 sec - exiting 08:50:35 (10799): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:56:08 (10863): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:58:26 (10942): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:49 (10959): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:52 (10959): No heartbeat from core client for 30 sec - exiting 09:16:13 (11052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:16:25 (11052): No heartbeat from core client for 30 sec - exiting 09:23:58 (11358): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:24:10 (11358): No heartbeat from core client for 30 sec - exiting 09:41:27 (11453): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish SETPOS: Seek Failed: Invalid argument SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Seek Failed: Invalid argument SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Seek Failed: Invalid argument SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Seek Failed: Invalid argument SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Seek Failed: Invalid argument SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Seek Failed: Invalid argument SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Nov 2011 18:50:11	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	622,080	2,837,622	4.5615
04 Nov 2011 00:17:27	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	596,160	2,713,163	4.5511
02 Nov 2011 04:04:59	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	570,240	2,570,157	4.5071
31 Oct 2011 18:42:25	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	544,320	2,427,236	4.4592
31 Oct 2011 15:41:34	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	518,400	2,284,015	4.4059
31 Oct 2011 13:12:17	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	492,480	2,141,944	4.3493
18 Oct 2011 17:29:03	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	466,560	2,000,173	4.2871
17 Oct 2011 20:00:51	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	440,640	1,927,023	4.3732
16 Oct 2011 19:43:13	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	414,720	1,877,058	4.5261
11 Oct 2011 01:15:04	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	388,800	1,766,147	4.5426
09 Oct 2011 09:12:09	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	362,880	1,631,662	4.4964
07 Oct 2011 17:07:34	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	336,960	1,559,910	4.6294
05 Oct 2011 23:44:36	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	311,040	1,664,482	5.3513
04 Oct 2011 06:19:17	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	285,120	1,520,114	5.3315
02 Oct 2011 12:49:07	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	259,200	1,376,388	5.3101
30 Sep 2011 18:22:01	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	233,280	1,241,391	5.3215
29 Sep 2011 02:07:34	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	207,360	1,107,497	5.3409
27 Sep 2011 09:08:20	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	181,440	970,455	5.3486
25 Sep 2011 15:49:59	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	155,520	827,205	5.3190
23 Sep 2011 22:34:44	1068723	13345602	hadcm3n_p5dz_1940_40_007421061_2	129,600	684,487	5.2815