Task 15681858

Name	hadcm3n_3f4x_1940_40_008268591_1
Workunit	8423715
Created	24 Mar 2013, 20:56:06 UTC
Sent	24 Mar 2013, 20:56:07 UTC
Report deadline	24 Jun 2013, 4:23:18 UTC
Received	7 Apr 2013, 18:10:02 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1205535
Run time	9 days 7 hours 45 min 14 sec
CPU time	8 days 3 hours 15 min 31 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.12 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/3f4xko.pjf9c10 is not a valid UM file. Error converting file to netcdf: dataout/3f4xko.pjf9c10 Error: Input file: dataout/3f4xko.pif9c10 is not a valid UM file. Error converting file to netcdf: dataout/3f4xko.pif9c10 Error: Input file: dataout/3f4xko.pff9c10 is not a valid UM file. Error converting file to netcdf: dataout/3f4xko.pff9c10 Error: Input file: dataout/3f4xka.phf9c10 is not a valid UM file. Error converting file to netcdf: dataout/3f4xka.phf9c10 Error: Input file: dataout/3f4xka.pgf9c10 is not a valid UM file. Error converting file to netcdf: dataout/3f4xka.pgf9c10 Error: Input file: dataout/3f4xka.pef9c10 is not a valid UM file. Error converting file to netcdf: dataout/3f4xka.pef9c10 Error: Input file: dataout/3f4xka.pdf9c10 is not a valid UM file. Error converting file to netcdf: dataout/3f4xka.pdf9c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x1860404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x1860400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x832c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x84e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x84e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x84e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x84e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x84e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x84e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x1844e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x832c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x84e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x84e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x84e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x84e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x84e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x84e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x84e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x84e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x1844e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x834404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x834400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x834404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x834400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x834404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x834400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x180e604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x180e600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x1800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x1800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96932,0xa0b7b540) malloc: * error for object 0x800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=96934, selfPID=96934, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(97853,0xa0b7b540) malloc: * error for object 0x846c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97853,0xa0b7b540) malloc: * error for object 0x182f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97853,0xa0b7b540) malloc: * error for object 0x182f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97853,0xa0b7b540) malloc: * error for object 0x846c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97853,0xa0b7b540) malloc: * error for object 0x83de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97853,0xa0b7b540) malloc: * error for object 0x83de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97853,0xa0b7b540) malloc: * error for object 0x83de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97853,0xa0b7b540) malloc: * error for object 0x83de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97853,0xa0b7b540) malloc: * error for object 0x83de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97853,0xa0b7b540) malloc: * error for object 0x83de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(97853,0xa0b7b540) malloc: * error for object 0x83de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(98635,0xa0b7b540) malloc: * error for object 0x1040604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(99046,0xa0b7b540) malloc: * error for object 0x82ec04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99046,0xa0b7b540) malloc: * error for object 0x1012c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99046,0xa0b7b540) malloc: * error for object 0x1012c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99046,0xa0b7b540) malloc: * error for object 0x1000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(99450,0xa0b7b540) malloc: * error for object 0x83de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99450,0xa0b7b540) malloc: * error for object 0x83de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99450,0xa0b7b540) malloc: * error for object 0x1000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99450,0xa0b7b540) malloc: * error for object 0x1001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99450,0xa0b7b540) malloc: * error for object 0x1001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99450,0xa0b7b540) malloc: * error for object 0x1001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99450,0xa0b7b540) malloc: * error for object 0x1001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99450,0xa0b7b540) malloc: * error for object 0x83de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99450,0xa0b7b540) malloc: * error for object 0x83de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99450,0xa0b7b540) malloc: * error for object 0x800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99450,0xa0b7b540) malloc: * error for object 0x800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3f4x_1940_40_008268591/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3f4x_1940_40_008268591/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3f4x_1940_40_008268591/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3f4x_1940_40_008268591/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3f4x_1940_40_008268591/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3f4x_1940_40_008268591/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3f4x_1940_40_008268591/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3f4x_1940_40_008268591/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3f4x_1940_40_008268591/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3f4x_1940_40_008268591/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3f4x_1940_40_008268591/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3f4x_1940_40_008268591/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Apr 2013 12:08:59	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	518,400	702,959	1.3560
06 Apr 2013 14:25:27	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	492,480	669,712	1.3599
05 Apr 2013 00:06:22	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	466,560	635,124	1.3613
02 Apr 2013 16:11:04	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	440,640	599,775	1.3611
01 Apr 2013 11:20:26	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	414,720	564,409	1.3609
01 Apr 2013 00:21:20	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	388,800	529,115	1.3609
31 Mar 2013 13:15:36	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	362,880	493,806	1.3608
31 Mar 2013 02:07:23	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	336,960	458,501	1.3607
30 Mar 2013 15:00:31	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	311,040	423,124	1.3604
30 Mar 2013 03:52:52	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	285,120	387,768	1.3600
29 Mar 2013 16:35:54	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	259,200	352,387	1.3595
29 Mar 2013 05:24:24	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	233,280	317,043	1.3591
28 Mar 2013 18:25:47	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	207,360	281,901	1.3595
28 Mar 2013 07:23:13	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	181,440	246,709	1.3597
27 Mar 2013 20:26:27	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	155,520	211,544	1.3602
27 Mar 2013 09:32:36	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	129,600	176,292	1.3603
26 Mar 2013 22:51:54	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	103,680	141,065	1.3606
26 Mar 2013 11:32:09	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	77,760	105,833	1.3610
26 Mar 2013 00:37:58	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	51,840	70,588	1.3617
25 Mar 2013 13:45:06	1205535	15681858	hadcm3n_3f4x_1940_40_008268591_1	25,920	35,302	1.3620