Task 14364248

Name	hadcm3n_o1g3_2020_40_007857868_1
Workunit	8012980
Created	5 Apr 2012, 13:57:34 UTC
Sent	5 Apr 2012, 13:57:58 UTC
Report deadline	5 Jul 2012, 21:25:09 UTC
Received	6 Jun 2012, 7:30:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1193004
Run time	47 days 11 hours 35 min 31 sec
CPU time	42 days 19 hours 24 min 52 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	1.31 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:42:41 (6923): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7131, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7131, iMonCtr=1 Model crash detected, will try to restart... 07:51:16 (7131): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14039, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14039, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14039, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18161, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18161, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18161, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18161, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18161, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18161, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18161, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18161, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=18161, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:00:52 (6962): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09635ef0 *** ======= Backtrace: ========= /lib/i386-linux-gnu/libc.so.6(+0x6ff22)[0x73cf22] /lib/i386-linux-gnu/libc.so.6(+0x70bc2)[0x73dbc2] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0x740cad] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x1f)[0xc0880f] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1b)[0xc0886b] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x6e6113] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 00312000-00330000 r-xp 00000000 08:01 787373 /lib/i386-linux-gnu/ld-2.13.so 00330000-00331000 r--p 0001d000 08:01 787373 /lib/i386-linux-gnu/ld-2.13.so 00331000-00332000 rw-p 0001e000 08:01 787373 /lib/i386-linux-gnu/ld-2.13.so 00554000-00555000 r-xp 00000000 00:00 0 [vdso] 005a9000-005c5000 r-xp 00000000 08:01 787386 /lib/i386-linux-gnu/libgcc_s.so.1 005c5000-005c6000 r--p 0001b000 08:01 787386 /lib/i386-linux-gnu/libgcc_s.so.1 005c6000-005c7000 rw-p 0001c000 08:01 787386 /lib/i386-linux-gnu/libgcc_s.so.1 006cd000-00845000 r-xp 00000000 08:01 796383 /lib/i386-linux-gnu/libc-2.13.so 00845000-00847000 r--p 00178000 08:01 796383 /lib/i386-linux-gnu/libc-2.13.so 00847000-00848000 rw-p 0017a000 08:01 796383 /lib/i386-linux-gnu/libc-2.13.so 00848000-0084b000 rw-p 00000000 00:00 0 00993000-009bb000 r-xp 00000000 08:01 796387 /lib/i386-linux-gnu/libm-2.13.so 009bb000-009bc000 r--p 00028000 08:01 796387 /lib/i386-linux-gnu/libm-2.13.so 009bc000-009bd000 rw-p 00029000 08:01 796387 /lib/i386-linux-gnu/libm-2.13.so 00aac000-00ac3000 r-xp 00000000 08:01 796397 /lib/i386-linux-gnu/libpthread-2.13.so 00ac3000-00ac4000 r--p 00016000 08:01 796397 /lib/i386-linux-gnu/libpthread-2.13.so 00ac4000-00ac5000 rw-p 00017000 08:01 796397 /lib/i386-linux-gnu/libpthread-2.13.so 00ac5000-00ac7000 rw-p 00000000 00:00 0 00b5b000-00c39000 r-xp 00000000 08:01 1316219 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 00c39000-00c3a000 ---p 000de000 08:01 1316219 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 00c3a000-00c3e000 r--p 000de000 08:01 1316219 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 00c3e000-00c3f000 rw-p 000e2000 08:01 1316219 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 00c3f000-00c46000 rw-p 00000000 00:00 0 00fd4000-00fd7000 r-xp 00000000 08:01 796386 /lib/i386-linux-gnu/libdl-2.13.so 00fd7000-00fd8000 r--p 00002000 08:01 796386 /lib/i386-linux-gnu/libdl-2.13.so 00fd8000-00fd9000 rw-p 00003000 08:01 796386 /lib/i386-linux-gnu/libdl-2.13.so 08048000-080e3000 r-xp 00000000 08:01 1850948 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 1850948 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 095e0000-09646000 rw-p 00000000 00:00 0 [heap] b7100000-b7121000 rw-p 00000000 00:00 0 b7121000-b7200000 ---p 00000000 00:00 0 b72b9000-b7734000 rw-s 00000000 08:01 1584222 /var/lib/boinc-client/slots/3/134745 b7734000-b7738000 rw-p 00000000 00:00 0 b7743000-b7744000 rw-p 00000000 00:00 0 b7744000-b7745000 ---p 00000000 00:00 0 b7745000-b7748000 rw-p 00000000 00:00 0 b7748000-b774a000 rw-s 00000000 08:01 1584647 /var/lib/boinc-client/slots/3/boinc_mmap_file b774a000-b774c000 rw-p 00000000 00:00 0 bff69000-bffd8000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x554400] [0x554416] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0x6fac8f] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0x6fe2b5] /lib/i386-linux-gnu/libc.so.6(+0x6515c)[0x73215c] /lib/i386-linux-gnu/libc.so.6(+0x6ff22)[0x73cf22] /lib/i386-linux-gnu/libc.so.6(+0x70bc2)[0x73dbc2] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0x740cad] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x1f)[0xc0880f] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1b)[0xc0886b] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x6e6113] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Jun 2012 16:38:05	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	1,036,800	3,698,677	3.5674
04 Jun 2012 12:45:19	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	1,010,880	3,601,556	3.5628
02 Jun 2012 22:28:40	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	984,960	3,505,525	3.5591
01 Jun 2012 05:37:54	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	959,040	3,409,182	3.5548
31 May 2012 01:50:39	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	933,120	3,312,076	3.5495
29 May 2012 20:37:10	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	907,200	3,215,199	3.5441
28 May 2012 05:41:53	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	881,280	3,119,435	3.5397
26 May 2012 14:02:24	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	855,360	3,024,248	3.5356
24 May 2012 23:40:51	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	829,440	2,928,315	3.5305
23 May 2012 09:42:48	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	803,520	2,833,199	3.5260
22 May 2012 05:55:08	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	777,600	2,736,433	3.5191
21 May 2012 02:09:40	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	751,680	2,639,412	3.5114
19 May 2012 11:27:49	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	725,760	2,543,394	3.5045
18 May 2012 07:40:14	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	699,840	2,446,112	3.4952
17 May 2012 03:53:21	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	673,920	2,349,081	3.4857
15 May 2012 16:41:30	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	648,000	2,253,881	3.4782
14 May 2012 01:34:04	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	622,080	2,158,271	3.4694
12 May 2012 11:39:31	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	596,160	2,063,536	3.4614
10 May 2012 23:45:09	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	570,240	1,967,540	3.4504
09 May 2012 19:05:37	1193004	14364248	hadcm3n_o1g3_2020_40_007857868_1	544,320	1,870,464	3.4363