Task 22025748

Name	hadcm3s_ec38_190012_240_837_011898893_1
Workunit	11898893
Created	28 Feb 2021, 11:49:00 UTC
Sent	1 Mar 2021, 6:59:02 UTC
Report deadline	11 Feb 2022, 12:19:02 UTC
Received	8 Apr 2021, 6:11:01 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1483845
Run time	20 days 6 hours 19 min 51 sec
CPU time	20 days 5 hours 2 min 19 sec
Validate state	Valid
Credit	6,220.80
Device peak FLOPS	2.85 GFLOPS
Application version	UK Met Office HadCM3 short v8.36 i686-pc-linux-gnu
Peak working set size	166.34 MB
Peak swap size	214.83 MB
Peak disk usage	4.52 MB
Stderr	<core_client_version>7.9.3</core_client_version> <![CDATA[ <stderr_txt> MainError: 09:50:06 PM No files match the supplied pattern. MainError: 09:50:06 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 09:19:25 AM No files match the supplied pattern. MainError: 09:19:25 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... MainError: 07:39:38 AM No files match the supplied pattern. MainError: 07:39:38 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Signal 15 received: Software termination signal from kill Signal 15 received: Abnormal termination triggered by abort call Signal 15 received, exiting... SIGSEGV: segmentation violation Stack trace (21 frames): ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x80bc177] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7f28090] /lib/i386-linux-gnu/libc.so.6(getenv+0x99)[0xf7a62b49] /lib/i386-linux-gnu/libc.so.6(+0xaf448)[0xf7ae2448] /lib/i386-linux-gnu/libc.so.6(+0xaf825)[0xf7ae2825] /lib/i386-linux-gnu/libc.so.6(localtime_r+0x12)[0xf7ae0d82] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x80b7632] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x80b7d80] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x80b7e71] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7f28090] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x804cb98] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7f28090] linux-gate.so.1(__kernel_vsyscall+0x9)[0xf7f28079] /lib/i386-linux-gnu/libc.so.6(nanosleep+0x4b)[0xf7af21bb] /lib/i386-linux-gnu/libc.so.6(usleep+0x41)[0xf7b249b1] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x80cead5] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x804fe8b] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x80515bd] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x80517bd] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0xf7a4bf21] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x804c7a1] Exiting... MainError: 11:36:40 AM No files match the supplied pattern. MainError: 11:36:40 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 03:36:16 PM No files match the supplied pattern. MainError: 03:36:16 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 04:42:18 PM No files match the supplied pattern. MainError: 04:42:18 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... MainError: 07:57:52 PM No files match the supplied pattern. MainError: 07:57:52 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 08:02:53 PM No files match the supplied pattern. MainError: 08:02:53 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... MainError: 06:19:52 PM No files match the supplied pattern. MainError: 06:19:52 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... MainError: 02:10:38 PM No files match the supplied pattern. MainError: 02:10:38 PM No files match the supplied pattern. SIGSEGV: segmentation violation Stack trace (29 frames): /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x84ff4f7] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7f1a090] /lib/i386-linux-gnu/libc.so.6(cfree+0x2d)[0xf7c4452d] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x84e8472] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x84e849e] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x8498cba] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x84a13d9] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7f1a090] linux-gate.so.1(__kernel_vsyscall+0x9)[0xf7f1a079] /lib/i386-linux-gnu/libc.so.6(__munmap+0x16)[0xf7cbd436] /lib/i386-linux-gnu/libc.so.6(+0x76f83)[0xf7c3ff83] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x84e8472] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x84e849e] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x8498cba] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x84a13d9] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7f1a090] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x843f783] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x8105c23] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x833e699] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x80a0453] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x809871f] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x8358895] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x82ba112] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x82b7877] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x8483d60] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x8494d6d] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x848be04] /scratch/wes/BOINC/projects/climateprediction.net/hadcm3s_um_8.36_i686-pc-linux-gnu[0x8496bad] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0xf7be1f21] Exiting... MainError: 08:36:15 PM No files match the supplied pattern. MainError: 08:36:15 PM No files match the supplied pattern. MainError: 10:00:33 PM No files match the supplied pattern. MainError: 10:00:33 PM No files match the supplied pattern. MainError: 11:31:48 PM No files match the supplied pattern. MainError: 11:31:48 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 10:02:39 PM No files match the supplied pattern. MainError: 10:02:39 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 08:49:11 PM No files match the supplied pattern. MainError: 08:49:11 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 11:51:16 PM No files match the supplied pattern. MainError: 11:51:16 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Signal 1 received, exiting... SIGSEGV: segmentation violation Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... MainError: 08:12:12 AM No files match the supplied pattern. Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... MainError: 08:12:12 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... MainError: 12:24:49 AM No files match the supplied pattern. MainError: 12:24:49 AM No files match the supplied pattern. 22:50:55 (29761): BOINC client no longer exists - exiting 22:50:55 (29761): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... MainError: 11:35:44 PM No files match the supplied pattern. MainError: 11:35:44 PM No files match the supplied pattern. Processing restart Year 1920 Month 12 Day 1 MainError: 05:10:14 AM No files match the supplied pattern. MainError: 05:10:14 AM No files match the supplied pattern. 07:10:45 (7231): called boinc_finish(0) </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Apr 2021 05:14:09	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	518,400	1,746,112	3.3683
06 Apr 2021 23:50:57	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	492,480	1,639,876	3.3298
06 Apr 2021 00:26:40	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	466,560	1,556,966	3.3371
05 Apr 2021 08:16:11	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	447,120	1,498,920	3.3524
03 Apr 2021 23:55:53	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	414,720	1,426,560	3.4398
02 Apr 2021 20:51:28	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	388,800	1,330,057	3.4209
01 Apr 2021 22:06:36	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	362,880	1,249,515	3.4433
31 Mar 2021 23:34:38	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	336,960	1,169,291	3.4701
30 Mar 2021 22:04:01	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	311,040	1,077,636	3.4646
29 Mar 2021 20:40:40	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	285,120	986,443	3.4597
28 Mar 2021 14:34:50	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	259,200	893,575	3.4474
27 Mar 2021 18:22:31	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	233,280	822,449	3.5256
26 Mar 2021 20:11:22	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	207,360	742,690	3.5816
25 Mar 2021 20:00:53	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	181,440	660,141	3.6383
24 Mar 2021 16:45:52	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	155,520	566,206	3.6407
23 Mar 2021 15:39:00	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	129,600	481,539	3.7156
22 Mar 2021 11:37:13	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	103,680	396,298	3.8223
21 Mar 2021 07:42:35	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	77,760	304,118	3.9110
20 Mar 2021 09:27:01	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	51,840	224,201	4.3249
18 Mar 2021 22:27:06	1483845	22025748	hadcm3s_ec38_190012_240_837_011898893_1	25,920	106,852	4.1224