Task 21744521

Name	hadcm3s_c120_191512_240_835_011892813_0
Workunit	11892813
Created	25 Sep 2019, 10:43:27 UTC
Sent	26 Sep 2019, 6:41:06 UTC
Report deadline	7 Sep 2020, 12:01:06 UTC
Received	8 Oct 2019, 3:38:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1455853
Run time	10 days 6 hours 23 min 28 sec
CPU time	8 days 19 hours 46 min 39 sec
Validate state	Invalid
Credit	4,665.60
Device peak FLOPS	4.83 GFLOPS
Application version	UK Met Office HadCM3 short v8.36 i686-pc-linux-gnu
Peak working set size	166.46 MB
Peak swap size	214.92 MB
Peak disk usage	99.06 MB
Stderr	<core_client_version>7.14.2</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234)</message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 12:23:55 AM No files match the supplied pattern. MainError: 12:23:55 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 03:47:30 PM No files match the supplied pattern. MainError: 03:47:30 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 10:32:20 AM No files match the supplied pattern. MainError: 10:32:20 AM No files match the supplied pattern. MainError: 12:01:44 AM No files match the supplied pattern. MainError: 12:01:44 AM No files match the supplied pattern. MainError: 01:13:05 PM No files match the supplied pattern. MainError: 01:13:05 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 04:29:32 AM No files match the supplied pattern. MainError: 04:29:32 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 02:14:38 AM No files match the supplied pattern. MainError: 02:14:38 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 08:11:38 PM No files match the supplied pattern. MainError: 08:11:38 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 03:58:38 PM No files match the supplied pattern. MainError: 03:58:38 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received: Software termination signal from kill Signal 15 received: Abnormal termination triggered by abort call Signal 15 received, exiting... SIGSEGV: segmentation violation Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x80bc177] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7f2a950] /lib/i386-linux-gnu/libc.so.6(getenv+0x99)[0xf7a1b0c9] /lib/i386-linux-gnu/libc.so.6(+0xb538d)[0xf7a9b38d] /lib/i386-linux-gnu/libc.so.6(+0xb571b)[0xf7a9b71b] /lib/i386-linux-gnu/libc.so.6(localtime_r+0x17)[0xf7a99857] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x80b7632] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x80b7d80] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x80b7e71] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7f2a950] linux-gate.so.1(__kernel_vsyscall+0x9)[0xf7f2a939] /lib/i386-linux-gnu/libc.so.6(nanosleep+0x4b)[0xf7aab61b] /lib/i386-linux-gnu/libc.so.6(usleep+0x41)[0xf7adecd1] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x80cead5] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x804fe8b] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x80515bd] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x80517bd] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0xf7a04751] ../../projects/climateprediction.net/hadcm3s_8.36_i686-pc-linux-gnu[0x804c7a1] Exiting... MainError: 12:38:02 AM No files match the supplied pattern. MainError: 12:38:02 AM No files match the supplied pattern. MainError: 05:47:21 AM No files match the supplied pattern. MainError: 05:47:21 AM No files match the supplied pattern. MainError: 11:20:52 PM No files match the supplied pattern. MainError: 11:20:52 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 08:08:49 PM No files match the supplied pattern. MainError: 08:08:49 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 04:16:33 PM No files match the supplied pattern. MainError: 04:16:33 PM No files match the supplied pattern. MainError: 09:12:19 AM No files match the supplied pattern. MainError: 09:12:19 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3s_c120_191512_240_835_011892813/dataout/stdout_um.txt Image PC Routine Line Source hadcm3s_um_8.36_i 0851D9E5 Unknown Unknown Unknown hadcm3s_um_8.36_i 0855E2E0 Unknown Unknown Unknown hadcm3s_um_8.36_i 0855C53F Unknown Unknown Unknown hadcm3s_um_8.36_i 0821C61D Unknown Unknown Unknown hadcm3s_um_8.36_i 084643F6 Unknown Unknown Unknown hadcm3s_um_8.36_i 0847E15A Unknown Unknown Unknown hadcm3s_um_8.36_i 084940B2 Unknown Unknown Unknown hadcm3s_um_8.36_i 0848BE04 Unknown Unknown Unknown hadcm3s_um_8.36_i 08496BAD Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27328, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 forrtl: No space left on device forrtl: severe (38): error during write, unit -1, file /proc/9006/fd/1 Image PC Routine Line Source hadcm3s_um_8.36_i 0851D9E5 Unknown Unknown Unknown hadcm3s_um_8.36_i 0856077D Unknown Unknown Unknown hadcm3s_um_8.36_i 0855C53F Unknown Unknown Unknown hadcm3s_um_8.36_i 083E3EA1 Unknown Unknown Unknown hadcm3s_um_8.36_i 0848B335 Unknown Unknown Unknown hadcm3s_um_8.36_i 08496BAD Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27328, iMonCtr=1 Model crash detected, will try to restart... SETPOS: Seek Failed: No space left on device SETPOS: Unit 22 to Word Address 806912 Failed with Error Code -1 Model crashed: SETPOS: Unit 22 to Word Address 806912 Failed with Error Code -1 SETPOS: Seek Failed: No space left on device SETPOS: Unit 22 to Word Address 806912 Failed with Error Code -1 Model crashed: SETPOS: Unit 22 to Word Address 806912 Failed with Error Code -1 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3s_c120_191512_240_835_011892813/dataout/stdout_um.txt Image PC Routine Sorry, too many model crashes! :-( 19:38:10 (27328): called boinc_finish(22) </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Oct 2019 09:13:19	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	388,800	721,935	1.8568
06 Oct 2019 16:21:15	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	362,880	666,921	1.8379
05 Oct 2019 20:10:44	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	336,960	610,155	1.8108
04 Oct 2019 23:24:02	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	311,040	555,220	1.7850
04 Oct 2019 05:52:39	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	285,120	501,454	1.7587
03 Oct 2019 12:39:02	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	259,200	448,318	1.7296
02 Oct 2019 16:01:27	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	233,280	391,996	1.6804
01 Oct 2019 20:14:04	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	207,360	341,097	1.6450
01 Oct 2019 10:44:52	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	181,440	290,655	1.6019
01 Oct 2019 10:44:08	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	155,520	245,329	1.5775
01 Oct 2019 10:43:29	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	129,600	199,479	1.5392
01 Oct 2019 10:42:48	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	103,680	159,819	1.5415
01 Oct 2019 10:41:38	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	77,760	119,217	1.5331
27 Sep 2019 16:47:19	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	51,840	80,089	1.5449
27 Sep 2019 15:48:34	1455853	21744521	hadcm3s_c120_191512_240_835_011892813_0	25,920	40,046	1.5450