Task 18453055

Name	hadam3pm2_pe5t_1991_10_009528650_2
Workunit	9610393
Created	18 May 2015, 16:05:46 UTC
Sent	18 May 2015, 17:05:36 UTC
Report deadline	29 Apr 2016, 22:25:36 UTC
Received	15 Jun 2015, 10:34:44 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1291166
Run time	10 days 4 hours 30 min 48 sec
CPU time	9 days 13 hours 7 min 3 sec
Validate state	Invalid
Credit	8,068.48
Device peak FLOPS	2.41 GFLOPS
Application version	UK Met Office HadAM3P (global only) with MOSES II landsurface scheme v7.03 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Signal 15 received, exiting... 21:53:04 (7408): called boinc_finish Signal 15 received, exiting... 21:41:31 (1667): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:52:49 (26475): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:38:47 (15014): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 10:05:55 (2585): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:55:00 (10887): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:41:19 (12290): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:25:07 (9924): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:11:55 (3074): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:27:49 (21392): called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 09:40:36 (4122): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:59:03 (17350): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 22:00:54 (15372): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:04:49 (16698): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:56:07 (13966): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 21:55:50 (1634): called boinc_finish Signal 15 received, exiting... 21:41:45 (1605): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:20:35 (9147): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 17:05:14 (9358): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:45:24 (18724): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (13 frames): /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0f9ce50] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b39] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/libc.so.6(__libc_start_main+0xf3)[0xf0d9f993] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9333, iMonCtr=1 Model crash detected, will try to restart... Signal 15 received, exiting... 21:30:54 (15524): called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 10:03:34 (2791): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:49:00 (4718): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:31:45 (2741): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (13 frames): /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf0ef9e50] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8165b06] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816b8f8] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8162a76] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x816d014] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8161751] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8085677] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /home/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/libc.so.6(__libc_start_main+0xf3)[0xf0cfc993] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3250, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 22:08:21 (11180): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... 21:59:31 (5635): called boinc_finish Signal 15 received, exiting... 21:49:01 (16104): called boinc_finish Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=16104, selfPID=16104, iMonCtr=1 Signal 15 received, exiting... 12:59:43 (1533): called boinc_finish Signal 15 received, exiting... 22:00:03 (1539): called boinc_finish Model crashed: æM Model crashed: § Model crashed: § Model crashed: § Model crashed: § Model crashed: § Sorry, too many model crashes! :-( 10:57:54 (1424): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Jun 2015 14:31:52	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	279,428	741,049	2.6520
09 Jun 2015 19:40:30	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	256,388	693,571	2.7052
09 Jun 2015 17:27:08	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	253,508	685,688	2.7048
09 Jun 2015 15:08:26	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	250,628	677,795	2.7044
09 Jun 2015 12:59:42	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	247,748	670,093	2.7047
09 Jun 2015 10:46:30	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	244,868	662,375	2.7050
06 Jun 2015 15:28:58	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	210,308	581,253	2.7638
03 Jun 2015 19:08:26	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	178,628	493,898	2.7650
03 Jun 2015 16:55:03	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	175,748	485,998	2.7653
01 Jun 2015 20:12:08	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	144,068	399,713	2.7745
01 Jun 2015 17:59:04	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	141,188	392,036	2.7767
29 May 2015 16:03:54	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	106,628	296,451	2.7802
26 May 2015 09:07:37	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	74,948	209,307	2.7927
26 May 2015 06:54:45	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	72,068	201,533	2.7964
22 May 2015 08:26:09	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	37,508	105,316	2.8078
18 May 2015 19:29:23	1291166	18453055	hadam3pm2_pe5t_1991_10_009528650_2	2,948	8,305	2.8172