Task 15772627

Name	hadcm3n_3998_1980_40_008363463_0
Workunit	8514322
Created	10 May 2013, 16:57:28 UTC
Sent	10 May 2013, 16:58:00 UTC
Report deadline	10 Aug 2013, 0:25:11 UTC
Received	12 May 2013, 18:27:52 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1282911
Run time	1 days 20 hours 52 min 30 sec
CPU time	1 days 18 hours 37 min 16 sec
Validate state	Invalid
Credit	622.08
Device peak FLOPS	2.00 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 00:55:16 (11496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:59:36 (11738): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:59:37 (11738): No heartbeat from core client for 30 sec - exiting 00:59:38 (11738): No heartbeat from core client for 30 sec - exiting 00:59:39 (11738): No heartbeat from core client for 30 sec - exiting 00:59:40 (11738): No heartbeat from core client for 30 sec - exiting 00:59:41 (11738): No heartbeat from core client for 30 sec - exiting 01:03:43 (12006): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:03:44 (12006): No heartbeat from core client for 30 sec - exiting 01:03:45 (12006): No heartbeat from core client for 30 sec - exiting 01:03:46 (12006): No heartbeat from core client for 30 sec - exiting 01:07:32 (12127): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:07:33 (12127): No heartbeat from core client for 30 sec - exiting 01:07:34 (12127): No heartbeat from core client for 30 sec - exiting 01:07:35 (12127): No heartbeat from core client for 30 sec - exiting 01:07:36 (12127): No heartbeat from core client for 30 sec - exiting 01:07:37 (12127): No heartbeat from core client for 30 sec - exiting 01:07:38 (12127): No heartbeat from core client for 30 sec - exiting 01:07:39 (12127): No heartbeat from core client for 30 sec - exiting 01:07:40 (12127): No heartbeat from core client for 30 sec - exiting 01:07:41 (12127): No heartbeat from core client for 30 sec - exiting 01:07:42 (12127): No heartbeat from core client for 30 sec - exiting 01:07:43 (12127): No heartbeat from core client for 30 sec - exiting 01:07:44 (12127): No heartbeat from core client for 30 sec - exiting 01:07:45 (12127): No heartbeat from core client for 30 sec - exiting 01:07:46 (12127): No heartbeat from core client for 30 sec - exiting 01:07:47 (12127): No heartbeat from core client for 30 sec - exiting 01:07:48 (12127): No heartbeat from core client for 30 sec - exiting 01:07:49 (12127): No heartbeat from core client for 30 sec - exiting 01:12:08 (12224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:12:09 (12224): No heartbeat from core client for 30 sec - exiting 01:12:10 (12224): No heartbeat from core client for 30 sec - exiting 01:12:11 (12224): No heartbeat from core client for 30 sec - exiting 01:12:12 (12224): No heartbeat from core client for 30 sec - exiting 01:12:13 (12224): No heartbeat from core client for 30 sec - exiting 01:12:14 (12224): No heartbeat from core client for 30 sec - exiting 01:16:15 (12324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:16:16 (12324): No heartbeat from core client for 30 sec - exiting 01:16:17 (12324): No heartbeat from core client for 30 sec - exiting 01:16:18 (12324): No heartbeat from core client for 30 sec - exiting 01:16:19 (12324): No heartbeat from core client for 30 sec - exiting 01:16:20 (12324): No heartbeat from core client for 30 sec - exiting 01:16:21 (12324): No heartbeat from core client for 30 sec - exiting 01:16:22 (12324): No heartbeat from core client for 30 sec - exiting 01:20:14 (12418): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:20:15 (12418): No heartbeat from core client for 30 sec - exiting 01:20:16 (12418): No heartbeat from core client for 30 sec - exiting 01:20:17 (12418): No heartbeat from core client for 30 sec - exiting 01:20:18 (12418): No heartbeat from core client for 30 sec - exiting 01:20:19 (12418): No heartbeat from core client for 30 sec - exiting 01:20:20 (12418): No heartbeat from core client for 30 sec - exiting 01:20:21 (12418): No heartbeat from core client for 30 sec - exiting 01:20:22 (12418): No heartbeat from core client for 30 sec - exiting 01:20:23 (12418): No heartbeat from core client for 30 sec - exiting 03:22:16 (12515): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:26:25 (12643): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:10 (12745): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:11 (12745): No heartbeat from core client for 30 sec - exiting 03:30:12 (12745): No heartbeat from core client for 30 sec - exiting 03:30:13 (12745): No heartbeat from core client for 30 sec - exiting 03:30:14 (12745): No heartbeat from core client for 30 sec - exiting 03:30:15 (12745): No heartbeat from core client for 30 sec - exiting 03:30:16 (12745): No heartbeat from core client for 30 sec - exiting 03:30:17 (12745): No heartbeat from core client for 30 sec - exiting 03:30:18 (12745): No heartbeat from core client for 30 sec - exiting 03:41:30 (12831): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:27 (12930): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:28 (12930): No heartbeat from core client for 30 sec - exiting 03:45:29 (12930): No heartbeat from core client for 30 sec - exiting 03:45:30 (12930): No heartbeat from core client for 30 sec - exiting 03:45:31 (12930): No heartbeat from core client for 30 sec - exiting 03:45:32 (12930): No heartbeat from core client for 30 sec - exiting 03:45:33 (12930): No heartbeat from core client for 30 sec - exiting 03:45:34 (12930): No heartbeat from core client for 30 sec - exiting 03:45:35 (12930): No heartbeat from core client for 30 sec - exiting 03:45:36 (12930): No heartbeat from core client for 30 sec - exiting 03:49:26 (13013): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:53:27 (13097): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:45 (13181): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:46 (13181): No heartbeat from core client for 30 sec - exiting 04:24:33 (13312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:28:36 (13404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:33:08 (13496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:36:45 (13572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:44:30 (13675): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:47:10 (13759): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:51:09 (13914): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:17 (13996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:55:34 (14862): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:59:35 (15054): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:59:36 (15054): No heartbeat from core client for 30 sec - exiting 05:59:37 (15054): No heartbeat from core client for 30 sec - exiting 06:07:42 (15161): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:09:58 (15332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:09:59 (15332): No heartbeat from core client for 30 sec - exiting 18:10:00 (15332): No heartbeat from core client for 30 sec - exiting 18:10:01 (15332): No heartbeat from core client for 30 sec - exiting 18:10:02 (15332): No heartbeat from core client for 30 sec - exiting 18:10:03 (15332): No heartbeat from core client for 30 sec - exiting 18:13:43 (16048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:44 (16048): No heartbeat from core client for 30 sec - exiting 18:13:45 (16048): No heartbeat from core client for 30 sec - exiting 18:17:25 (16128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:17:26 (16128): No heartbeat from core client for 30 sec - exiting 18:17:27 (16128): No heartbeat from core client for 30 sec - exiting 18:17:28 (16128): No heartbeat from core client for 30 sec - exiting 18:17:29 (16128): No heartbeat from core client for 30 sec - exiting 18:17:30 (16128): No heartbeat from core client for 30 sec - exiting 18:17:31 (16128): No heartbeat from core client for 30 sec - exiting 18:17:32 (16128): No heartbeat from core client for 30 sec - exiting 18:17:33 (16128): No heartbeat from core client for 30 sec - exiting 18:17:34 (16128): No heartbeat from core client for 30 sec - exiting 18:17:35 (16128): No heartbeat from core client for 30 sec - exiting 18:17:36 (16128): No heartbeat from core client for 30 sec - exiting 18:17:37 (16128): No heartbeat from core client for 30 sec - exiting 18:17:38 (16128): No heartbeat from core client for 30 sec - exiting 18:17:39 (16128): No heartbeat from core client for 30 sec - exiting 18:17:40 (16128): No heartbeat from core client for 30 sec - exiting 18:17:41 (16128): No heartbeat from core client for 30 sec - exiting 18:17:42 (16128): No heartbeat from core client for 30 sec - exiting 18:17:43 (16128): No heartbeat from core client for 30 sec - exiting 18:21:14 (16211): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:21:15 (16211): No heartbeat from core client for 30 sec - exiting 18:21:16 (16211): No heartbeat from core client for 30 sec - exiting 18:21:17 (16211): No heartbeat from core client for 30 sec - exiting 18:21:18 (16211): No heartbeat from core client for 30 sec - exiting 18:21:19 (16211): No heartbeat from core client for 30 sec - exiting 18:21:20 (16211): No heartbeat from core client for 30 sec - exiting 18:21:21 (16211): No heartbeat from core client for 30 sec - exiting 18:21:22 (16211): No heartbeat from core client for 30 sec - exiting 18:21:23 (16211): No heartbeat from core client for 30 sec - exiting 18:21:24 (16211): No heartbeat from core client for 30 sec - exiting 18:21:25 (16211): No heartbeat from core client for 30 sec - exiting 18:21:26 (16211): No heartbeat from core client for 30 sec - exiting 18:21:27 (16211): No heartbeat from core client for 30 sec - exiting 18:25:13 (16291): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:14 (16291): No heartbeat from core client for 30 sec - exiting 18:25:15 (16291): No heartbeat from core client for 30 sec - exiting 18:25:16 (16291): No heartbeat from core client for 30 sec - exiting 18:25:17 (16291): No heartbeat from core client for 30 sec - exiting 18:25:18 (16291): No heartbeat from core client for 30 sec - exiting 18:25:19 (16291): No heartbeat from core client for 30 sec - exiting 18:25:20 (16291): No heartbeat from core client for 30 sec - exiting 18:25:21 (16291): No heartbeat from core client for 30 sec - exiting 18:25:22 (16291): No heartbeat from core client for 30 sec - exiting 18:25:23 (16291): No heartbeat from core client for 30 sec - exiting 18:25:24 (16291): No heartbeat from core client for 30 sec - exiting 18:25:25 (16291): No heartbeat from core client for 30 sec - exiting 18:25:26 (16291): No heartbeat from core client for 30 sec - exiting 18:25:27 (16291): No heartbeat from core client for 30 sec - exiting 18:25:28 (16291): No heartbeat from core client for 30 sec - exiting 18:25:29 (16291): No heartbeat from core client for 30 sec - exiting 18:25:30 (16291): No heartbeat from core client for 30 sec - exiting 18:25:31 (16291): No heartbeat from core client for 30 sec - exiting 18:25:32 (16291): No heartbeat from core client for 30 sec - exiting 18:25:33 (16291): No heartbeat from core client for 30 sec - exiting 18:25:34 (16291): No heartbeat from core client for 30 sec - exiting 18:25:35 (16291): No heartbeat from core client for 30 sec - exiting 18:29:06 (16371): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:32:55 (16451): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:32:56 (16451): No heartbeat from core client for 30 sec - exiting 18:32:57 (16451): No heartbeat from core client for 30 sec - exiting 18:32:58 (16451): No heartbeat from core client for 30 sec - exiting 18:32:59 (16451): No heartbeat from core client for 30 sec - exiting 18:33:00 (16451): No heartbeat from core client for 30 sec - exiting 18:33:01 (16451): No heartbeat from core client for 30 sec - exiting 18:33:02 (16451): No heartbeat from core client for 30 sec - exiting 18:33:03 (16451): No heartbeat from core client for 30 sec - exiting 18:33:04 (16451): No heartbeat from core client for 30 sec - exiting 18:33:05 (16451): No heartbeat from core client for 30 sec - exiting 18:33:06 (16451): No heartbeat from core client for 30 sec - exiting 18:33:07 (16451): No heartbeat from core client for 30 sec - exiting 18:33:08 (16451): No heartbeat from core client for 30 sec - exiting 18:33:09 (16451): No heartbeat from core client for 30 sec - exiting 18:33:10 (16451): No heartbeat from core client for 30 sec - exiting 18:33:11 (16451): No heartbeat from core client for 30 sec - exiting 18:33:12 (16451): No heartbeat from core client for 30 sec - exiting 18:33:13 (16451): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7754400] [0xf7754425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75711df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7574825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf755c4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16541, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf770d400] [0xf770d425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf752a1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf752d825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75154d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16541, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf779d400] [0xf779d425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75ba1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75bd825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75a54d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16541, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7763400] [0xf7763425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75801df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7583825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf756b4d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16541, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77b1400] [0xf77b1425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75ce1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75d1825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75b94d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16541, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf774b400] [0xf774b425] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75681df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf756b825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75534d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16541, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 May 2013 08:46:05	1281428	15772627	hadcm3n_3998_1980_40_008363463_0	51,840	123,176	2.3761
11 May 2013 12:44:37	1281428	15772627	hadcm3n_3998_1980_40_008363463_0	25,920	60,338	2.3279