Task 15770501

Name	hadcm3n_4jqh_1940_40_008310209_2
Workunit	8461344
Created	10 May 2013, 8:47:51 UTC
Sent	10 May 2013, 8:48:13 UTC
Report deadline	9 Aug 2013, 16:15:24 UTC
Received	4 Sep 2013, 18:10:33 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1291528
Run time	3 days 11 hours 0 min 11 sec
CPU time	3 days 5 hours 46 min 41 sec
Validate state	Invalid
Credit	1,244.16
Device peak FLOPS	2.00 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 00:55:15 (10152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:59:36 (11706): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:03:44 (11974): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:03:45 (11974): No heartbeat from core client for 30 sec - exiting 01:03:46 (11974): No heartbeat from core client for 30 sec - exiting 01:07:31 (12097): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:07:32 (12097): No heartbeat from core client for 30 sec - exiting 01:07:33 (12097): No heartbeat from core client for 30 sec - exiting 01:07:34 (12097): No heartbeat from core client for 30 sec - exiting 01:07:35 (12097): No heartbeat from core client for 30 sec - exiting 01:07:36 (12097): No heartbeat from core client for 30 sec - exiting 01:07:37 (12097): No heartbeat from core client for 30 sec - exiting 01:07:38 (12097): No heartbeat from core client for 30 sec - exiting 01:07:39 (12097): No heartbeat from core client for 30 sec - exiting 01:07:40 (12097): No heartbeat from core client for 30 sec - exiting 01:07:41 (12097): No heartbeat from core client for 30 sec - exiting 01:07:42 (12097): No heartbeat from core client for 30 sec - exiting 01:07:43 (12097): No heartbeat from core client for 30 sec - exiting 01:07:44 (12097): No heartbeat from core client for 30 sec - exiting 01:07:45 (12097): No heartbeat from core client for 30 sec - exiting 01:07:46 (12097): No heartbeat from core client for 30 sec - exiting 01:07:47 (12097): No heartbeat from core client for 30 sec - exiting 01:07:48 (12097): No heartbeat from core client for 30 sec - exiting 01:07:49 (12097): No heartbeat from core client for 30 sec - exiting 01:12:07 (12190): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:16:15 (12296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:16:16 (12296): No heartbeat from core client for 30 sec - exiting 01:16:17 (12296): No heartbeat from core client for 30 sec - exiting 01:16:18 (12296): No heartbeat from core client for 30 sec - exiting 01:16:19 (12296): No heartbeat from core client for 30 sec - exiting 01:16:20 (12296): No heartbeat from core client for 30 sec - exiting 01:16:21 (12296): No heartbeat from core client for 30 sec - exiting 01:20:15 (12388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:20:16 (12388): No heartbeat from core client for 30 sec - exiting 01:20:17 (12388): No heartbeat from core client for 30 sec - exiting 01:20:18 (12388): No heartbeat from core client for 30 sec - exiting 01:20:19 (12388): No heartbeat from core client for 30 sec - exiting 01:20:20 (12388): No heartbeat from core client for 30 sec - exiting 01:20:21 (12388): No heartbeat from core client for 30 sec - exiting 01:20:22 (12388): No heartbeat from core client for 30 sec - exiting 01:20:23 (12388): No heartbeat from core client for 30 sec - exiting 03:22:16 (12489): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:26:25 (12623): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:09 (12729): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:30:10 (12729): No heartbeat from core client for 30 sec - exiting 03:30:11 (12729): No heartbeat from core client for 30 sec - exiting 03:30:12 (12729): No heartbeat from core client for 30 sec - exiting 03:30:13 (12729): No heartbeat from core client for 30 sec - exiting 03:30:14 (12729): No heartbeat from core client for 30 sec - exiting 03:30:15 (12729): No heartbeat from core client for 30 sec - exiting 03:30:16 (12729): No heartbeat from core client for 30 sec - exiting 03:30:17 (12729): No heartbeat from core client for 30 sec - exiting 03:41:31 (12813): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:27 (12926): No heartbeat from core client for 30 sec - exiting 03:45:38 (12926): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:49:26 (13009): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:53:27 (13093): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:45 (13177): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:24:32 (13308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:24:33 (13308): No heartbeat from core client for 30 sec - exiting 04:24:34 (13308): No heartbeat from core client for 30 sec - exiting 04:24:35 (13308): No heartbeat from core client for 30 sec - exiting 04:24:36 (13308): No heartbeat from core client for 30 sec - exiting 04:28:36 (13400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:33:08 (13492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:36:45 (13568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:44:30 (13671): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:47:10 (13755): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:47:11 (13755): No heartbeat from core client for 30 sec - exiting 06:47:12 (13755): No heartbeat from core client for 30 sec - exiting 06:47:13 (13755): No heartbeat from core client for 30 sec - exiting 06:47:14 (13755): No heartbeat from core client for 30 sec - exiting 06:47:15 (13755): No heartbeat from core client for 30 sec - exiting 06:47:16 (13755): No heartbeat from core client for 30 sec - exiting 06:51:08 (13910): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:18 (13992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:55:34 (14858): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:59:35 (15050): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:59:36 (15050): No heartbeat from core client for 30 sec - exiting 05:59:37 (15050): No heartbeat from core client for 30 sec - exiting 06:07:42 (15157): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:09:58 (15328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:43 (16044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:17:25 (16124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:21:14 (16207): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:13 (16287): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:29:06 (16367): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:32:55 (16447): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:36:45 (16537): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:33 (16625): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:19:56 (16693): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:23:21 (16864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:48:56 (3908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:39:50 (4278): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:39:51 (4278): No heartbeat from core client for 30 sec - exiting 16:42:10 (4915): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:59 (5021): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:58:09 (5232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:46 (5324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:49:36 (5617): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:54:20 (35870): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:05:45 (43160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:10:11 (43253): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:15:54 (3495): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:18:21 (4236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:29:55 (3745): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:29:56 (3745): No heartbeat from core client for 30 sec - exiting 13:29:57 (3745): No heartbeat from core client for 30 sec - exiting 13:40:01 (4479): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:28:24 (2929): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:28:25 (2929): No heartbeat from core client for 30 sec - exiting 14:28:26 (2929): No heartbeat from core client for 30 sec - exiting 14:28:27 (2929): No heartbeat from core client for 30 sec - exiting 15:54:53 (3913): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:58:56 (4409): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:58:57 (4409): No heartbeat from core client for 30 sec - exiting 16:02:40 (4503): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:06:02 (4620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:06:03 (4620): No heartbeat from core client for 30 sec - exiting 16:06:04 (4620): No heartbeat from core client for 30 sec - exiting 16:37:43 (4706): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:42:03 (4802): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:46:10 (4907): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:08 (5029): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:09 (5029): No heartbeat from core client for 30 sec - exiting 16:50:10 (5029): No heartbeat from core client for 30 sec - exiting 16:50:11 (5029): No heartbeat from core client for 30 sec - exiting 16:50:12 (5029): No heartbeat from core client for 30 sec - exiting 16:54:16 (5142): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:54:17 (5142): No heartbeat from core client for 30 sec - exiting 16:54:18 (5142): No heartbeat from core client for 30 sec - exiting 16:54:19 (5142): No heartbeat from core client for 30 sec - exiting 16:54:20 (5142): No heartbeat from core client for 30 sec - exiting 16:54:21 (5142): No heartbeat from core client for 30 sec - exiting 16:54:22 (5142): No heartbeat from core client for 30 sec - exiting 16:54:23 (5142): No heartbeat from core client for 30 sec - exiting 16:54:24 (5142): No heartbeat from core client for 30 sec - exiting 16:54:25 (5142): No heartbeat from core client for 30 sec - exiting 16:54:26 (5142): No heartbeat from core client for 30 sec - exiting 16:54:27 (5142): No heartbeat from core client for 30 sec - exiting 17:17:26 (5247): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:21:33 (5470): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:21:34 (5470): No heartbeat from core client for 30 sec - exiting 17:21:35 (5470): No heartbeat from core client for 30 sec - exiting 17:21:36 (5470): No heartbeat from core client for 30 sec - exiting 17:21:37 (5470): No heartbeat from core client for 30 sec - exiting 17:21:38 (5470): No heartbeat from core client for 30 sec - exiting 17:21:39 (5470): No heartbeat from core client for 30 sec - exiting 17:49:50 (5580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:57:35 (5673): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:02:10 (5791): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:06:48 (5930): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:11:47 (6046): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:11:48 (6046): No heartbeat from core client for 30 sec - exiting 18:11:49 (6046): No heartbeat from core client for 30 sec - exiting 18:11:50 (6046): No heartbeat from core client for 30 sec - exiting 18:11:51 (6046): No heartbeat from core client for 30 sec - exiting 18:11:52 (6046): No heartbeat from core client for 30 sec - exiting 18:11:53 (6046): No heartbeat from core client for 30 sec - exiting 18:11:54 (6046): No heartbeat from core client for 30 sec - exiting 18:11:55 (6046): No heartbeat from core client for 30 sec - exiting 18:11:56 (6046): No heartbeat from core client for 30 sec - exiting 18:11:57 (6046): No heartbeat from core client for 30 sec - exiting 18:11:58 (6046): No heartbeat from core client for 30 sec - exiting 18:11:59 (6046): No heartbeat from core client for 30 sec - exiting 18:12:00 (6046): No heartbeat from core client for 30 sec - exiting 18:12:01 (6046): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7fdb400] [0xf7fdb430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7df81df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7dfb825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf7de34d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6154, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7fdb400] [0xf7fdb430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7df81df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7dfb825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf7de34d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6154, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7fdb400] [0xf7fdb430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7df81df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7dfb825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf7de34d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6154, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7fdb400] [0xf7fdb430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7df81df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7dfb825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf7de34d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6154, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7fdb400] [0xf7fdb430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7df81df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7dfb825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf7de34d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6154, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7fdb400] [0xf7fdb430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf7df81df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7dfb825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf7de34d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6154, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 May 2013 20:11:56	1282911	15770501	hadcm3n_4jqh_1940_40_008310209_2	103,680	242,582	2.3397
11 May 2013 22:12:08	1281428	15770501	hadcm3n_4jqh_1940_40_008310209_2	51,840	120,757	2.3294
11 May 2013 04:05:57	1281428	15770501	hadcm3n_4jqh_1940_40_008310209_2	25,920	59,373	2.2906