Task 15655601

Name	hadcm3n_3gun_1940_40_008262966_2
Workunit	8418090
Created	10 Mar 2013, 3:06:25 UTC
Sent	10 Mar 2013, 3:06:43 UTC
Report deadline	9 Jun 2013, 10:33:54 UTC
Received	11 Mar 2013, 20:39:44 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1270421
Run time	1 days 1 hours 6 min 9 sec
CPU time	1 days 0 hours 3 min 2 sec
Validate state	Invalid
Credit	622.08
Device peak FLOPS	3.22 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 04:44:22 (9895): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:48:52 (13639): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:28:17 (15193): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:12:57 (23112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:19:42 (32029): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:58:14 (21297): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 01:01:58 (5994): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:40:00 (6116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:02:29 (13562): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:08:03 (18486): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:35:47 (19648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:09:15 (25330): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:00:44 (31980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:00:45 (31980): No heartbeat from core client for 30 sec - exiting 04:00:46 (31980): No heartbeat from core client for 30 sec - exiting 04:00:47 (31980): No heartbeat from core client for 30 sec - exiting 04:00:48 (31980): No heartbeat from core client for 30 sec - exiting 04:00:49 (31980): No heartbeat from core client for 30 sec - exiting 04:00:50 (31980): No heartbeat from core client for 30 sec - exiting 04:00:51 (31980): No heartbeat from core client for 30 sec - exiting 04:00:52 (31980): No heartbeat from core client for 30 sec - exiting 04:00:53 (31980): No heartbeat from core client for 30 sec - exiting 04:00:54 (31980): No heartbeat from core client for 30 sec - exiting 04:00:55 (31980): No heartbeat from core client for 30 sec - exiting 04:00:56 (31980): No heartbeat from core client for 30 sec - exiting 04:00:57 (31980): No heartbeat from core client for 30 sec - exiting 04:00:58 (31980): No heartbeat from core client for 30 sec - exiting 04:00:59 (31980): No heartbeat from core client for 30 sec - exiting 04:01:00 (31980): No heartbeat from core client for 30 sec - exiting 04:01:01 (31980): No heartbeat from core client for 30 sec - exiting 04:01:02 (31980): No heartbeat from core client for 30 sec - exiting 04:01:03 (31980): No heartbeat from core client for 30 sec - exiting 04:01:04 (31980): No heartbeat from core client for 30 sec - exiting 04:01:05 (31980): No heartbeat from core client for 30 sec - exiting 04:01:06 (31980): No heartbeat from core client for 30 sec - exiting 04:01:07 (31980): No heartbeat from core client for 30 sec - exiting 04:01:08 (31980): No heartbeat from core client for 30 sec - exiting 04:01:09 (31980): No heartbeat from core client for 30 sec - exiting 04:01:10 (31980): No heartbeat from core client for 30 sec - exiting 04:01:11 (31980): No heartbeat from core client for 30 sec - exiting 04:01:12 (31980): No heartbeat from core client for 30 sec - exiting 04:01:13 (31980): No heartbeat from core client for 30 sec - exiting 04:01:14 (31980): No heartbeat from core client for 30 sec - exiting 04:01:15 (31980): No heartbeat from core client for 30 sec - exiting 04:01:16 (31980): No heartbeat from core client for 30 sec - exiting 04:01:17 (31980): No heartbeat from core client for 30 sec - exiting 04:04:04 (15806): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:42:35 (17401): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:42:36 (17401): No heartbeat from core client for 30 sec - exiting 10:42:37 (17401): No heartbeat from core client for 30 sec - exiting 10:42:38 (17401): No heartbeat from core client for 30 sec - exiting 10:42:39 (17401): No heartbeat from core client for 30 sec - exiting 10:42:40 (17401): No heartbeat from core client for 30 sec - exiting 10:42:41 (17401): No heartbeat from core client for 30 sec - exiting 10:42:42 (17401): No heartbeat from core client for 30 sec - exiting 10:42:43 (17401): No heartbeat from core client for 30 sec - exiting 10:42:44 (17401): No heartbeat from core client for 30 sec - exiting 10:42:45 (17401): No heartbeat from core client for 30 sec - exiting 10:42:46 (17401): No heartbeat from core client for 30 sec - exiting 10:42:47 (17401): No heartbeat from core client for 30 sec - exiting 10:42:48 (17401): No heartbeat from core client for 30 sec - exiting 10:42:49 (17401): No heartbeat from core client for 30 sec - exiting 10:42:50 (17401): No heartbeat from core client for 30 sec - exiting 10:42:51 (17401): No heartbeat from core client for 30 sec - exiting 10:42:52 (17401): No heartbeat from core client for 30 sec - exiting 10:42:53 (17401): No heartbeat from core client for 30 sec - exiting 10:42:54 (17401): No heartbeat from core client for 30 sec - exiting 10:42:55 (17401): No heartbeat from core client for 30 sec - exiting 10:42:56 (17401): No heartbeat from core client for 30 sec - exiting 10:42:57 (17401): No heartbeat from core client for 30 sec - exiting 10:42:58 (17401): No heartbeat from core client for 30 sec - exiting 10:42:59 (17401): No heartbeat from core client for 30 sec - exiting 10:43:00 (17401): No heartbeat from core client for 30 sec - exiting 10:43:01 (17401): No heartbeat from core client for 30 sec - exiting 10:43:02 (17401): No heartbeat from core client for 30 sec - exiting 10:43:03 (17401): No heartbeat from core client for 30 sec - exiting 10:43:04 (17401): No heartbeat from core client for 30 sec - exiting 10:43:05 (17401): No heartbeat from core client for 30 sec - exiting 10:43:06 (17401): No heartbeat from core client for 30 sec - exiting 10:43:07 (17401): No heartbeat from core client for 30 sec - exiting 10:43:08 (17401): No heartbeat from core client for 30 sec - exiting 10:43:09 (17401): No heartbeat from core client for 30 sec - exiting 10:43:10 (17401): No heartbeat from core client for 30 sec - exiting 10:43:11 (17401): No heartbeat from core client for 30 sec - exiting 10:43:12 (17401): No heartbeat from core client for 30 sec - exiting 10:43:13 (17401): No heartbeat from core client for 30 sec - exiting 10:43:14 (17401): No heartbeat from core client for 30 sec - exiting 10:43:15 (17401): No heartbeat from core client for 30 sec - exiting 10:43:16 (17401): No heartbeat from core client for 30 sec - exiting 10:43:17 (17401): No heartbeat from core client for 30 sec - exiting 10:43:18 (17401): No heartbeat from core client for 30 sec - exiting 10:43:19 (17401): No heartbeat from core client for 30 sec - exiting 10:43:20 (17401): No heartbeat from core client for 30 sec - exiting 10:43:21 (17401): No heartbeat from core client for 30 sec - exiting 10:43:22 (17401): No heartbeat from core client for 30 sec - exiting 10:43:23 (17401): No heartbeat from core client for 30 sec - exiting 10:43:24 (17401): No heartbeat from core client for 30 sec - exiting 10:43:25 (17401): No heartbeat from core client for 30 sec - exiting 10:43:26 (17401): No heartbeat from core client for 30 sec - exiting 10:47:59 (20970): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:48:00 (20970): No heartbeat from core client for 30 sec - exiting 10:48:01 (20970): No heartbeat from core client for 30 sec - exiting 10:48:02 (20970): No heartbeat from core client for 30 sec - exiting 10:48:03 (20970): No heartbeat from core client for 30 sec - exiting 10:48:04 (20970): No heartbeat from core client for 30 sec - exiting 10:48:05 (20970): No heartbeat from core client for 30 sec - exiting 10:48:06 (20970): No heartbeat from core client for 30 sec - exiting 10:48:07 (20970): No heartbeat from core client for 30 sec - exiting 10:48:08 (20970): No heartbeat from core client for 30 sec - exiting 10:48:09 (20970): No heartbeat from core client for 30 sec - exiting 10:48:10 (20970): No heartbeat from core client for 30 sec - exiting 10:48:11 (20970): No heartbeat from core client for 30 sec - exiting 10:48:12 (20970): No heartbeat from core client for 30 sec - exiting 10:48:13 (20970): No heartbeat from core client for 30 sec - exiting 10:48:14 (20970): No heartbeat from core client for 30 sec - exiting 10:48:15 (20970): No heartbeat from core client for 30 sec - exiting 10:48:16 (20970): No heartbeat from core client for 30 sec - exiting 10:48:17 (20970): No heartbeat from core client for 30 sec - exiting 10:48:18 (20970): No heartbeat from core client for 30 sec - exiting 10:48:19 (20970): No heartbeat from core client for 30 sec - exiting 10:52:49 (22079): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:19:27 (23387): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:26:37 (29829): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:37:39 (32322): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 18:49:29 (20541): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:50:12 (25749): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:55:12 (26146): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:00:48 (27676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:00:49 (27676): No heartbeat from core client for 30 sec - exiting 19:00:50 (27676): No heartbeat from core client for 30 sec - exiting 19:00:51 (27676): No heartbeat from core client for 30 sec - exiting 19:00:52 (27676): No heartbeat from core client for 30 sec - exiting 19:00:53 (27676): No heartbeat from core client for 30 sec - exiting 19:00:54 (27676): No heartbeat from core client for 30 sec - exiting 19:00:55 (27676): No heartbeat from core client for 30 sec - exiting 19:00:56 (27676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 20:37:33 (12936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7737400] [0xf7737430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf755a1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf755d825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75454d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16161, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77b5400] [0xf77b5430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75d81df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75db825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75c34d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16161, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7717400] [0xf7717430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf753a1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf753d825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75254d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16161, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf76e6400] [0xf76e6430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75091df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf750c825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74f44d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16161, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7709400] [0xf7709430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf752c1df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf752f825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75174d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16161, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (9 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7716400] [0xf7716430] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75391df] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf753c825] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75244d3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16161, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Mar 2013 14:40:30	1270421	15655601	hadcm3n_3gun_1940_40_008262966_2	51,840	81,936	1.5806
10 Mar 2013 19:41:41	1270421	15655601	hadcm3n_3gun_1940_40_008262966_2	25,920	39,997	1.5431