Task 11654874

Name	famous_ucax_1999_200_006648788_5
Workunit	6852160
Created	13 Aug 2010, 8:17:44 UTC
Sent	13 Aug 2010, 8:30:57 UTC
Report deadline	12 Nov 2010, 15:58:08 UTC
Received	15 Sep 2010, 2:34:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1164431
Run time	13 days 14 hours 58 min 44 sec
CPU time	12 days 22 hours 7 min 48 sec
Validate state	Invalid
Credit	2,717.67
Device peak FLOPS	1.25 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4416, selfPID=4416, iMonCtr=1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 MainError: 02:49:15 AM No files match the supplied pattern. MainError: 02:49:15 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 1 received, exiting... (5624): called boinc_finish (3191): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7278, selfPID=7278, iMonCtr=1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... (8334): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (8334): No heartbeat from core client for 30 sec - exiting (8334): No heartbeat from core client for 30 sec - exiting (8334): No heartbeat from core client for 30 sec - exiting (10099): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (10115): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (10122): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 1 received, exiting... (10133): called boinc_finish (3162): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (3185): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... (6035): called boinc_finish (3243): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... (6059): called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x809e59c] [0xffffe420] ../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x804f906] ../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x805085a] ../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x8050ad6] /lib/libc.so.6(__libc_start_main+0xe0)[0xb7cbb390] ../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu(__gxx_personality_v0+0xe1)[0x804c449] Exiting... (3183): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... (9750): called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9748, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... (3165): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... (3316): called boinc_finish (3217): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 15 received, exiting... (3273): called boinc_finish (3190): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... (6131): called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu(boinc_catch_signal+0x58)[0x809e59c] [0xffffe420] ../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x804f906] ../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x805085a] ../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu[0x8050ad6] /lib/libc.so.6(__libc_start_main+0xe0)[0xb7c8f390] ../../projects/climateprediction.net/famous_6.11_i686-pc-linux-gnu(__gxx_personality_v0+0xe1)[0x804c449] Exiting... (5166): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... (5490): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6865, iMonCtr=1 Model crash detected, will try to restart... (6865): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8966, iMonCtr=1 Model crash detected, will try to restart... (8966): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (3213): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... OPEN: File Open Failed: Permission denied OPEN: Unable to Open File dataout/ucaxla#pa000002088c1+ for Read/Write Model crashed: PPCTL : Error opening preassigned PPfile tmp/pipe_dummy OPEN: File Open Failed: Permission denied OPEN: Unable to Open File dataout/ucaxla#pa000002088c1+ for Read/Write Model crashed: PPCTL : Error opening preassigned PPfile tmp/pipe_dummy OPEN: File Open Failed: Permission denied OPEN: Unable to Open File dataout/ucaxla#pa000002088c1+ for Read/Write Model crashed: PPCTL : Error opening preassigned PPfile tmp/pipe_dummy OPEN: File Open Failed: Permission denied OPEN: Unable to Open File dataout/ucaxla#pa000002088c1+ for Read/Write Model crashed: PPCTL : Error opening preassigned PPfile tmp/pipe_dummy OPEN: File Open Failed: Permission denied OPEN: Unable to Open File dataout/ucaxla#pa000002088c1+ for Read/Write Model crashed: PPCTL : Error opening preassigned PPfile tmp/pipe_dummy Sorry, too many model crashes! :-( (3223): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Sep 2010 20:22:09	909533	11654874	famous_ucax_1999_200_006648788_5	823,706	1,113,023	1.3512
14 Sep 2010 15:05:08	909533	11654874	famous_ucax_1999_200_006648788_5	814,346	1,099,453	1.3501
13 Sep 2010 19:44:51	909533	11654874	famous_ucax_1999_200_006648788_5	804,986	1,086,657	1.3499
13 Sep 2010 13:10:49	909533	11654874	famous_ucax_1999_200_006648788_5	795,626	1,074,621	1.3507
12 Sep 2010 18:02:23	909533	11654874	famous_ucax_1999_200_006648788_5	786,266	1,062,451	1.3513
12 Sep 2010 13:56:07	909533	11654874	famous_ucax_1999_200_006648788_5	776,906	1,048,062	1.3490
12 Sep 2010 09:50:11	909533	11654874	famous_ucax_1999_200_006648788_5	767,546	1,033,205	1.3461
12 Sep 2010 05:39:05	909533	11654874	famous_ucax_1999_200_006648788_5	758,186	1,018,633	1.3435
11 Sep 2010 22:19:58	909533	11654874	famous_ucax_1999_200_006648788_5	748,826	1,004,473	1.3414
11 Sep 2010 18:22:24	909533	11654874	famous_ucax_1999_200_006648788_5	739,466	991,948	1.3414
11 Sep 2010 13:03:04	909533	11654874	famous_ucax_1999_200_006648788_5	730,106	980,057	1.3423
11 Sep 2010 08:59:41	909533	11654874	famous_ucax_1999_200_006648788_5	720,746	969,193	1.3447
11 Sep 2010 02:57:56	909533	11654874	famous_ucax_1999_200_006648788_5	711,386	956,314	1.3443
10 Sep 2010 21:27:01	909533	11654874	famous_ucax_1999_200_006648788_5	702,026	943,775	1.3444
10 Sep 2010 17:29:32	909533	11654874	famous_ucax_1999_200_006648788_5	692,666	930,015	1.3427
10 Sep 2010 10:07:22	909533	11654874	famous_ucax_1999_200_006648788_5	683,306	916,603	1.3414
09 Sep 2010 00:05:49	909533	11654874	famous_ucax_1999_200_006648788_5	673,946	903,118	1.3400
08 Sep 2010 15:56:46	909533	11654874	famous_ucax_1999_200_006648788_5	664,586	890,748	1.3403
07 Sep 2010 21:19:32	909533	11654874	famous_ucax_1999_200_006648788_5	655,226	876,742	1.3381
07 Sep 2010 04:55:24	909533	11654874	famous_ucax_1999_200_006648788_5	645,866	864,102	1.3379