Task 11853078

Name	famous_vpcc_799_200_006718610_2
Workunit	6921863
Created	26 Aug 2010, 18:20:56 UTC
Sent	13 Sep 2010, 6:09:26 UTC
Report deadline	13 Dec 2010, 13:36:37 UTC
Received	13 Sep 2010, 11:55:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	904604
Run time
CPU time	1 hours 40 min 19 sec
Validate state	Invalid
Credit	123.61
Device peak FLOPS	2.16 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-apple-darwin
Stderr	<core_client_version>6.2.18</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> (23718): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (23724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (23724): No heartbeat from core client for 30 sec - exiting (23724): No heartbeat from core client for 30 sec - exiting (23724): No heartbeat from core client for 30 sec - exiting (23724): No heartbeat from core client for 30 sec - exiting (23724): No heartbeat from core client for 30 sec - exiting (23724): No heartbeat from core client for 30 sec - exiting (23724): No heartbeat from core client for 30 sec - exiting (23724): No heartbeat from core client for 30 sec - exiting (23783): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (23783): No heartbeat from core client for 30 sec - exiting (23783): No heartbeat from core client for 30 sec - exiting (23783): No heartbeat from core client for 30 sec - exiting (23797): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (23815): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (23893): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (23893): No heartbeat from core client for 30 sec - exiting (23925): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (23964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (23989): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24006): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24026): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24084): No heartbeat from core client for 30 sec - exiting (24084): No heartbeat from core client for 30 sec - exiting (24084): No heartbeat from core client for 30 sec - exiting (24084): No heartbeat from core client for 30 sec - exiting (24084): No heartbeat from core client for 30 sec - exiting (24084): No heartbeat from core client for 30 sec - exiting (24084): No heartbeat from core client for 30 sec - exiting (24084): No heartbeat from core client for 30 sec - exiting (24084): No heartbeat from core client for 30 sec - exiting (24084): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24118): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting (24138): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... (24207): No heartbeat from core client for 30 sec - exiting (24207): No heartbeat from core client for 30 sec - exiting (24207): No heartbeat from core client for 30 sec - exiting (24207): No heartbeat from core client for 30 sec - exiting (24207): No heartbeat from core client for 30 sec - exiting (24207): No heartbeat from core client for 30 sec - exiting (24207): No heartbeat from core client for 30 sec - exiting (24207): No heartbeat from core client for 30 sec - exiting (24207): No heartbeat from core client for 30 sec - exiting (24207): No heartbeat from core client for 30 sec - exiting (24207): No heartbeat from core client for 30 sec - exiting (24207): No heartbeat from core client for 30 sec - exiting (24207): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24245): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24245): No heartbeat from core client for 30 sec - exiting (24245): No heartbeat from core client for 30 sec - exiting (24245): No heartbeat from core client for 30 sec - exiting (24245): No heartbeat from core client for 30 sec - exiting (24245): No heartbeat from core client for 30 sec - exiting (24245): No heartbeat from core client for 30 sec - exiting (24245): No heartbeat from core client for 30 sec - exiting (24245): No heartbeat from core client for 30 sec - exiting (24245): No heartbeat from core client for 30 sec - exiting (24245): No heartbeat from core client for 30 sec - exiting (24245): No heartbeat from core client for 30 sec - exiting (24245): No heartbeat from core client for 30 sec - exiting (24270): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24284): No heartbeat from core client for 30 sec - exiting (24284): No heartbeat from core client for 30 sec - exiting (24284): No heartbeat from core client for 30 sec - exiting (24284): No heartbeat from core client for 30 sec - exiting (24284): No heartbeat from core client for 30 sec - exiting (24284): No heartbeat from core client for 30 sec - exiting (24284): No heartbeat from core client for 30 sec - exiting (24284): No heartbeat from core client for 30 sec - exiting (24284): No heartbeat from core client for 30 sec - exiting (24284): No heartbeat from core client for 30 sec - exiting (24284): No heartbeat from core client for 30 sec - exiting (24305): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24321): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24412): No heartbeat from core client for 30 sec - exiting (24412): No heartbeat from core client for 30 sec - exiting (24412): No heartbeat from core client for 30 sec - exiting (24412): No heartbeat from core client for 30 sec - exiting (24412): No heartbeat from core client for 30 sec - exiting (24412): No heartbeat from core client for 30 sec - exiting (24412): No heartbeat from core client for 30 sec - exiting (24412): No heartbeat from core client for 30 sec - exiting (24412): No heartbeat from core client for 30 sec - exiting (24412): No heartbeat from core client for 30 sec - exiting (24412): No heartbeat from core client for 30 sec - exiting (24412): No heartbeat from core client for 30 sec - exiting (24412): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24447): No heartbeat from core client for 30 sec - exiting (24469): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24487): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24507): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24507): No heartbeat from core client for 30 sec - exiting (24507): No heartbeat from core client for 30 sec - exiting (24571): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24571): No heartbeat from core client for 30 sec - exiting (24571): No heartbeat from core client for 30 sec - exiting (24571): No heartbeat from core client for 30 sec - exiting (24571): No heartbeat from core client for 30 sec - exiting (24571): No heartbeat from core client for 30 sec - exiting (24571): No heartbeat from core client for 30 sec - exiting (24571): No heartbeat from core client for 30 sec - exiting (24571): No heartbeat from core client for 30 sec - exiting (24590): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24621): No heartbeat from core client for 30 sec - exiting (24621): No heartbeat from core client for 30 sec - exiting (24621): No heartbeat from core client for 30 sec - exiting (24621): No heartbeat from core client for 30 sec - exiting (24621): No heartbeat from core client for 30 sec - exiting (24621): No heartbeat from core client for 30 sec - exiting (24621): No heartbeat from core client for 30 sec - exiting (24621): No heartbeat from core client for 30 sec - exiting (24621): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (24697): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24774): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24787): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24787): No heartbeat from core client for 30 sec - exiting (24812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24812): No heartbeat from core client for 30 sec - exiting (24896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (24973): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (25018): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (25043): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (25095): No heartbeat from core client for 30 sec - exiting (25095): No heartbeat from core client for 30 sec - exiting (25095): No heartbeat from core client for 30 sec - exiting (25095): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (25095): No heartbeat from core client for 30 sec - exiting BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (25421): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (25440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (25440): No heartbeat from core client for 30 sec - exiting (25440): No heartbeat from core client for 30 sec - exiting BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (25461): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting (25461): No heartbeat from core client for 30 sec - exiting BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy (25494): No heartbeat from core client for 30 sec - exiting (25494): No heartbeat from core client for 30 sec - exiting (25494): No heartbeat from core client for 30 sec - exiting (25494): No heartbeat from core client for 30 sec - exiting (25494): No heartbeat from core client for 30 sec - exiting (25494): No heartbeat from core client for 30 sec - exiting (25494): No heartbeat from core client for 30 sec - exiting (25494): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy Sorry, too many model crashes! :-( (25514): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Sep 2010 11:42:52	904604	11853078	famous_vpcc_799_200_006718610_2	37,466	5,623	0.1501
13 Sep 2010 10:38:52	904604	11853078	famous_vpcc_799_200_006718610_2	28,106	4,028	0.1433
13 Sep 2010 09:34:03	904604	11853078	famous_vpcc_799_200_006718610_2	18,746	2,614	0.1394
13 Sep 2010 08:25:04	904604	11853078	famous_vpcc_799_200_006718610_2	9,386	1,438	0.1532