Task 18625987

Name	hadcm3s_9tc7_1986_2_009894404_1
Workunit	9923891
Created	25 Jun 2015, 13:06:30 UTC
Sent	25 Jun 2015, 13:08:56 UTC
Report deadline	24 Sep 2015, 20:36:07 UTC
Received	3 Oct 2015, 7:49:09 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1325929
Run time	21 hours 30 min 55 sec
CPU time	20 hours 55 min 59 sec
Validate state	Invalid
Credit	311.04
Device peak FLOPS	3.43 GFLOPS
Application version	UK Met Office HadCM3 short v7.24 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> MonID=3548, UMPID=3554, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1933, UMPID=2277, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1844, UMPID=2253, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1799, UMPID=1879, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=3028, UMPID=3031, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=3053, UMPID=3105, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=2588, UMPID=2590, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1841, UMPID=1869, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1811, UMPID=1818, RM3PID=0 MonID=1834, UMPID=1864, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1861, UMPID=1894, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1892, UMPID=1904, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1876, UMPID=1911, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=2587, UMPID=2589, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1845, UMPID=1879, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1836, UMPID=1874, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1838, UMPID=1875, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=2546, UMPID=2548, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1838, UMPID=1879, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1805, UMPID=1839, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=2621, UMPID=2623, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=2570, UMPID=2572, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1838, UMPID=1841, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=2583, UMPID=2585, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1863, UMPID=1897, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1835, UMPID=1870, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=2984, UMPID=2986, RM3PID=0 CPDN Monitor - Quit request from BOINC... MonID=1910, UMPID=1914, RM3PID=0 MonID=1832, UMPID=1862, RM3PID=0 17:37:40 (1832): No heartbeat from client for 30 sec - exiting 17:37:40 (1832): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... MonID=5042, UMPID=5051, RM3PID=0 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... MonID=1812, UMPID=1830, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 18:48:31 (1812): No heartbeat from client for 30 sec - exiting 18:48:31 (1812): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... MonID=2561, UMPID=2563, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 18:49:42 (2561): No heartbeat from client for 30 sec - exiting 18:49:42 (2561): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... MonID=2960, UMPID=2962, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 19:03:05 (2960): No heartbeat from client for 30 sec - exiting 19:03:05 (2960): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... MonID=4221, UMPID=4223, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... MonID=2587, UMPID=2589, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 13:21:57 (2587): No heartbeat from client for 30 sec - exiting 13:21:57 (2587): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... MonID=2593, UMPID=2595, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 13:23:09 (2593): No heartbeat from client for 30 sec - exiting 13:23:09 (2593): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... MonID=2996, UMPID=2998, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... MonID=2620, UMPID=2622, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 MonID=2620, UMPID=4721, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 MonID=2620, UMPID=4828, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 MonID=2620, UMPID=4958, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... MonID=2571, UMPID=2573, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 MonID=2571, UMPID=3374, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 MonID=2571, UMPID=26522, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... MonID=2642, UMPID=2644, RM3PID=0 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( SIGSEGV: segmentation violation Stack trace (14 frames): ../../projects/climateprediction.net/hadcm3s_7.24_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80bd86f] [0xf7762410] /lib/i386-linux-gnu/libc.so.6(getenv+0x91)[0xf747b261] /lib/i386-linux-gnu/libc.so.6(+0xa9540)[0xf74f2540] /lib/i386-linux-gnu/libc.so.6(+0xa98b2)[0xf74f28b2] /lib/i386-linux-gnu/libc.so.6(localtime_r+0x2b)[0xf74f0fab] ../../projects/climateprediction.net/hadcm3s_7.24_i686-pc-linux-gnu[0x80b8ea6] ../../projects/climateprediction.net/hadcm3s_7.24_i686-pc-linux-gnu[0x80b9356] ../../projects/climateprediction.net/hadcm3s_7.24_i686-pc-linux-gnu[0x804fbfb] ../../projects/climateprediction.net/hadcm3s_7.24_i686-pc-linux-gnu[0x805051b] ../../projects/climateprediction.net/hadcm3s_7.24_i686-pc-linux-gnu[0x80512eb] ../../projects/climateprediction.net/hadcm3s_7.24_i686-pc-linux-gnu[0x805153c] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74624d3] ../../projects/climateprediction.net/hadcm3s_7.24_i686-pc-linux-gnu(__gxx_personality_v0+0x191)[0x804cc91] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Sep 2015 15:40:49	1325929	18625987	hadcm3s_9tc7_1986_2_009894404_1	25,920	70,846	2.7333