Task 21788182

Name	hadam4_a1s8_201310_6_848_011925487_0
Workunit	11925487
Created	5 Nov 2019, 6:56:37 UTC
Sent	6 Nov 2019, 15:58:31 UTC
Report deadline	18 Oct 2020, 21:18:31 UTC
Received	3 Dec 2019, 22:12:59 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1462145
Run time	12 days 16 hours 35 min 40 sec
CPU time	12 days 11 hours 21 min 16 sec
Validate state	Invalid
Credit	3,230.53
Device peak FLOPS	1.13 GFLOPS
Application version	UK Met Office HadAM4 at N144 resolution v8.09 i686-pc-linux-gnu
Peak working set size	626.12 MB
Peak swap size	650.50 MB
Peak disk usage	5.79 MB
Stderr	<core_client_version>7.9.3</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234)</message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2769, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2769, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received: Software termination signal from kill Signal 15 received: Abnormal termination triggered by abort call Signal 15 received, exiting... SIGSEGV: segmentation violation Stack trace (19 frames): ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x80d4cf7] linux-gate.so.1(__kernel_sigreturn+0x0)[0xb7f65d14] /lib/i386-linux-gnu/libc.so.6(getenv+0x99)[0xb7acea69] /lib/i386-linux-gnu/libc.so.6(+0xaf468)[0xb7b4e468] /lib/i386-linux-gnu/libc.so.6(+0xaf835)[0xb7b4e835] /lib/i386-linux-gnu/libc.so.6(localtime_r+0x12)[0xb7b4cdb2] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d01b2] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d0900] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d09f1] linux-gate.so.1(__kernel_sigreturn+0x0)[0xb7f65d14] linux-gate.so.1(__kernel_vsyscall+0x9)[0xb7f65d09] /lib/i386-linux-gnu/libc.so.6(nanosleep+0x4b)[0xb7b5e1cb] /lib/i386-linux-gnu/libc.so.6(usleep+0x41)[0xb7b90be1] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80e78a5] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80503e8] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x8051b13] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x8051d8b] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0xb7ab7e81] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x804cd21] Exiting... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (8 frames): /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x83d8117] linux-gate.so.1(__kernel_sigreturn+0x0)[0x1ff17d14] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81142af] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x811f6bf] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81d82a7] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81e121b] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x836e63f] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x1fc0de81] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2028, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2028, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2028, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2028, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received: Software termination signal from kill Signal 15 received: Abnormal termination triggered by abort call Signal 15 received, exiting... SIGSEGV: segmentation violation Stack trace (19 frames): ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x80d4cf7] linux-gate.so.1(__kernel_sigreturn+0x0)[0xb7fcfd14] /lib/i386-linux-gnu/libc.so.6(getenv+0x99)[0xb7b38a69] /lib/i386-linux-gnu/libc.so.6(+0xaf468)[0xb7bb8468] /lib/i386-linux-gnu/libc.so.6(+0xaf835)[0xb7bb8835] /lib/i386-linux-gnu/libc.so.6(localtime_r+0x12)[0xb7bb6db2] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d01b2] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d0900] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d09f1] linux-gate.so.1(__kernel_sigreturn+0x0)[0xb7fcfd14] linux-gate.so.1(__kernel_vsyscall+0x9)[0xb7fcfd09] /lib/i386-linux-gnu/libc.so.6(nanosleep+0x4b)[0xb7bc81cb] /lib/i386-linux-gnu/libc.so.6(usleep+0x41)[0xb7bfabe1] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80e78a5] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80503e8] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x8051b13] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x8051d8b] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0xb7b21e81] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x804cd21] Exiting... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received: Software termination signal from kill Signal 15 received: Abnormal termination triggered by abort call Signal 15 received, exiting... SIGSEGV: segmentation violation Stack trace (19 frames): ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x80d4cf7] linux-gate.so.1(__kernel_sigreturn+0x0)[0xb7f44d14] /lib/i386-linux-gnu/libc.so.6(getenv+0x99)[0xb7aada69] /lib/i386-linux-gnu/libc.so.6(+0xaf468)[0xb7b2d468] /lib/i386-linux-gnu/libc.so.6(+0xaf835)[0xb7b2d835] /lib/i386-linux-gnu/libc.so.6(localtime_r+0x12)[0xb7b2bdb2] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d01b2] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d0900] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d09f1] linux-gate.so.1(__kernel_sigreturn+0x0)[0xb7f44d14] linux-gate.so.1(__kernel_vsyscall+0x9)[0xb7f44d09] /lib/i386-linux-gnu/libc.so.6(nanosleep+0x4b)[0xb7b3d1cb] /lib/i386-linux-gnu/libc.so.6(usleep+0x41)[0xb7b6fbe1] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80e78a5] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80503e8] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x8051b13] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x8051d8b] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0xb7a96e81] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x804cd21] Exiting... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (8 frames): /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x83d8117] linux-gate.so.1(__kernel_sigreturn+0x0)[0x1ff3fd14] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81142af] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x811f6bf] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81d82a7] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81e121b] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x836e63f] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x1fc35e81] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1803, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (8 frames): /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x83d8117] linux-gate.so.1(__kernel_sigreturn+0x0)[0x1ffced14] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81142af] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x811f6bf] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81d82a7] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81e121b] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x836e63f] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x1fcc4e81] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1803, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (8 frames): /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x83d8117] linux-gate.so.1(__kernel_sigreturn+0x0)[0x1ff76d14] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81142af] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x811f6bf] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81d82a7] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81e121b] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x836e63f] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x1fc6ce81] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1803, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (8 frames): /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x83d8117] linux-gate.so.1(__kernel_sigreturn+0x0)[0x1fedad14] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81142af] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x811f6bf] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81d82a7] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81e121b] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x836e63f] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x1fbd0e81] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1803, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (8 frames): /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x83d8117] linux-gate.so.1(__kernel_sigreturn+0x0)[0x1fee8d14] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81142af] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x811f6bf] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81d82a7] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81e121b] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x836e63f] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x1fbdee81] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1803, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... SIGSEGV: segmentation violation Stack trace (8 frames): /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x83d8117] linux-gate.so.1(__kernel_sigreturn+0x0)[0x1ff7fd14] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81142af] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x811f6bf] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81d82a7] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x81e121b] /var/lib/boinc-client/projects/climateprediction.net/hadam4_um_8.09_i686-pc-linux-gnu[0x836e63f] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x1fc75e81] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1803, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( 15:44:19 (1803): called boinc_finish(22) </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Nov 2019 18:30:08	1462145	21788182	hadam4_a1s8_201310_6_848_011925487_0	8,741	1,090,027	124.7028
13 Nov 2019 18:34:48	1462145	21788182	hadam4_a1s8_201310_6_848_011925487_0	4,421	556,195	125.8075