Task 21815722

Name	hadam4_a1pw_209710_6_853_011942263_0
Workunit	11942263
Created	28 Nov 2019, 6:25:36 UTC
Sent	18 Dec 2019, 15:22:44 UTC
Report deadline	29 Nov 2020, 20:42:44 UTC
Received	31 Dec 2019, 15:13:52 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1495246
Run time	4 days 7 hours 29 min 47 sec
CPU time	4 days 1 hours 15 min 43 sec
Validate state	Invalid
Credit	4,827.12
Device peak FLOPS	4.13 GFLOPS
Application version	UK Met Office HadAM4 at N144 resolution v8.09 i686-pc-linux-gnu
Peak working set size	627.94 MB
Peak swap size	650.50 MB
Peak disk usage	5.79 MB
Stderr	<core_client_version>7.9.3</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234)</message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received: Software termination signal from kill Signal 15 received: Abnormal termination triggered by abort call Signal 15 received, exiting... 00:07:28 (5507): called boinc_finish(193) Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 2 received: Interrupt Signal 2 received: Illegal instruction - invalid function image Signal 2 received: Floating point exception Signal 2 received: Segment violation Signal 2 received: Software termination signal from kill Signal 2 received: Abnormal termination triggered by abort call Signal 2 received, exiting... 11:46:36 (4585): called boinc_finish(193) Signal 2 received: Interrupt Signal 2 received: Illegal instruction - invalid function image Signal 2 received: Floating point exception Signal 2 received: Segment violation Signal 2 received: Software termination signal from kill Signal 2 received: Abnormal termination triggered by abort call Signal 2 received, exiting... SIGSEGV: segmentation violation Stack trace (19 frames): ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x80d4cf7] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7f4c090] /lib/i386-linux-gnu/libc.so.6(getenv+0x99)[0xf7ab1a69] /lib/i386-linux-gnu/libc.so.6(+0xaf468)[0xf7b31468] /lib/i386-linux-gnu/libc.so.6(+0xaf835)[0xf7b31835] /lib/i386-linux-gnu/libc.so.6(localtime_r+0x12)[0xf7b2fdb2] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d01b2] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d0900] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d09f1] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7f4c090] linux-gate.so.1(__kernel_vsyscall+0x9)[0xf7f4c079] /lib/i386-linux-gnu/libc.so.6(nanosleep+0x4b)[0xf7b411cb] /lib/i386-linux-gnu/libc.so.6(usleep+0x41)[0xf7b73be1] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80e78a5] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80503e8] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x8051b13] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x8051d8b] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0xf7a9ae81] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x804cd21] Exiting... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:22:49 (3066): BOINC client no longer exists - exiting 22:22:49 (3066): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 2 received: Interrupt Signal 2 received: Illegal instruction - invalid function image Signal 2 received: Floating point exception Signal 2 received: Segment violation Signal 2 received: Software termination signal from kill Signal 2 received: Abnormal termination triggered by abort call Signal 2 received, exiting... 03:03:54 (11199): called boinc_finish(193) Signal 2 received: Interrupt Signal 2 received: Illegal instruction - invalid function image Signal 2 received: Floating point exception Signal 2 received: Segment violation Signal 2 received: Software termination signal from kill Signal 2 received: Abnormal termination triggered by abort call Signal 2 received, exiting... SIGSEGV: segmentation violation Stack trace (19 frames): ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x80d4cf7] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7f87090] /lib/i386-linux-gnu/libc.so.6(getenv+0x99)[0xf7aeca69] /lib/i386-linux-gnu/libc.so.6(+0xaf468)[0xf7b6c468] /lib/i386-linux-gnu/libc.so.6(+0xaf835)[0xf7b6c835] /lib/i386-linux-gnu/libc.so.6(localtime_r+0x12)[0xf7b6adb2] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d01b2] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d0900] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80d09f1] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7f87090] linux-gate.so.1(__kernel_vsyscall+0x9)[0xf7f87079] /lib/i386-linux-gnu/libc.so.6(nanosleep+0x4b)[0xf7b7c1cb] /lib/i386-linux-gnu/libc.so.6(usleep+0x41)[0xf7baebe1] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80e78a5] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x80503e8] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x8051b13] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x8051d8b] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0xf7ad5e81] ../../projects/climateprediction.net/hadam4_8.09_i686-pc-linux-gnu[0x804cd21] Exiting... OPEN: File Open Failed: Permission denied OPEN: Unable to Open File dataout/a1pwga.pat8jan for Read/Write Model crashed: PPCTL : Error opening preassigned PPfile tmp/xnnuj.pipe_dummy OPEN: File Open Failed: Permission denied OPEN: Unable to Open File dataout/a1pwga.pat8jan for Read/Write Model crashed: PPCTL : Error opening preassigned PPfile tmp/xnnuj.pipe_dummy OPEN: File Open Failed: Permission denied OPEN: Unable to Open File dataout/a1pwga.pat8jan for Read/Write Model crashed: PPCTL : Error opening preassigned PPfile tmp/xnnuj.pipe_dummy OPEN: File Open Failed: Permission denied OPEN: Unable to Open File dataout/a1pwga.pat8jan for Read/Write Model crashed: PPCTL : Error opening preassigned PPfile tmp/xnnuj.pipe_dummy OPEN: File Open Failed: Permission denied OPEN: Unable to Open File dataout/a1pwga.pat8jan for Read/Write Model crashed: PPCTL : Error opening preassigned PPfile tmp/xnnuj.pipe_dummy Sorry, too many model crashes! :-( 16:09:25 (4643): called boinc_finish(22) </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Dec 2019 21:47:12	1495246	21815722	hadam4_a1pw_209710_6_853_011942263_0	13,061	320,983	24.5757
23 Dec 2019 02:23:49	1495246	21815722	hadam4_a1pw_209710_6_853_011942263_0	8,741	215,547	24.6593
19 Dec 2019 23:07:05	1495246	21815722	hadam4_a1pw_209710_6_853_011942263_0	4,421	109,651	24.8023