Task 22027644

Name	hadam4h_d01a_206711_5_897_012067267_0
Workunit	12067267
Created	1 Mar 2021, 13:22:42 UTC
Sent	17 Mar 2021, 22:36:54 UTC
Report deadline	28 Feb 2022, 3:56:54 UTC
Received	11 Apr 2021, 20:38:38 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	12 (0x0000000C) Unknown error code
Computer ID	1241909
Run time	3 days 11 hours 52 min 26 sec
CPU time	3 days 10 hours 20 min 3 sec
Validate state	Invalid
Credit	0.00
Device peak FLOPS	3.53 GFLOPS
Application version	UK Met Office HadAM4 at N216 resolution v8.52 i686-pc-linux-gnu
Peak working set size	1,365.55 MB
Peak swap size	1,386.04 MB
Peak disk usage	12.92 MB
Stderr	<core_client_version>7.16.11</core_client_version> <![CDATA[ <message> process exited with code 12 (0xc, -244)</message> <stderr_txt> CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc/projects/climateprediction.net/hadam4h_d01a_206711_5_897_012067267/dataout/xnnuj.out Image PC Routine Line Source hadam4_um_8.52_i6 083F6605 Unknown Unknown Unknown hadam4_um_8.52_i6 0843ABB0 Unknown Unknown Unknown hadam4_um_8.52_i6 08438E0F Unknown Unknown Unknown hadam4_um_8.52_i6 081ABA99 Unknown Unknown Unknown hadam4_um_8.52_i6 0811E285 Unknown Unknown Unknown hadam4_um_8.52_i6 0811FF0C Unknown Unknown Unknown hadam4_um_8.52_i6 081D82A7 Unknown Unknown Unknown hadam4_um_8.52_i6 081E121B Unknown Unknown Unknown hadam4_um_8.52_i6 0836E63F Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15894, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: STWORK : Error in PP_FILE tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: STWORK : Error in PP_FILE tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy Signal 15 received: Software termination signal from kill Signal 15 received: Abnormal termination triggered by abort call Signal 15 received, exiting... SIGSEGV: segmentation violation Stack trace (19 frames): ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x80d4cf7] [0xf7f67ba0] /lib/libc.so.6(getenv+0x72)[0xf7c42de2] /lib/libc.so.6(+0xaf21e)[0xf7cc021e] /lib/libc.so.6(+0xafb2f)[0xf7cc0b2f] /lib/libc.so.6(localtime_r+0x2b)[0xf7cbed3b] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80d01b2] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80d0900] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80d09f1] [0xf7f67ba0] [0xf7f67b89] /lib/libc.so.6(nanosleep+0x46)[0xf7ccf8c6] /lib/libc.so.6(usleep+0x3d)[0xf7d05e8d] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80e78a5] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80503e8] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x8051b13] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x8051d8b] /lib/libc.so.6(__libc_start_main+0xf3)[0xf7c2b2d3] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x804cd21] Exiting... Signal 15 received: Software termination signal from kill Signal 15 received: Abnormal termination triggered by abort call Signal 15 received, exiting... SIGSEGV: segmentation violation Stack trace (19 frames): ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x80d4cf7] [0xf7f91ba0] /lib/libc.so.6(getenv+0x72)[0xf7c6cde2] /lib/libc.so.6(+0xaf21e)[0xf7cea21e] /lib/libc.so.6(+0xafb2f)[0xf7ceab2f] /lib/libc.so.6(localtime_r+0x2b)[0xf7ce8d3b] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80d01b2] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80d0900] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80d09f1] [0xf7f91ba0] [0xf7f91b89] /lib/libc.so.6(nanosleep+0x46)[0xf7cf98c6] /lib/libc.so.6(usleep+0x3d)[0xf7d2fe8d] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80e78a5] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80503e8] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x8051b13] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x8051d8b] /lib/libc.so.6(__libc_start_main+0xf3)[0xf7c552d3] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x804cd21] Exiting... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA OPEN: File Creation Failed: No such file or directory OPEN: Unable to Open File dataout/d01aga.pbq7nov for Read/Write Model crashed: STWORK : Error opening output PP file on unit 61 tmp/xnnuj.pipe_dummy cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadam4h_d01a_206711_5_897_012067267/jobs/xnnuj.ihist after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadam4h_d01a_206711_5_897_012067267/jobs/xnnuj.namelists after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadam4h_d01a_206711_5_897_012067267/dataout/atmos_restart.day after 11 attempts forrtl: Bad file descriptor forrtl: severe (30): open failure, unit 6, file /proc/9132/fd/ Image PC Routine Line Source hadam4_um_8.52_i6 083F6605 Unknown Unknown Unknown hadam4_um_8.52_i6 0843E0A0 Unknown Unknown Unknown hadam4_um_8.52_i6 081DF6C9 Unknown Unknown Unknown hadam4_um_8.52_i6 0836E63F Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5776, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadam4h_d01a_206711_5_897_012067267/jobs/xnnuj.ihist after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadam4h_d01a_206711_5_897_012067267/jobs/xnnuj.namelists after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadam4h_d01a_206711_5_897_012067267/dataout/atmos_restart.day after 11 attempts forrtl: Bad file descriptor forrtl: severe (30): open failure, unit 6, file /proc/9138/fd/ Image PC Routine Line Source hadam4_um_8.52_i6 083F6605 Unknown Unknown Unknown hadam4_um_8.52_i6 0843E0A0 Unknown Unknown Unknown hadam4_um_8.52_i6 081DF6C9 Unknown Unknown Unknown hadam4_um_8.52_i6 0836E63F Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5776, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadam4h_d01a_206711_5_897_012067267/jobs/xnnuj.ihist after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadam4h_d01a_206711_5_897_012067267/jobs/xnnuj.namelists after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadam4h_d01a_206711_5_897_012067267/dataout/atmos_restart.day after 11 attempts forrtl: Bad file descriptor forrtl: severe (30): open failure, unit 6, file /proc/9144/fd/ Image PC Routine Line Source hadam4_um_8.52_i6 083F6605 Unknown Unknown Unknown hadam4_um_8.52_i6 0843E0A0 Unknown Unknown Unknown hadam4_um_8.52_i6 081DF6C9 Unknown Unknown Unknown hadam4_um_8.52_i6 0836E63F Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5776, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadam4h_d01a_206711_5_897_012067267/jobs/xnnuj.ihist after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadam4h_d01a_206711_5_897_012067267/jobs/xnnuj.namelists after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadam4h_d01a_206711_5_897_012067267/dataout/atmos_restart.day after 11 attempts forrtl: Bad file descriptor forrtl: severe (30): open failure, unit 6, file /proc/9153/fd/ Image PC Routine Line Source hadam4_um_8.52_i6 083F6605 Unknown Unknown Unknown hadam4_um_8.52_i6 0843E0A0 Unknown Unknown Unknown hadam4_um_8.52_i6 081DF6C9 Unknown Unknown Unknown hadam4_um_8.52_i6 0836E63F Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5776, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( </stderr_txt> ]]>