Task 15775902

Name	hadcm3n_zawr_1960_40_008365824_0
Workunit	8516683
Created	11 May 2013, 2:46:00 UTC
Sent	11 May 2013, 2:51:10 UTC
Report deadline	10 Aug 2013, 10:18:21 UTC
Received	15 May 2013, 19:48:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	861060
Run time	2 days 15 hours 10 min 55 sec
CPU time	2 days 13 hours 30 min 13 sec
Validate state	Invalid
Credit	1,244.16
Device peak FLOPS	2.67 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 20:06:49 (6081): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:06:51 (6081): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:32:59 (9588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:33:02 (9588): No heartbeat from core client for 30 sec - exiting 18:33:03 (9588): No heartbeat from core client for 30 sec - exiting 18:33:04 (9588): No heartbeat from core client for 30 sec - exiting 18:33:05 (9588): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 18:34:35 (9201): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:38:48 (9636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:39:25 (9700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:56:55 (9717): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:58:05 (12736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:17:27 (12797): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:52:59 (13154): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:26:32 (14448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:23 (15039): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:29:43 (15196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:30:52 (15297): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:32:51 (15331): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:35:55 (15410): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:48:56 (15475): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:51:43 (15677): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:58:28 (15723): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:37:19 (15854): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:11:07 (16522): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:03 (17751): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:14:37 (17794): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:15:17 (17836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:17:54 (17903): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:46:00 (17997): No heartbeat from core client for 30 sec - exiting 04:46:02 (17997): No heartbeat from core client for 30 sec - exiting 04:46:04 (17997): No heartbeat from core client for 30 sec - exiting 04:46:05 (17997): No heartbeat from core client for 30 sec - exiting 04:46:06 (17997): No heartbeat from core client for 30 sec - exiting 04:46:07 (17997): No heartbeat from core client for 30 sec - exiting 04:46:08 (17997): No heartbeat from core client for 30 sec - exiting 04:46:09 (17997): No heartbeat from core client for 30 sec - exiting 04:46:10 (17997): No heartbeat from core client for 30 sec - exiting 04:46:11 (17997): No heartbeat from core client for 30 sec - exiting 04:46:12 (17997): No heartbeat from core client for 30 sec - exiting 04:46:13 (17997): No heartbeat from core client for 30 sec - exiting 04:46:14 (17997): No heartbeat from core client for 30 sec - exiting 04:46:15 (17997): No heartbeat from core client for 30 sec - exiting 04:46:16 (17997): No heartbeat from core client for 30 sec - exiting 04:46:17 (17997): No heartbeat from core client for 30 sec - exiting 04:46:18 (17997): No heartbeat from core client for 30 sec - exiting 04:46:19 (17997): No heartbeat from core client for 30 sec - exiting 04:46:20 (17997): No heartbeat from core client for 30 sec - exiting 04:46:21 (17997): No heartbeat from core client for 30 sec - exiting 04:46:22 (17997): No heartbeat from core client for 30 sec - exiting 04:46:23 (17997): No heartbeat from core client for 30 sec - exiting 04:46:24 (17997): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/zawrko.pjg4c10 is not a valid UM file. Error converting file to netcdf: dataout/zawrko.pjg4c10 Error: Input file: dataout/zawrko.pig4c10 is not a valid UM file. Error converting file to netcdf: dataout/zawrko.pig4c10 Error: Input file: dataout/zawrko.pfg4c10 is not a valid UM file. Error converting file to netcdf: dataout/zawrko.pfg4c10 Error: Input file: dataout/zawrka.phg4c10 is not a valid UM file. Error converting file to netcdf: dataout/zawrka.phg4c10 Error: Input file: dataout/zawrka.pgg4c10 is not a valid UM file. Error converting file to netcdf: dataout/zawrka.pgg4c10 Error: Input file: dataout/zawrka.peg4c10 is not a valid UM file. Error converting file to netcdf: dataout/zawrka.peg4c10 Error: Input file: dataout/zawrka.pdg4c10 is not a valid UM file. Error converting file to netcdf: dataout/zawrka.pdg4c10 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:23:13 (24826): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:24:16 (31573): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:27:10 (31647): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:28:04 (31704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:30:48 (31767): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:34:06 (31827): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:36:54 (31900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:38:32 (31956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:29 (32017): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:44:34 (32090): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:45:45 (32137): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:46:51 (32184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:49:41 (32248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:51:20 (32303): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:35:25 (909): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:35:26 (909): No heartbeat from core client for 30 sec - exiting 14:21:16 (6564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:29:38 (7322): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:37:04 (7482): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:39:31 (7648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:40:37 (7705): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:41:40 (7783): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:42:30 (7847): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:43:20 (7875): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:44:49 (7901): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:44:50 (7901): No heartbeat from core client for 30 sec - exiting 14:44:51 (7901): No heartbeat from core client for 30 sec - exiting 14:44:52 (7901): No heartbeat from core client for 30 sec - exiting 14:44:53 (7901): No heartbeat from core client for 30 sec - exiting 14:44:54 (7901): No heartbeat from core client for 30 sec - exiting 14:44:55 (7901): No heartbeat from core client for 30 sec - exiting 14:44:56 (7901): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77a3400] [0xf77a3430] /lib32/libc.so.6(gsignal+0x51)[0xf760f951] /lib32/libc.so.6(abort+0x182)[0xf7612d82] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75fbbd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7958, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf76e9400] [0xf76e9430] /lib32/libc.so.6(gsignal+0x51)[0xf7555951] /lib32/libc.so.6(abort+0x182)[0xf7558d82] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7541bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7958, iMonCtr=1 Model crash detected, will try to restart... 14:45:48 (7958): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:45:49 (7958): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77a7400] [0xf77a7430] /lib32/libc.so.6(gsignal+0x51)[0xf7613951] /lib32/libc.so.6(abort+0x182)[0xf7616d82] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75ffbd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7985, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77ca400] [0xf77ca430] /lib32/libc.so.6(gsignal+0x51)[0xf7636951] /lib32/libc.so.6(abort+0x182)[0xf7639d82] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7622bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7985, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7779400] [0xf7779430] /lib32/libc.so.6(gsignal+0x51)[0xf75e5951] /lib32/libc.so.6(abort+0x182)[0xf75e8d82] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75d1bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7985, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf770a400] [0xf770a430] /lib32/libc.so.6(gsignal+0x51)[0xf7576951] /lib32/libc.so.6(abort+0x182)[0xf7579d82] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7562bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7985, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish SIGSEGV: segmentation violation Stack trace (10 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf774d400] /lib32/libc.so.6(getenv+0x64)[0xf74a9c14] /lib32/libc.so.6(+0x90a10)[0xf750ba10] /lib32/libc.so.6(+0x90bc1)[0xf750bbc1] /lib32/libc.so.6(localtime_r+0x2c)[0xf750a2fc] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80b0d9c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80b2dc4] /lib32/libpthread.so.0(+0x596e)[0xf771f96e] /lib32/libc.so.6(clone+0x5e)[0xf755050e] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 May 2013 09:50:54	861060	15775902	hadcm3n_zawr_1960_40_008365824_0	103,680	192,877	1.8603
12 May 2013 19:31:08	861060	15775902	hadcm3n_zawr_1960_40_008365824_0	77,760	142,744	1.8357
12 May 2013 05:19:52	861060	15775902	hadcm3n_zawr_1960_40_008365824_0	51,840	93,595	1.8055
11 May 2013 16:11:16	861060	15775902	hadcm3n_zawr_1960_40_008365824_0	25,920	46,862	1.8079