Task 15420174

Name	hadcm3n_zgi0_1880_40_008242799_2
Workunit	8397923
Created	30 Oct 2012, 16:41:03 UTC
Sent	30 Oct 2012, 16:41:14 UTC
Report deadline	30 Jan 2013, 0:08:25 UTC
Received	14 Dec 2012, 21:40:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1193569
Run time	23 days 3 hours 7 min 43 sec
CPU time	20 days 17 hours 29 min 46 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(255,0xa0a6b540) malloc: * error for object 0x1057404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(255,0xa0a6b540) malloc: * error for object 0x1057400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(255,0xa0a6b540) malloc: * error for object 0x859a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(255,0xa0a6b540) malloc: * error for object 0x859a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation SIGSEGV: segmentation violation Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish hadcm3n_6.07_i686-apple-darwin(244,0xa0a6b540) malloc: * error for object 0x1038604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(244,0xa0a6b540) malloc: * error for object 0x82a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(244,0xa0a6b540) malloc: * error for object 0x82a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(244,0xa0a6b540) malloc: * error for object 0x82a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(244,0xa0a6b540) malloc: * error for object 0x82a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(244,0xa0a6b540) malloc: * error for object 0x82a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(244,0xa0a6b540) malloc: * error for object 0x82a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(244,0xa0a6b540) malloc: * error for object 0x82a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(244,0xa0a6b540) malloc: * error for object 0x82a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(244,0xa0a6b540) malloc: * error for object 0x82a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish hadcm3n_6.07_i686-apple-darwin(262,0xa0a6b540) malloc: * error for object 0x1037c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(262,0xa0a6b540) malloc: * error for object 0x841004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(262,0xa0a6b540) malloc: * error for object 0x841004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(262,0xa0a6b540) malloc: * error for object 0x82a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(262,0xa0a6b540) malloc: * error for object 0x82a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(262,0xa0a6b540) malloc: * error for object 0x82a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(262,0xa0a6b540) malloc: * error for object 0x82a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(262,0xa0a6b540) malloc: * error for object 0x82a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(262,0xa0a6b540) malloc: * error for object 0x82a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(262,0xa0a6b540) malloc: * error for object 0x82a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(262,0xa0a6b540) malloc: * error for object 0x82a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(262,0xa0a6b540) malloc: * error for object 0x82a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish hadcm3n_6.07_i686-apple-darwin(237,0xa0a6b540) malloc: * error for object 0x1032c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xa0a6b540) malloc: * error for object 0x845004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Thu Dec 13 06:27:49 2012 Thread 0 Crashed: Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x82d004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x1000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x1000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x1000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x1000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x1000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x1000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x1000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x1000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x1000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x1000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x1011e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x1020404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x102ea04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(217,0xa0a6b540) malloc: * error for object 0x102ea00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zgi0_1880_40_008242799/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zgi0_1880_40_008242799/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zgi0_1880_40_008242799/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zgi0_1880_40_008242799/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zgi0_1880_40_008242799/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zgi0_1880_40_008242799/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zgi0_1880_40_008242799/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zgi0_1880_40_008242799/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zgi0_1880_40_008242799/dataout/atmos_restart.day after 11 attempts 05:59:50 (217): No heartbeat from core client for 30 sec - exiting cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zgi0_1880_40_008242799/dataout/ocean_restart.day after 11 attempts CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zgi0_1880_40_008242799/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zgi0_1880_40_008242799/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zgi0_1880_40_008242799/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zgi0_1880_40_008242799/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Dec 2012 20:56:34	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	518,400	1,770,516	3.4153
07 Dec 2012 22:39:13	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	492,480	1,683,394	3.4182
05 Dec 2012 11:45:04	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	466,560	1,596,144	3.4211
01 Dec 2012 21:20:48	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	440,640	1,512,178	3.4318
30 Nov 2012 19:01:43	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	414,720	1,427,256	3.4415
28 Nov 2012 12:27:42	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	388,800	1,340,150	3.4469
25 Nov 2012 19:10:21	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	362,880	1,254,458	3.4569
24 Nov 2012 14:59:44	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	336,960	1,166,690	3.4624
23 Nov 2012 10:56:25	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	311,040	1,078,833	3.4685
21 Nov 2012 13:34:11	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	285,120	989,748	3.4713
18 Nov 2012 15:25:43	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	259,200	902,881	3.4833
17 Nov 2012 11:55:52	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	233,280	813,827	3.4886
15 Nov 2012 12:59:51	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	207,360	725,136	3.4970
12 Nov 2012 12:07:45	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	181,440	635,434	3.5022
10 Nov 2012 17:22:03	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	155,520	542,511	3.4884
09 Nov 2012 12:02:58	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	129,600	452,184	3.4891
06 Nov 2012 20:09:42	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	103,680	360,904	3.4809
05 Nov 2012 06:57:52	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	77,760	270,479	3.4784
03 Nov 2012 19:42:12	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	51,840	180,316	3.4783
02 Nov 2012 14:40:44	1193569	15420174	hadcm3n_zgi0_1880_40_008242799_2	25,920	90,151	3.4780