Task 15934893

Name	hadcm3n_n16h_1920_40_008410467_0
Workunit	8561323
Created	22 Aug 2013, 5:35:12 UTC
Sent	22 Aug 2013, 6:06:44 UTC
Report deadline	21 Nov 2013, 13:33:55 UTC
Received	19 Sep 2013, 22:22:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1110299
Run time	22 days 5 hours 7 min 29 sec
CPU time	18 days 7 hours 29 min 58 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.92 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 01:12:51 (89935): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(1724,0xac8372c0) malloc: * error for object 0x2beb004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1724,0xac8372c0) malloc: * error for object 0x2beb000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1724,0xac8372c0) malloc: * error for object 0x73e9204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1724,0xac8372c0) malloc: * error for object 0x73e9200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1724,0xac8372c0) malloc: * error for object 0x73e9204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1724,0xac8372c0) malloc: * error for object 0x73e9200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1724,0xac8372c0) malloc: * error for object 0x73e9204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1724,0xac8372c0) malloc: * error for object 0x73e9200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1724,0xac8372c0) malloc: * error for object 0x73e9204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1724,0xac8372c0) malloc: * error for object 0x73e9200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1724,0xac8372c0) malloc: * error for object 0x73e9204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1724,0xac8372c0) malloc: * error for object 0x73e9200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug 19:38:38 (1724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:26:11 (401): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:15:54 (778): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:15:56 (778): No heartbeat from core client for 30 sec - exiting 15:15:57 (778): No heartbeat from core client for 30 sec - exiting 17:05:21 (7654): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:33:48 (9534): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:00:05 (13704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:00:08 (13704): No heartbeat from core client for 30 sec - exiting 08:00:09 (13704): No heartbeat from core client for 30 sec - exiting 08:00:10 (13704): No heartbeat from core client for 30 sec - exiting 08:00:11 (13704): No heartbeat from core client for 30 sec - exiting 08:00:12 (13704): No heartbeat from core client for 30 sec - exiting 08:00:13 (13704): No heartbeat from core client for 30 sec - exiting 08:00:14 (13704): No heartbeat from core client for 30 sec - exiting 11:07:25 (30317): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:08:49 (33436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:31:48 (34557): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:31:49 (34557): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:03:09 (42802): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:13:21 (61596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:35:41 (75298): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:28:44 (99181): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:22:58 (8175): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:14:33 (12058): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:14:34 (12058): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 10:42:38 (20505): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:42:39 (20505): No heartbeat from core client for 30 sec - exiting 10:42:40 (20505): No heartbeat from core client for 30 sec - exiting 10:42:41 (20505): No heartbeat from core client for 30 sec - exiting 10:42:42 (20505): No heartbeat from core client for 30 sec - exiting 10:42:43 (20505): No heartbeat from core client for 30 sec - exiting 10:42:44 (20505): No heartbeat from core client for 30 sec - exiting 10:42:45 (20505): No heartbeat from core client for 30 sec - exiting 10:42:46 (20505): No heartbeat from core client for 30 sec - exiting 10:42:47 (20505): No heartbeat from core client for 30 sec - exiting 10:42:48 (20505): No heartbeat from core client for 30 sec - exiting 10:42:49 (20505): No heartbeat from core client for 30 sec - exiting 10:42:50 (20505): No heartbeat from core client for 30 sec - exiting 10:42:51 (20505): No heartbeat from core client for 30 sec - exiting 10:42:52 (20505): No heartbeat from core client for 30 sec - exiting 10:42:53 (20505): No heartbeat from core client for 30 sec - exiting 10:42:54 (20505): No heartbeat from core client for 30 sec - exiting 10:42:55 (20505): No heartbeat from core client for 30 sec - exiting 10:42:56 (20505): No heartbeat from core client for 30 sec - exiting 12:22:38 (40068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:22:40 (40068): No heartbeat from core client for 30 sec - exiting 12:22:41 (40068): No heartbeat from core client for 30 sec - exiting 12:22:42 (40068): No heartbeat from core client for 30 sec - exiting 12:22:43 (40068): No heartbeat from core client for 30 sec - exiting 12:22:44 (40068): No heartbeat from core client for 30 sec - exiting 12:22:45 (40068): No heartbeat from core client for 30 sec - exiting 12:22:46 (40068): No heartbeat from core client for 30 sec - exiting 12:22:47 (40068): No heartbeat from core client for 30 sec - exiting 12:22:48 (40068): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 14:08:13 (41732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:38:25 (43498): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:39:47 (45988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:22:56 (47043): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:21:43 (72720): No heartbeat from core client for 30 sec - exiting 22:21:44 (72720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n16h_1920_40_008410467/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n16h_1920_40_008410467/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n16h_1920_40_008410467/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n16h_1920_40_008410467/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n16h_1920_40_008410467/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n16h_1920_40_008410467/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n16h_1920_40_008410467/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n16h_1920_40_008410467/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n16h_1920_40_008410467/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n16h_1920_40_008410467/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n16h_1920_40_008410467/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n16h_1920_40_008410467/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Sep 2013 21:24:49	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	777,600	1,582,261	2.0348
19 Sep 2013 00:15:12	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	751,680	1,531,038	2.0368
18 Sep 2013 02:29:07	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	725,760	1,480,119	2.0394
18 Sep 2013 02:29:07	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	725,760	1,480,119	2.0394
17 Sep 2013 05:32:07	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	699,840	1,428,614	2.0413
16 Sep 2013 08:46:50	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	673,920	1,376,374	2.0423
15 Sep 2013 12:05:03	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	648,000	1,325,240	2.0451
14 Sep 2013 15:14:37	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	622,080	1,275,663	2.0506
13 Sep 2013 19:38:29	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	596,160	1,227,086	2.0583
12 Sep 2013 22:03:33	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	570,240	1,176,893	2.0639
12 Sep 2013 01:26:31	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	544,320	1,123,049	2.0632
11 Sep 2013 06:00:27	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	518,400	1,070,610	2.0652
10 Sep 2013 10:50:02	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	492,480	1,019,880	2.0709
09 Sep 2013 14:06:10	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	466,560	970,907	2.0810
08 Sep 2013 13:43:27	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	440,640	924,239	2.0975
07 Sep 2013 21:48:04	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	414,720	875,726	2.1116
06 Sep 2013 18:15:22	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	388,800	824,077	2.1195
05 Sep 2013 22:58:03	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	362,880	771,109	2.1250
05 Sep 2013 03:06:49	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	336,960	709,404	2.1053
04 Sep 2013 07:41:15	1110299	15934893	hadcm3n_n16h_1920_40_008410467_0	311,040	649,742	2.0889