Task 15193451

Name	hadcm3n_zjmh_1880_40_008026378_3
Workunit	8181492
Created	27 Aug 2012, 18:40:05 UTC
Sent	27 Aug 2012, 18:40:49 UTC
Report deadline	27 Nov 2012, 2:08:00 UTC
Received	2 Oct 2012, 22:12:10 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1159137
Run time	18 days 23 hours 1 min 37 sec
CPU time	17 days 6 hours 19 min 2 sec
Validate state	Invalid
Credit	12,130.56
Device peak FLOPS	3.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:33:52 (6059): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:10:07 (1997): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:10:08 (1997): No heartbeat from core client for 30 sec - exiting 05:25:44 (4401): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:25:45 (4401): No heartbeat from core client for 30 sec - exiting 05:25:46 (4401): No heartbeat from core client for 30 sec - exiting 05:25:47 (4401): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:00:40 (5059): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:09:11 (40132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:09:12 (40132): No heartbeat from core client for 30 sec - exiting 20:54:09 (40861): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:54:10 (40861): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:52:06 (48685): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:52:07 (48685): No heartbeat from core client for 30 sec - exiting 19:52:08 (48685): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:34:59 (9625): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:35:00 (9625): No heartbeat from core client for 30 sec - exiting 22:35:01 (9625): No heartbeat from core client for 30 sec - exiting 23:10:34 (10632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:10:35 (10632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:20:15 (6672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:25:37 (11195): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:38:22 (593): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:55:50 (1810): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x10c1604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x10c1600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x20c1004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x20c1000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x10c1604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x10c1600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x10c1604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x10c1600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x10c1604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x10c1600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x10c1604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x10c1600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x8bfa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x28caa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x2097a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x2097a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x89d204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x89d200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x89d204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x89d200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x89d204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x89d200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x89d204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x89d200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x89d204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5856,0xad0e22c0) malloc: * error for object 0x89d200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:06:12 (12834): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:05:49 (44082): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:05:50 (44082): No heartbeat from core client for 30 sec - exiting 15:05:51 (44082): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Oct 2012 21:14:02	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	1,010,880	1,500,550	1.4844
02 Oct 2012 08:11:56	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	984,960	1,463,857	1.4862
01 Oct 2012 21:13:30	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	959,040	1,427,245	1.4882
01 Oct 2012 08:05:15	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	933,120	1,391,321	1.4910
29 Sep 2012 03:09:40	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	907,200	1,353,338	1.4918
28 Sep 2012 09:51:33	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	881,280	1,313,646	1.4906
27 Sep 2012 12:34:59	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	855,360	1,274,691	1.4902
25 Sep 2012 12:18:45	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	829,440	1,237,709	1.4922
24 Sep 2012 16:27:03	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	803,520	1,198,042	1.4910
24 Sep 2012 04:33:58	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	777,600	1,158,319	1.4896
23 Sep 2012 10:01:31	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	751,680	1,118,487	1.4880
22 Sep 2012 23:08:47	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	725,760	1,078,810	1.4865
22 Sep 2012 07:52:47	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	699,840	1,039,375	1.4852
20 Sep 2012 16:23:49	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	673,920	1,000,927	1.4852
19 Sep 2012 03:46:53	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	648,000	961,134	1.4832
17 Sep 2012 14:41:34	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	622,080	923,474	1.4845
16 Sep 2012 11:10:07	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	596,160	886,171	1.4865
15 Sep 2012 11:08:03	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	570,240	848,716	1.4883
14 Sep 2012 22:25:01	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	544,320	810,042	1.4882
14 Sep 2012 10:02:46	1159137	15193451	hadcm3n_zjmh_1880_40_008026378_3	518,400	769,990	1.4853