Task 17436387

Name	hadcm3n_xasv_1940_40_009150293_2
Workunit	9280629
Created	19 Nov 2014, 11:09:20 UTC
Sent	19 Nov 2014, 11:15:08 UTC
Report deadline	18 Feb 2015, 18:42:19 UTC
Received	31 Dec 2014, 7:41:11 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1170047
Run time	6 days 20 hours 15 min 25 sec
CPU time	5 days 19 hours 58 min 52 sec
Validate state	Invalid
Credit	4,043.52
Device peak FLOPS	2.69 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.21</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:08:12 (44545): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:08:13 (44545): No heartbeat from core client for 30 sec - exiting 13:22:58 (95047): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:24:43 (95551): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:28:45 (95580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:28:47 (95580): No heartbeat from core client for 30 sec - exiting 13:28:48 (95580): No heartbeat from core client for 30 sec - exiting 13:28:49 (95580): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(95607,0xa05a1540) malloc: * error for object 0x400f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95607,0xa05a1540) malloc: * error for object 0x4010404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95607,0xa05a1540) malloc: * error for object 0x4010400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95607,0xa05a1540) malloc: * error for object 0x806604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95607,0xa05a1540) malloc: * error for object 0x806600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95607,0xa05a1540) malloc: * error for object 0x806604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95607,0xa05a1540) malloc: * error for object 0x806600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95607,0xa05a1540) malloc: * error for object 0x806604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95607,0xa05a1540) malloc: * error for object 0x806600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95607,0xa05a1540) malloc: * error for object 0x806604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95607,0xa05a1540) malloc: * error for object 0x806600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95607,0xa05a1540) malloc: * error for object 0x800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95607,0xa05a1540) malloc: * error for object 0x800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:17:10 (95607): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:17:14 (95607): No heartbeat from core client for 30 sec - exiting 06:17:15 (95607): No heartbeat from core client for 30 sec - exiting 06:17:16 (95607): No heartbeat from core client for 30 sec - exiting 06:17:17 (95607): No heartbeat from core client for 30 sec - exiting 06:17:18 (95607): No heartbeat from core client for 30 sec - exiting 06:17:19 (95607): No heartbeat from core client for 30 sec - exiting 06:17:20 (95607): No heartbeat from core client for 30 sec - exiting 06:17:21 (95607): No heartbeat from core client for 30 sec - exiting Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Dec 2014 06:38:44	1170047	17436387	hadcm3n_xasv_1940_40_009150293_2	336,960	564,489	1.6752
29 Dec 2014 21:06:03	1170047	17436387	hadcm3n_xasv_1940_40_009150293_2	311,040	531,122	1.7076
29 Dec 2014 09:49:16	1170047	17436387	hadcm3n_xasv_1940_40_009150293_2	285,120	492,075	1.7259
28 Dec 2014 06:04:39	1170047	17436387	hadcm3n_xasv_1940_40_009150293_2	259,200	441,670	1.7040
27 Dec 2014 08:28:47	1170047	17436387	hadcm3n_xasv_1940_40_009150293_2	233,280	408,815	1.7525
26 Dec 2014 13:03:47	1170047	17436387	hadcm3n_xasv_1940_40_009150293_2	207,360	375,860	1.8126
26 Dec 2014 03:37:00	1170047	17436387	hadcm3n_xasv_1940_40_009150293_2	181,440	342,843	1.8896
25 Dec 2014 15:19:11	1170047	17436387	hadcm3n_xasv_1940_40_009150293_2	155,520	301,188	1.9367
25 Dec 2014 00:29:19	1170047	17436387	hadcm3n_xasv_1940_40_009150293_2	129,600	249,770	1.9272
24 Dec 2014 09:51:56	1170047	17436387	hadcm3n_xasv_1940_40_009150293_2	103,680	199,390	1.9231
23 Dec 2014 12:59:18	1170047	17436387	hadcm3n_xasv_1940_40_009150293_2	77,760	148,029	1.9037
22 Dec 2014 21:59:50	1170047	17436387	hadcm3n_xasv_1940_40_009150293_2	51,840	96,567	1.8628
22 Dec 2014 04:56:15	1170047	17436387	hadcm3n_xasv_1940_40_009150293_2	25,920	44,783	1.7277