Task 15591704

Name	hadcm3n_4d1i_1940_40_008305577_0
Workunit	8456712
Created	7 Feb 2013, 6:40:58 UTC
Sent	7 Feb 2013, 6:45:50 UTC
Report deadline	9 May 2013, 14:13:01 UTC
Received	28 May 2013, 17:17:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1122605
Run time	14 days 6 hours 33 min 37 sec
CPU time	12 days 1 hours 41 min 13 sec
Validate state	Invalid
Credit	7,464.96
Device peak FLOPS	2.60 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(64016,0xa0210540) malloc: * error for object 0x7000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(64016,0xa0210540) malloc: * error for object 0x7000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(64016,0xa0210540) malloc: * error for object 0x2801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=64040, selfPID=64040, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(87907,0xa0210540) malloc: * error for object 0x881c404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(87907,0xa0210540) malloc: * error for object 0x881d204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(87907,0xa0210540) malloc: * error for object 0x881d200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(87907,0xa0210540) malloc: * error for object 0x881d204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(87907,0xa0210540) malloc: * error for object 0x881d200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(87907,0xa0210540) malloc: * error for object 0x881d204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(87907,0xa0210540) malloc: * error for object 0x881d200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(87907,0xa0210540) malloc: * error for object 0x2801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(87907,0xa0210540) malloc: * error for object 0x2801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(87907,0xa0210540) malloc: * error for object 0x2801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(87907,0xa0210540) malloc: * error for object 0x2801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 May 2013 19:51:46	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	622,080	1,032,142	1.6592
18 May 2013 19:25:45	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	596,160	987,235	1.6560
15 May 2013 18:38:32	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	570,240	941,538	1.6511
13 May 2013 17:28:49	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	544,320	892,875	1.6403
10 May 2013 22:06:57	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	518,400	843,065	1.6263
08 May 2013 20:06:24	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	492,480	798,632	1.6217
27 Apr 2013 21:22:07	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	466,560	760,739	1.6305
23 Apr 2013 21:25:42	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	440,640	720,592	1.6353
20 Apr 2013 21:33:29	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	414,720	677,093	1.6327
16 Apr 2013 21:17:28	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	388,800	634,365	1.6316
09 Apr 2013 20:22:21	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	362,880	590,878	1.6283
04 Apr 2013 20:29:02	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	336,960	549,464	1.6307
02 Apr 2013 21:00:21	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	311,040	509,142	1.6369
21 Mar 2013 19:42:24	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	285,120	465,176	1.6315
19 Mar 2013 19:39:31	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	259,200	421,515	1.6262
13 Mar 2013 19:42:19	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	233,280	378,328	1.6218
05 Mar 2013 19:40:18	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	207,360	335,868	1.6197
03 Mar 2013 18:40:03	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	181,440	292,618	1.6128
01 Mar 2013 18:49:14	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	155,520	250,294	1.6094
26 Feb 2013 19:07:05	1122605	15591704	hadcm3n_4d1i_1940_40_008305577_0	129,600	207,982	1.6048