Task 19133210

Name	hadcm3n_sava_198012_480_010221803_0
Workunit	10221803
Created	4 Dec 2015, 19:22:21 UTC
Sent	5 Dec 2015, 0:15:23 UTC
Report deadline	16 Nov 2016, 5:35:23 UTC
Received	27 Dec 2015, 19:38:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1314566
Run time	14 days 21 hours 13 min 15 sec
CPU time	11 days 18 hours 32 min 21 sec
Validate state	Invalid
Credit	6,842.88
Device peak FLOPS	3.36 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.6.12</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 06:27:19 (2706): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:29:27 (32502): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:44:16 (32758): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:03:24 (43226): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:34:35 (11310): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:15:52 (7719): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:00:05 (54509): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:57:02 (75539): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x204c004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x204c000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x102ba04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x302ba04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x3047004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x3047000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x1821604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x1821600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x2030a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x302ba04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x824604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x824600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x3005004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x3005000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x3005004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x3005000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x3005004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x3005000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x3005004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x3005000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x3005004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x3005000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x201a204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x2028804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x2036e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4623,0xa3d6a000) malloc: * error for object 0x2036e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Suspended CPDN Monitor - Suspend request from BOINC... 06:24:23 (4623): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 06:24:24 (4623): No heartbeat from core client for 30 sec - exiting Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 05:14:10 (52613): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:30:32 (58612): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 00:45:46 (68720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:07:36 (68780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:09:03 (69027): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:22:37 (69059): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:44:44 (71949): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:33:29 (72219): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:25:12 (72871): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:35:15 (87270): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:56:38 (46036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:49:50 (29103): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:53:19 (47681): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:14:14 (58716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:56:25 (59487): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:03:30 (62351): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:46:47 (19472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 41 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Dec 2015 16:37:08	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	570,240	1,004,338	1.7613
24 Dec 2015 02:32:29	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	544,320	959,817	1.7633
23 Dec 2015 11:51:24	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	518,400	915,862	1.7667
22 Dec 2015 19:30:33	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	492,480	870,861	1.7683
22 Dec 2015 05:31:59	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	466,560	825,985	1.7704
21 Dec 2015 15:19:56	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	440,640	781,935	1.7745
20 Dec 2015 17:45:02	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	414,720	736,780	1.7766
20 Dec 2015 01:05:06	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	388,800	690,669	1.7764
19 Dec 2015 04:49:44	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	362,880	642,133	1.7695
17 Dec 2015 11:20:58	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	336,960	595,934	1.7686
15 Dec 2015 18:26:21	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	311,040	548,060	1.7620
15 Dec 2015 01:35:10	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	285,120	502,543	1.7626
12 Dec 2015 12:29:48	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	259,200	457,038	1.7633
11 Dec 2015 21:49:45	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	233,280	414,436	1.7766
11 Dec 2015 07:16:28	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	207,360	368,803	1.7786
10 Dec 2015 10:22:18	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	181,440	324,693	1.7895
08 Dec 2015 22:59:04	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	155,520	278,181	1.7887
08 Dec 2015 07:09:17	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	129,600	227,945	1.7588
07 Dec 2015 15:52:29	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	103,680	179,608	1.7323
06 Dec 2015 20:29:25	1314566	19133210	hadcm3n_sava_198012_480_010221803_0	77,760	134,842	1.7341