Task 15841638

Name	hadcm3n_3jbj_1940_40_008259434_2
Workunit	8414558
Created	13 Jun 2013, 15:37:45 UTC
Sent	13 Jun 2013, 17:25:49 UTC
Report deadline	13 Sep 2013, 0:53:00 UTC
Received	29 Jun 2013, 8:51:05 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1230144
Run time	14 days 8 hours 37 min 49 sec
CPU time	13 days 1 hours 48 min 35 sec
Validate state	Invalid
Credit	8,087.04
Device peak FLOPS	2.20 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5052, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 00:57:43 (2268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:48:22 (4596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:02:50 (5084): No heartbeat from core client for 30 sec - exiting 10:02:51 (5084): No heartbeat from core client for 30 sec - exiting 10:02:52 (5084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:35:46 (4176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:44:25 (4828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:05:04 (4504): No heartbeat from core client for 30 sec - exiting 05:05:05 (4504): No heartbeat from core client for 30 sec - exiting 05:05:06 (4504): No heartbeat from core client for 30 sec - exiting 05:05:07 (4504): No heartbeat from core client for 30 sec - exiting 05:05:08 (4504): No heartbeat from core client for 30 sec - exiting 05:05:09 (4504): No heartbeat from core client for 30 sec - exiting 05:05:10 (4504): No heartbeat from core client for 30 sec - exiting 05:05:11 (4504): No heartbeat from core client for 30 sec - exiting 05:05:12 (4504): No heartbeat from core client for 30 sec - exiting 05:05:13 (4504): No heartbeat from core client for 30 sec - exiting 05:05:15 (4504): No heartbeat from core client for 30 sec - exiting 05:05:16 (4504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3752, iMonCtr=1 Model crash detected, will try to restart... 05:07:38 (3428): No heartbeat from core client for 30 sec - exiting 05:07:40 (3428): No heartbeat from core client for 30 sec - exiting 05:07:41 (3428): No heartbeat from core client for 30 sec - exiting 05:07:42 (3428): No heartbeat from core client for 30 sec - exiting 05:07:43 (3428): No heartbeat from core client for 30 sec - exiting 05:07:44 (3428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:12:42 (4472): No heartbeat from core client for 30 sec - exiting 05:12:44 (4472): No heartbeat from core client for 30 sec - exiting 05:12:45 (4472): No heartbeat from core client for 30 sec - exiting 05:12:46 (4472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:04:58 (4372): No heartbeat from core client for 30 sec - exiting 05:04:59 (4372): No heartbeat from core client for 30 sec - exiting 05:05:00 (4372): No heartbeat from core client for 30 sec - exiting 05:05:01 (4372): No heartbeat from core client for 30 sec - exiting 05:05:02 (4372): No heartbeat from core client for 30 sec - exiting 05:05:03 (4372): No heartbeat from core client for 30 sec - exiting 05:05:05 (4372): No heartbeat from core client for 30 sec - exiting 05:05:06 (4372): No heartbeat from core client for 30 sec - exiting 05:05:07 (4372): No heartbeat from core client for 30 sec - exiting 05:05:08 (4372): No heartbeat from core client for 30 sec - exiting 05:05:09 (4372): No heartbeat from core client for 30 sec - exiting 05:05:10 (4372): No heartbeat from core client for 30 sec - exiting 05:05:11 (4372): No heartbeat from core client for 30 sec - exiting 05:05:12 (4372): No heartbeat from core client for 30 sec - exiting 05:05:13 (4372): No heartbeat from core client for 30 sec - exiting 05:05:14 (4372): No heartbeat from core client for 30 sec - exiting 05:05:15 (4372): No heartbeat from core client for 30 sec - exiting 05:05:16 (4372): No heartbeat from core client for 30 sec - exiting 05:05:17 (4372): No heartbeat from core client for 30 sec - exiting 05:05:18 (4372): No heartbeat from core client for 30 sec - exiting 05:05:19 (4372): No heartbeat from core client for 30 sec - exiting 05:05:20 (4372): No heartbeat from core client for 30 sec - exiting 05:05:21 (4372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:29:44 (5240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:21:32 (3272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:27:01 (3200): No heartbeat from core client for 30 sec - exiting 07:27:02 (3200): No heartbeat from core client for 30 sec - exiting 07:27:03 (3200): No heartbeat from core client for 30 sec - exiting 07:27:05 (3200): No heartbeat from core client for 30 sec - exiting 07:27:06 (3200): No heartbeat from core client for 30 sec - exiting 07:27:07 (3200): No heartbeat from core client for 30 sec - exiting 07:27:08 (3200): No heartbeat from core client for 30 sec - exiting 07:27:09 (3200): No heartbeat from core client for 30 sec - exiting 07:27:10 (3200): No heartbeat from core client for 30 sec - exiting 07:27:11 (3200): No heartbeat from core client for 30 sec - exiting 07:27:12 (3200): No heartbeat from core client for 30 sec - exiting 07:27:13 (3200): No heartbeat from core client for 30 sec - exiting 07:27:14 (3200): No heartbeat from core client for 30 sec - exiting 07:27:16 (3200): No heartbeat from core client for 30 sec - exiting 07:27:17 (3200): No heartbeat from core client for 30 sec - exiting 07:27:18 (3200): No heartbeat from core client for 30 sec - exiting 07:27:19 (3200): No heartbeat from core client for 30 sec - exiting 07:27:20 (3200): No heartbeat from core client for 30 sec - exiting 07:27:21 (3200): No heartbeat from core client for 30 sec - exiting 07:27:22 (3200): No heartbeat from core client for 30 sec - exiting 07:27:23 (3200): No heartbeat from core client for 30 sec - exiting 07:27:24 (3200): No heartbeat from core client for 30 sec - exiting 07:27:25 (3200): No heartbeat from core client for 30 sec - exiting 07:27:26 (3200): No heartbeat from core client for 30 sec - exiting 07:27:28 (3200): No heartbeat from core client for 30 sec - exiting 07:27:29 (3200): No heartbeat from core client for 30 sec - exiting 07:27:30 (3200): No heartbeat from core client for 30 sec - exiting 07:27:31 (3200): No heartbeat from core client for 30 sec - exiting 07:27:32 (3200): No heartbeat from core client for 30 sec - exiting 07:27:33 (3200): No heartbeat from core client for 30 sec - exiting 07:27:34 (3200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:19:31 (4348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:09:52 (4120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:32:39 (2696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:35:48 (7784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:47:02 (7276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:16:58 (4512): No heartbeat from core client for 30 sec - exiting 05:16:59 (4512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:17:34 (3852): No heartbeat from core client for 30 sec - exiting 05:17:35 (3852): No heartbeat from core client for 30 sec - exiting 05:17:36 (3852): No heartbeat from core client for 30 sec - exiting 05:17:38 (3852): No heartbeat from core client for 30 sec - exiting 05:17:39 (3852): No heartbeat from core client for 30 sec - exiting 05:17:40 (3852): No heartbeat from core client for 30 sec - exiting 05:17:41 (3852): No heartbeat from core client for 30 sec - exiting 05:17:42 (3852): No heartbeat from core client for 30 sec - exiting 05:17:43 (3852): No heartbeat from core client for 30 sec - exiting 05:17:44 (3852): No heartbeat from core client for 30 sec - exiting 05:17:45 (3852): No heartbeat from core client for 30 sec - exiting 05:17:46 (3852): No heartbeat from core client for 30 sec - exiting 05:17:47 (3852): No heartbeat from core client for 30 sec - exiting 05:17:48 (3852): No heartbeat from core client for 30 sec - exiting 05:17:50 (3852): No heartbeat from core client for 30 sec - exiting 05:17:51 (3852): No heartbeat from core client for 30 sec - exiting 05:17:52 (3852): No heartbeat from core client for 30 sec - exiting 05:17:53 (3852): No heartbeat from core client for 30 sec - exiting 05:17:54 (3852): No heartbeat from core client for 30 sec - exiting 05:17:55 (3852): No heartbeat from core client for 30 sec - exiting 05:17:56 (3852): No heartbeat from core client for 30 sec - exiting 05:17:57 (3852): No heartbeat from core client for 30 sec - exiting 05:17:58 (3852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:37:23 (388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:37:24 (388): No heartbeat from core client for 30 sec - exiting 00:37:25 (388): No heartbeat from core client for 30 sec - exiting 00:37:26 (388): No heartbeat from core client for 30 sec - exiting 00:37:27 (388): No heartbeat from core client for 30 sec - exiting 00:38:07 (1984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:22:41 (4428): No heartbeat from core client for 30 sec - exiting 05:22:42 (4428): No heartbeat from core client for 30 sec - exiting 05:22:43 (4428): No heartbeat from core client for 30 sec - exiting 05:22:44 (4428): No heartbeat from core client for 30 sec - exiting 05:22:45 (4428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2952, iMonCtr=1 Model crash detected, will try to restart... 04:58:54 (4820): No heartbeat from core client for 30 sec - exiting 04:58:55 (4820): No heartbeat from core client for 30 sec - exiting 04:58:56 (4820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Jul 2013 09:51:17	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	673,920	1,141,575	1.6939
28 Jun 2013 08:53:37	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	648,000	1,097,160	1.6931
27 Jun 2013 20:04:56	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	622,080	1,052,458	1.6918
27 Jun 2013 05:32:36	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	596,160	1,008,771	1.6921
26 Jun 2013 16:23:40	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	570,240	965,374	1.6929
26 Jun 2013 01:30:40	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	544,320	921,136	1.6923
25 Jun 2013 11:37:28	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	518,400	877,532	1.6928
24 Jun 2013 21:34:59	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	492,480	832,841	1.6911
24 Jun 2013 06:53:59	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	466,560	788,512	1.6901
23 Jun 2013 19:05:20	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	440,640	744,663	1.6900
23 Jun 2013 03:54:38	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	414,720	700,223	1.6884
22 Jun 2013 14:58:56	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	388,800	656,314	1.6881
21 Jun 2013 23:39:38	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	362,880	612,060	1.6867
21 Jun 2013 08:23:12	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	336,960	567,692	1.6847
20 Jun 2013 19:34:13	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	311,040	524,328	1.6857
20 Jun 2013 05:16:27	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	285,120	480,202	1.6842
19 Jun 2013 14:59:10	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	259,200	436,214	1.6829
18 Jun 2013 23:47:02	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	233,280	391,810	1.6796
18 Jun 2013 09:34:25	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	207,360	347,905	1.6778
17 Jun 2013 20:25:50	1230144	15841638	hadcm3n_3jbj_1940_40_008259434_2	181,440	303,337	1.6718