Task 16073838

Name	hadcm3n_odd3_1900_40_008472442_1
Workunit	8623281
Created	25 Oct 2013, 1:36:18 UTC
Sent	25 Oct 2013, 1:36:20 UTC
Report deadline	24 Jan 2014, 9:03:31 UTC
Received	3 Nov 2013, 11:27:01 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1266353
Run time	3 days 17 hours 44 min 13 sec
CPU time	3 days 2 hours 14 min 50 sec
Validate state	Invalid
Credit	8,398.08
Device peak FLOPS	3.60 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.26</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 21:07:38 (65772): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... 17:46:26 (86232): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=25016, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 08:00:26 (15260): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:02:15 (58884): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:56:40 (15504): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:57:41 (60532): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 05:36:11 (44760): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:32:00 (6512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 01:03:34 (47480): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 01:04:06 (47864): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 01:05:02 (44888): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 01:05:30 (46768): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Nov 2013 04:36:58	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	699,840	292,232	0.4176
02 Nov 2013 19:38:10	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	673,920	454,210	0.6740
02 Nov 2013 10:28:50	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	648,000	423,927	0.6542
02 Nov 2013 01:05:28	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	622,080	394,836	0.6347
01 Nov 2013 17:31:31	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	596,160	367,542	0.6165
01 Nov 2013 09:37:24	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	570,240	340,812	0.5977
01 Nov 2013 01:29:39	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	544,320	313,976	0.5768
31 Oct 2013 16:19:44	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	518,400	286,715	0.5531
31 Oct 2013 08:29:06	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	492,480	307,443	0.6243
31 Oct 2013 00:26:20	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	466,560	281,354	0.6030
30 Oct 2013 15:43:56	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	440,640	263,039	0.5969
30 Oct 2013 07:32:15	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	414,720	236,437	0.5701
29 Oct 2013 23:35:58	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	388,800	209,926	0.5399
29 Oct 2013 15:45:17	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	362,880	183,397	0.5054
29 Oct 2013 07:52:22	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	336,960	156,765	0.4652
29 Oct 2013 00:24:05	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	311,040	130,679	0.4201
28 Oct 2013 16:17:18	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	285,120	103,911	0.3644
28 Oct 2013 08:30:40	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	259,200	77,486	0.2989
28 Oct 2013 00:36:59	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	233,280	50,903	0.2182
27 Oct 2013 16:49:59	1266353	16073838	hadcm3n_odd3_1900_40_008472442_1	207,360	128,890	0.6216