Task 18708268

Name	hadcm3n_xdpa_1940_40_010021541_1
Workunit	10019612
Created	18 Jul 2015, 6:47:34 UTC
Sent	18 Jul 2015, 6:50:54 UTC
Report deadline	17 Oct 2015, 14:18:05 UTC
Received	16 Aug 2015, 15:35:33 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1365241
Run time	12 days 21 hours 5 min 30 sec
CPU time	11 days 14 hours 27 min 30 sec
Validate state	Invalid
Credit	10,264.32
Device peak FLOPS	3.35 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.6.2</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:33:57 (3612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/xdpako.pjg0c10 Error converting file to netcdf: dataout/xdpako.pig0c10 Error converting file to netcdf: dataout/xdpako.pfg0c10 Error converting file to netcdf: dataout/xdpako.pcg0c10 Error converting file to netcdf: dataout/xdpako.pbg0c10 Error converting file to netcdf: dataout/xdpako.pag0c10 Error converting file to netcdf: dataout/xdpaka.phg0c10 Error converting file to netcdf: dataout/xdpaka.pgg0c10 Error converting file to netcdf: dataout/xdpaka.peg0c10 Error converting file to netcdf: dataout/xdpaka.pdg0c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/xdpako.pjg1c10 Error converting file to netcdf: dataout/xdpako.pig1c10 Error converting file to netcdf: dataout/xdpako.pfg1c10 Error converting file to netcdf: dataout/xdpaka.phg1c10 Error converting file to netcdf: dataout/xdpaka.pgg1c10 Error converting file to netcdf: dataout/xdpaka.peg1c10 Error converting file to netcdf: dataout/xdpaka.pdg1c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:30:51 (3884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 08:16:35 (3396): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 04:42:43 (6212): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:11:43 (12264): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 06:16:07 (4684): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 09:42:39 (5652): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 05:43:22 (5704): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:38:43 (10020): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 03:27:24 (7496): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 10:37:56 (7552): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:44:41 (8404): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Aug 2015 17:13:13	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	855,360	996,419	1.1649
13 Aug 2015 16:01:13	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	829,440	958,851	1.1560
11 Aug 2015 11:16:41	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	803,520	949,992	1.1823
07 Aug 2015 21:53:48	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	777,600	1,049,128	1.3492
07 Aug 2015 11:56:40	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	751,680	1,013,754	1.3487
07 Aug 2015 02:17:36	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	725,760	979,081	1.3490
06 Aug 2015 16:24:35	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	699,840	944,006	1.3489
04 Aug 2015 20:53:47	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	673,920	908,242	1.3477
04 Aug 2015 07:19:35	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	648,000	873,474	1.3480
03 Aug 2015 20:33:28	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	622,080	838,994	1.3487
03 Aug 2015 06:18:42	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	596,160	803,005	1.3470
01 Aug 2015 18:38:18	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	570,240	767,522	1.3460
30 Jul 2015 10:09:31	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	544,320	732,448	1.3456
29 Jul 2015 18:53:52	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	518,400	698,361	1.3471
29 Jul 2015 06:57:25	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	492,480	664,610	1.3495
28 Jul 2015 14:00:06	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	466,560	628,932	1.3480
27 Jul 2015 19:35:29	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	440,640	594,804	1.3499
27 Jul 2015 09:17:15	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	414,720	560,448	1.3514
26 Jul 2015 20:59:31	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	388,800	525,604	1.3519
26 Jul 2015 09:41:00	1365241	18708268	hadcm3n_xdpa_1940_40_010021541_1	362,880	490,834	1.3526