Task 15554628

Name	hadcm3n_ze0b_1880_40_008250307_4
Workunit	8405431
Created	22 Jan 2013, 9:59:21 UTC
Sent	22 Jan 2013, 9:59:25 UTC
Report deadline	23 Apr 2013, 17:26:36 UTC
Received	14 Mar 2013, 1:45:52 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1131315
Run time	15 days 6 hours 10 min 44 sec
CPU time	14 days 1 hours 55 min 56 sec
Validate state	Invalid
Credit	9,642.24
Device peak FLOPS	3.00 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24012, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:56:41 (6148): No heartbeat from core client for 30 sec - exiting 07:56:42 (6148): No heartbeat from core client for 30 sec - exiting 07:56:43 (6148): No heartbeat from core client for 30 sec - exiting 07:56:44 (6148): No heartbeat from core client for 30 sec - exiting 07:56:45 (6148): No heartbeat from core client for 30 sec - exiting 07:56:47 (6148): No heartbeat from core client for 30 sec - exiting 07:56:48 (6148): No heartbeat from core client for 30 sec - exiting 07:56:49 (6148): No heartbeat from core client for 30 sec - exiting 07:56:50 (6148): No heartbeat from core client for 30 sec - exiting 07:56:51 (6148): No heartbeat from core client for 30 sec - exiting 07:56:52 (6148): No heartbeat from core client for 30 sec - exiting 07:56:53 (6148): No heartbeat from core client for 30 sec - exiting 07:56:54 (6148): No heartbeat from core client for 30 sec - exiting 07:56:55 (6148): No heartbeat from core client for 30 sec - exiting 07:56:56 (6148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:15:11 (6148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:15:26 (6148): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:14:32 (3800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:14:34 (3800): No heartbeat from core client for 30 sec - exiting 03:14:35 (3800): No heartbeat from core client for 30 sec - exiting 03:14:36 (3800): No heartbeat from core client for 30 sec - exiting 03:14:37 (3800): No heartbeat from core client for 30 sec - exiting 03:14:38 (3800): No heartbeat from core client for 30 sec - exiting 03:14:39 (3800): No heartbeat from core client for 30 sec - exiting 03:16:23 (15540): No heartbeat from core client for 30 sec - exiting 03:16:24 (15540): No heartbeat from core client for 30 sec - exiting 03:16:25 (15540): No heartbeat from core client for 30 sec - exiting 03:16:26 (15540): No heartbeat from core client for 30 sec - exiting 03:16:27 (15540): No heartbeat from core client for 30 sec - exiting 03:16:28 (15540): No heartbeat from core client for 30 sec - exiting 03:16:29 (15540): No heartbeat from core client for 30 sec - exiting 03:16:30 (15540): No heartbeat from core client for 30 sec - exiting 03:16:31 (15540): No heartbeat from core client for 30 sec - exiting 03:16:32 (15540): No heartbeat from core client for 30 sec - exiting 03:16:33 (15540): No heartbeat from core client for 30 sec - exiting 03:16:34 (15540): No heartbeat from core client for 30 sec - exiting 03:16:35 (15540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:53:03 (5236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8440, iMonCtr=1 Model crash detected, will try to restart... 14:56:08 (2308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Mar 2013 23:37:04	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	803,520	1,257,500	1.5650
13 Mar 2013 11:45:40	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	777,600	1,216,563	1.5645
12 Mar 2013 23:27:31	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	751,680	1,175,977	1.5645
12 Mar 2013 12:07:29	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	725,760	1,135,933	1.5652
11 Mar 2013 23:58:54	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	699,840	1,096,116	1.5662
11 Mar 2013 11:49:13	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	673,920	1,055,762	1.5666
10 Mar 2013 23:09:43	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	648,000	1,014,970	1.5663
10 Mar 2013 11:22:01	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	622,080	975,029	1.5674
09 Mar 2013 23:34:21	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	596,160	934,852	1.5681
09 Mar 2013 12:08:03	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	570,240	893,971	1.5677
07 Mar 2013 08:50:50	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	544,320	853,190	1.5674
06 Mar 2013 20:13:46	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	518,400	812,079	1.5665
06 Mar 2013 08:30:45	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	492,480	771,178	1.5659
05 Mar 2013 22:08:02	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	466,560	731,243	1.5673
05 Mar 2013 08:05:42	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	440,640	691,019	1.5682
04 Mar 2013 16:35:53	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	414,720	649,856	1.5670
04 Mar 2013 02:21:41	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	388,800	609,520	1.5677
03 Mar 2013 14:25:22	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	362,880	569,841	1.5703
02 Mar 2013 20:10:11	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	336,960	529,700	1.5720
02 Mar 2013 04:16:14	1131315	15554628	hadcm3n_ze0b_1880_40_008250307_4	311,040	489,738	1.5745