Task 18743113

Name	hadcm3n_xby2_1980_40_010022427_4
Workunit	10020497
Created	28 Jul 2015, 10:23:09 UTC
Sent	28 Jul 2015, 10:25:19 UTC
Report deadline	27 Oct 2015, 17:52:30 UTC
Received	4 Aug 2015, 2:37:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1316068
Run time	6 days 6 hours 34 min 21 sec
CPU time	2 days 19 hours 38 min 32 sec
Validate state	Invalid
Credit	7,153.92
Device peak FLOPS	3.95 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.4.42</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:07:26 (2122): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:07:48 (2122): No heartbeat from core client for 30 sec - exiting 05:07:49 (2122): No heartbeat from core client for 30 sec - exiting cpdnmonitor: error reading file dataout/ocean_restart.day cpdnmonitor: error reading file dataout/atmos_restart.hold 05:54:19 (76661): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:00:02 (76661): No heartbeat from core client for 30 sec - exiting 06:00:03 (76661): No heartbeat from core client for 30 sec - exiting 06:00:04 (76661): No heartbeat from core client for 30 sec - exiting 06:00:05 (76661): No heartbeat from core client for 30 sec - exiting 06:00:06 (76661): No heartbeat from core client for 30 sec - exiting 06:00:07 (76661): No heartbeat from core client for 30 sec - exiting 06:00:08 (76661): No heartbeat from core client for 30 sec - exiting 06:00:09 (76661): No heartbeat from core client for 30 sec - exiting 06:00:10 (76661): No heartbeat from core client for 30 sec - exiting 06:00:11 (76661): No heartbeat from core client for 30 sec - exiting 06:00:12 (76661): No heartbeat from core client for 30 sec - exiting 06:00:13 (76661): No heartbeat from core client for 30 sec - exiting 06:00:15 (76661): No heartbeat from core client for 30 sec - exiting 06:00:16 (76661): No heartbeat from core client for 30 sec - exiting 06:00:17 (76661): No heartbeat from core client for 30 sec - exiting 06:00:18 (76661): No heartbeat from core client for 30 sec - exiting 06:00:19 (76661): No heartbeat from core client for 30 sec - exiting 06:00:20 (76661): No heartbeat from core client for 30 sec - exiting 06:00:21 (76661): No heartbeat from core client for 30 sec - exiting 06:00:22 (76661): No heartbeat from core client for 30 sec - exiting 06:00:23 (76661): No heartbeat from core client for 30 sec - exiting 06:00:24 (76661): No heartbeat from core client for 30 sec - exiting 06:00:25 (76661): No heartbeat from core client for 30 sec - exiting 06:00:26 (76661): No heartbeat from core client for 30 sec - exiting 06:00:27 (76661): No heartbeat from core client for 30 sec - exiting 06:00:28 (76661): No heartbeat from core client for 30 sec - exiting 06:00:29 (76661): No heartbeat from core client for 30 sec - exiting 06:00:31 (76661): No heartbeat from core client for 30 sec - exiting 06:00:32 (76661): No heartbeat from core client for 30 sec - exiting 06:00:33 (76661): No heartbeat from core client for 30 sec - exiting 06:00:34 (76661): No heartbeat from core client for 30 sec - exiting 06:00:35 (76661): No heartbeat from core client for 30 sec - exiting 06:00:36 (76661): No heartbeat from core client for 30 sec - exiting 06:00:37 (76661): No heartbeat from core client for 30 sec - exiting 06:00:38 (76661): No heartbeat from core client for 30 sec - exiting 06:00:39 (76661): No heartbeat from core client for 30 sec - exiting 06:00:40 (76661): No heartbeat from core client for 30 sec - exiting 06:00:41 (76661): No heartbeat from core client for 30 sec - exiting 06:00:42 (76661): No heartbeat from core client for 30 sec - exiting 06:00:43 (76661): No heartbeat from core client for 30 sec - exiting 06:00:44 (76661): No heartbeat from core client for 30 sec - exiting 06:00:46 (76661): No heartbeat from core client for 30 sec - exiting 06:00:47 (76661): No heartbeat from core client for 30 sec - exiting 06:00:48 (76661): No heartbeat from core client for 30 sec - exiting 06:00:49 (76661): No heartbeat from core client for 30 sec - exiting 06:00:50 (76661): No heartbeat from core client for 30 sec - exiting 06:00:51 (76661): No heartbeat from core client for 30 sec - exiting 06:00:52 (76661): No heartbeat from core client for 30 sec - exiting 06:00:53 (76661): No heartbeat from core client for 30 sec - exiting 06:00:54 (76661): No heartbeat from core client for 30 sec - exiting 06:00:55 (76661): No heartbeat from core client for 30 sec - exiting 06:00:56 (76661): No heartbeat from core client for 30 sec - exiting 06:00:57 (76661): No heartbeat from core client for 30 sec - exiting 06:00:58 (76661): No heartbeat from core client for 30 sec - exiting 06:00:59 (76661): No heartbeat from core client for 30 sec - exiting 06:01:00 (76661): No heartbeat from core client for 30 sec - exiting 06:03:00 (77639): No heartbeat from core client for 30 sec - exiting 06:03:27 (77639): No heartbeat from core client for 30 sec - exiting 06:03:28 (77639): No heartbeat from core client for 30 sec - exiting 06:03:29 (77639): No heartbeat from core client for 30 sec - exiting 06:03:30 (77639): No heartbeat from core client for 30 sec - exiting 06:03:31 (77639): No heartbeat from core client for 30 sec - exiting 06:03:32 (77639): No heartbeat from core client for 30 sec - exiting 06:03:33 (77639): No heartbeat from core client for 30 sec - exiting 06:03:34 (77639): No heartbeat from core client for 30 sec - exiting 06:03:35 (77639): No heartbeat from core client for 30 sec - exiting 06:03:36 (77639): No heartbeat from core client for 30 sec - exiting 06:03:37 (77639): No heartbeat from core client for 30 sec - exiting 06:03:38 (77639): No heartbeat from core client for 30 sec - exiting 06:03:40 (77639): No heartbeat from core client for 30 sec - exiting 06:03:41 (77639): No heartbeat from core client for 30 sec - exiting 06:03:42 (77639): No heartbeat from core client for 30 sec - exiting 06:03:43 (77639): No heartbeat from core client for 30 sec - exiting 06:03:44 (77639): No heartbeat from core client for 30 sec - exiting 06:03:45 (77639): No heartbeat from core client for 30 sec - exiting 06:03:46 (77639): No heartbeat from core client for 30 sec - exiting 06:03:47 (77639): No heartbeat from core client for 30 sec - exiting 06:03:48 (77639): No heartbeat from core client for 30 sec - exiting 06:03:49 (77639): No heartbeat from core client for 30 sec - exiting 06:03:50 (77639): No heartbeat from core client for 30 sec - exiting 06:03:51 (77639): No heartbeat from core client for 30 sec - exiting 06:04:38 (77639): No heartbeat from core client for 30 sec - exiting 06:04:39 (77639): No heartbeat from core client for 30 sec - exiting 06:04:40 (77639): No heartbeat from core client for 30 sec - exiting 06:04:41 (77639): No heartbeat from core client for 30 sec - exiting 06:04:42 (77639): No heartbeat from core client for 30 sec - exiting 06:04:43 (77639): No heartbeat from core client for 30 sec - exiting 06:04:44 (77639): No heartbeat from core client for 30 sec - exiting 06:04:45 (77639): No heartbeat from core client for 30 sec - exiting 06:04:46 (77639): No heartbeat from core client for 30 sec - exiting 06:04:47 (77639): No heartbeat from core client for 30 sec - exiting 06:04:49 (77639): No heartbeat from core client for 30 sec - exiting 06:04:50 (77639): No heartbeat from core client for 30 sec - exiting 06:04:51 (77639): No heartbeat from core client for 30 sec - exiting 06:04:52 (77639): No heartbeat from core client for 30 sec - exiting 06:04:53 (77639): No heartbeat from core client for 30 sec - exiting 06:04:54 (77639): No heartbeat from core client for 30 sec - exiting 06:04:55 (77639): No heartbeat from core client for 30 sec - exiting 06:04:56 (77639): No heartbeat from core client for 30 sec - exiting 06:04:57 (77639): No heartbeat from core client for 30 sec - exiting 06:04:58 (77639): No heartbeat from core client for 30 sec - exiting 06:04:59 (77639): No heartbeat from core client for 30 sec - exiting 06:05:00 (77639): No heartbeat from core client for 30 sec - exiting 06:05:01 (77639): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:06:34 (77722): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:08:33 (77747): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:09:31 (77777): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:11:27 (77817): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:13:20 (77840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:15:15 (77875): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:15:16 (77875): No heartbeat from core client for 30 sec - exiting 06:17:09 (77897): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:19:10 (77932): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_xby2_1980_40_010022427/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0042E622 Unknown Unknown Unknown hadcm3n_um_6.07_i 0042D0EB Unknown Unknown Unknown hadcm3n_um_6.07_i 003EF4DD Unknown Unknown Unknown hadcm3n_um_6.07_i 003A66F5 Unknown Unknown Unknown hadcm3n_um_6.07_i 003A5E57 UnknowSuspended CPDN Monitor - Suspend request from BOINC... OPEN: File Creation Failed: No space left on device OPEN: Unable to Open File dataout/xby2ko.dak43n0 for Read/Write Model crashed: DUMPCTL : Fail to open output dump - may already exist tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_xby2_1980_40_010022427/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0042E622 Unknown Unknown Unknown hadcm3n_um_6.07_i 0042D0EB Unknown Unknown Unknown hadcm3n_um_6.07_i 003EF4DD Unknown Unknown Unknown hadcm3n_um_6.07_i 003A66F5 Unknown Unknown Unknown hadcm3n_um_6.07_i 003A5E57 Unknown Unknown Unknown hadcm3n_um_6.07_i 003E1E32 Unknown Unknown Unknown hadcm3n_um_6.07_i 003DF4CC Unknown Unknown Unknown hadcm3n_um_6.07_i 003651C5 Unknown Unknown Unknown hadcm3n_um_6.07_i 002DE95F Unknown Unknown Unknown hadcm3n_um_6.07_i 00389F90 Unknown Unknown Unknown hadcm3n_um_6.07_i 000048E1 Unknown Unknown Unknown hadcm3n_um_6.07_i 000027FB Unknown Unknown Unknown hadcm3n_um_6.07_i 00002729 Unknown Unknown Unknown Unknown 00000003 Unknown Unknown Unknown Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_xby2_1980_40_010022427/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0042E622 Unknown Unknown Unknown hadcm3n_um_6.07_i 0042D0EB Unknown Unknown Unknown hadcm3n_um_6.07_i 003EF4DD Unknown Unknown Unknown hadcm3n_um_6.07_i 003A66F5 Unknown Unknown Unknown hadcm3n_um_6.07_i 003A5E57 Unknown Unknown Unknown hadcm3n_um_6.07_i 003E1E32 Unknown Unknown Unknown hadcm3n_um_6.07_i 003DF4CC Unknown Unknown Unknown hadcm3n_um_6.07_i 003651C5 Unknown Unknown Unknown hadcm3n_um_6.07_i 002DE95F Unknown Unknown Unknown hadcm3n_um_6.07_i 00389F90 Unknown Unknown Unknown hadcm3n_um_6.07_i 000048E1 Unknown Unknown Unknown hadcm3n_um_6.07_i 000027FB Unknown Unknown Unknown hadcm3n_um_6.07_i 00002729 Unknown Unknown Unknown Unknown 00000003 Unknown Unknown Unknown Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_xby2_1980_40_010022427/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0042E622 Unknown Unknown Unknown hadcm3n_um_6.07_i 0042D0EB Unknown Unknown Unknown hadcm3n_um_6.07_i 003EF4DD Unknown Unknown Unknown hadcm3n_um_6.07_i 003A66F5 Unknown Unknown Unknown hadcm3n_um_6.07_i 003A5E57 Unknown Unknown Unknown hadcm3n_um_6.07_i 003E1E32 Unknown Unknown Unknown hadcm3n_um_6.07_i 003DF4CC Unknown Unknown Unknown hadcm3n_um_6.07_i 003651C5 Unknown Unknown Unknown hadcm3n_um_6.07_i 002DE95F Unknown Unknown Unknown hadcm3n_um_6.07_i 00389F90 Unknown Unknown Unknown hadcm3n_um_6.07_i Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Aug 2015 00:04:28	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	596,160	436,903	0.7329
03 Aug 2015 17:57:49	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	570,240	418,871	0.7346
03 Aug 2015 11:56:14	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	544,320	400,682	0.7361
03 Aug 2015 05:33:21	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	518,400	382,830	0.7385
02 Aug 2015 23:13:53	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	492,480	364,802	0.7407
02 Aug 2015 16:41:53	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	466,560	346,832	0.7434
02 Aug 2015 11:45:38	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	440,640	328,894	0.7464
02 Aug 2015 04:42:26	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	414,720	310,930	0.7497
01 Aug 2015 22:29:23	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	388,800	292,919	0.7534
01 Aug 2015 16:52:44	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	362,880	275,067	0.7580
01 Aug 2015 10:40:33	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	336,960	257,118	0.7631
01 Aug 2015 03:35:46	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	311,040	238,810	0.7678
31 Jul 2015 20:34:05	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	285,120	218,703	0.7671
31 Jul 2015 13:51:12	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	259,200	198,540	0.7660
31 Jul 2015 07:13:02	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	233,280	178,410	0.7648
30 Jul 2015 23:28:55	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	207,360	158,463	0.7642
30 Jul 2015 16:21:44	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	181,440	138,575	0.7638
30 Jul 2015 11:04:47	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	155,520	119,331	0.7673
30 Jul 2015 03:55:56	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	129,600	100,147	0.7727
29 Jul 2015 19:49:13	1316068	18743113	hadcm3n_xby2_1980_40_010022427_4	103,680	80,210	0.7736