climateprediction.net home page
Task 13613648

Task 13613648

Name hadcm3n_ya62_1900_40_007526667_3
Workunit 7724142
Created 6 Nov 2011, 18:56:32 UTC
Sent 19 Nov 2011, 5:16:42 UTC
Report deadline 18 Feb 2012, 12:43:53 UTC
Received 24 Dec 2011, 12:40:44 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1157390
Run time 25 days 2 hours 21 min 5 sec
CPU time 25 days 1 hours 13 min 9 sec
Validate state Invalid
Credit 9,331.20
Device peak FLOPS 1.37 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
11:10:16 (2341): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:54:14 (2422): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:58:37 (4340): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:01:58 (4399): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
05:51:31 (4455): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:46:38 (4553): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:47:29 (4718): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:48:19 (4782): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:49:24 (4841): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:51:07 (4908): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:52:50 (5062): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:54:33 (5146): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:56:17 (5208): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
07:57:07 (5267): No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
07:58:02 (5324): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:59:45 (5384): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:01:28 (5443): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:03:14 (5498): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:04:58 (5522): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
08:05:48 (5581): No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
08:06:41 (5640): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:08:24 (5699): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:10:08 (5760): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:11:54 (5814): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:13:37 (5839): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
08:14:27 (5898): No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
08:15:21 (5956): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:17:06 (6015): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:18:51 (6073): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:20:36 (6094): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:22:20 (6155): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
08:23:10 (6214): No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
08:24:02 (6273): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:25:45 (6332): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:27:29 (6387): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:29:13 (6445): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:30:57 (6468): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
08:31:47 (6527): No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
08:32:41 (6585): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:34:24 (6643): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:36:08 (6701): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:37:50 (6755): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:39:35 (6816): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:41:18 (6844): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
08:42:09 (6903): No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
08:43:02 (6958): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:44:48 (7016): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:46:31 (7078): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:48:15 (7132): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:49:58 (7156): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
08:50:48 (7215): No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - No 'heartbeat' from BOINC...
08:51:41 (7273): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:53:27 (7332): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:55:10 (7386): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:56:55 (7445): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
16:42:41 (7470): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:42:44 (7470): No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
16:31:12 (14210): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
18:25:29 (21781): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
23:56:14 (24421): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
23:56:17 (24421): No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - Suspend request from BOINC...
06:22:30 (25686): No heartbeat from core client for 30 sec - exiting
06:22:31 (25686): No heartbeat from core client for 30 sec - exiting
06:22:32 (25686): No heartbeat from core client for 30 sec - exiting
06:22:33 (25686): No heartbeat from core client for 30 sec - exiting
06:22:34 (25686): No heartbeat from core client for 30 sec - exiting
06:22:35 (25686): No heartbeat from core client for 30 sec - exiting
06:22:36 (25686): No heartbeat from core client for 30 sec - exiting
06:22:37 (25686): No heartbeat from core client for 30 sec - exiting
06:22:38 (25686): No heartbeat from core client for 30 sec - exiting
06:22:39 (25686): No heartbeat from core client for 30 sec - exiting
06:22:40 (25686): No heartbeat from core client for 30 sec - exiting
06:22:41 (25686): No heartbeat from core client for 30 sec - exiting
06:22:42 (25686): No heartbeat from core client for 30 sec - exiting
06:22:43 (25686): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_ya62_1900_40_007526667/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_ya62_1900_40_007526667/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_ya62_1900_40_007526667/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_ya62_1900_40_007526667/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_ya62_1900_40_007526667/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_ya62_1900_40_007526667/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_ya62_1900_40_007526667/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_ya62_1900_40_007526667/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_ya62_1900_40_007526667/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_ya62_1900_40_007526667/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_ya62_1900_40_007526667/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_ya62_1900_40_007526667/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
24 Dec 2011 12:45:42 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 777,600 2,164,455 2.7835
23 Dec 2011 16:27:51 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 751,680 2,091,869 2.7829
22 Dec 2011 19:41:40 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 725,760 2,019,065 2.7820
21 Dec 2011 23:40:33 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 699,840 1,947,367 2.7826
21 Dec 2011 03:41:42 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 673,920 1,875,955 2.7836
20 Dec 2011 08:17:43 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 648,000 1,804,860 2.7853
19 Dec 2011 11:53:23 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 622,080 1,733,064 2.7859
18 Dec 2011 15:38:54 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 596,160 1,660,632 2.7855
17 Dec 2011 20:29:28 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 570,240 1,588,776 2.7862
17 Dec 2011 00:21:40 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 544,320 1,516,690 2.7864
16 Dec 2011 03:55:18 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 518,400 1,444,625 2.7867
15 Dec 2011 07:23:57 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 492,480 1,372,639 2.7872
14 Dec 2011 11:13:27 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 466,560 1,300,186 2.7867
13 Dec 2011 14:58:07 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 440,640 1,227,577 2.7859
12 Dec 2011 18:43:15 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 414,720 1,155,094 2.7852
11 Dec 2011 22:40:30 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 388,800 1,082,989 2.7855
11 Dec 2011 02:35:20 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 362,880 1,010,476 2.7846
10 Dec 2011 06:40:38 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 336,960 937,882 2.7834
09 Dec 2011 10:43:07 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 311,040 865,051 2.7812
08 Dec 2011 13:49:27 1157390 13613648 hadcm3n_ya62_1900_40_007526667_3 285,120 792,687 2.7802


©2024 cpdn.org