climateprediction.net home page
Task 22277189

Task 22277189

Name oifs_43r3_ps_0860_2005050100_123_974_12191504_0
Workunit 12191504
Created 23 Dec 2022, 11:40:13 UTC
Sent 14 Jan 2023, 23:56:12 UTC
Report deadline 13 Feb 2023, 23:56:12 UTC
Received 15 Jan 2023, 17:17:50 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 5 (0x00000005) Unknown error code
Computer ID 1531398
Run time 11 hours 42 min 22 sec
CPU time 11 hours 28 min 40 sec
Validate state Invalid
Credit 6,652.80
Device peak FLOPS 7.92 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,803.62 MB
Peak swap size 4,978.07 MB
Peak disk usage 3,694.67 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 5 (0x5, -251)</message>
<stderr_txt>
+CPU= 19.448
  09:30:36 STEP  980 H= 980:00 +CPU= 21.652
  09:31:11 STEP  981 H= 981:00 +CPU= 34.879
  09:31:33 STEP  982 H= 982:00 +CPU= 22.117
  09:31:55 STEP  983 H= 983:00 +CPU= 21.515
  09:32:33 STEP  984 H= 984:00 +CPU= 37.703
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_40.zip
Uploading the intermediate file: upload_file_40.zip
  09:32:53 STEP  985 H= 985:00 +CPU= 20.057
  09:33:11 STEP  986 H= 986:00 +CPU= 18.782
  09:33:45 STEP  987 H= 987:00 +CPU= 33.501
  09:34:06 STEP  988 H= 988:00 +CPU= 20.681
  09:34:26 STEP  989 H= 989:00 +CPU= 20.670
  09:35:02 STEP  990 H= 990:00 +CPU= 35.091
  09:35:23 STEP  991 H= 991:00 +CPU= 21.396
  09:35:45 STEP  992 H= 992:00 +CPU= 21.937
  09:36:19 STEP  993 H= 993:00 +CPU= 34.210
  09:36:42 STEP  994 H= 994:00 +CPU= 22.398
  09:37:02 STEP  995 H= 995:00 +CPU= 20.190
  09:37:40 STEP  996 H= 996:00 +CPU= 37.549
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+000996
  09:38:00 STEP  997 H= 997:00 +CPU= 20.157
  09:38:21 STEP  998 H= 998:00 +CPU= 21.576
  09:38:55 STEP  999 H= 999:00 +CPU= 33.729
  09:39:17 STEP 1000 H=1000:00 +CPU= 21.742
  09:39:38 STEP 1001 H=1001:00 +CPU= 20.801
  09:40:15 STEP 1002 H=1002:00 +CPU= 36.567
  09:40:38 STEP 1003 H=1003:00 +CPU= 23.344
  09:41:02 STEP 1004 H=1004:00 +CPU= 24.260
  09:41:40 STEP 1005 H=1005:00 +CPU= 37.756
  09:42:05 STEP 1006 H=1006:00 +CPU= 24.894
  09:42:29 STEP 1007 H=1007:00 +CPU= 24.028
  09:43:09 STEP 1008 H=1008:00 +CPU= 39.329
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_41.zip
Uploading the intermediate file: upload_file_41.zip
  09:43:30 STEP 1009 H=1009:00 +CPU= 21.506
  09:43:52 STEP 1010 H=1010:00 +CPU= 21.258
  09:44:25 STEP 1011 H=1011:00 +CPU= 33.565
  09:44:46 STEP 1012 H=1012:00 +CPU= 20.682
  09:45:09 STEP 1013 H=1013:00 +CPU= 22.512
  09:45:44 STEP 1014 H=1014:00 +CPU= 35.237
  09:46:05 STEP 1015 H=1015:00 +CPU= 20.917
  09:46:27 STEP 1016 H=1016:00 +CPU= 21.128
  09:47:00 STEP 1017 H=1017:00 +CPU= 33.748
  09:47:23 STEP 1018 H=1018:00 +CPU= 22.931
  09:47:44 STEP 1019 H=1019:00 +CPU= 20.955
  09:48:20 STEP 1020 H=1020:00 +CPU= 35.842
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001020
  09:48:41 STEP 1021 H=1021:00 +CPU= 20.428
  09:49:02 STEP 1022 H=1022:00 +CPU= 20.892
  09:49:32 STEP 1023 H=1023:00 +CPU= 30.748
  09:49:52 STEP 1024 H=1024:00 +CPU= 19.269
  09:50:10 STEP 1025 H=1025:00 +CPU= 18.737
  09:50:42 STEP 1026 H=1026:00 +CPU= 31.641
  09:51:03 STEP 1027 H=1027:00 +CPU= 20.490
  09:51:24 STEP 1028 H=1028:00 +CPU= 21.399
  09:51:54 STEP 1029 H=1029:00 +CPU= 30.093
  09:52:15 STEP 1030 H=1030:00 +CPU= 20.703
  09:52:35 STEP 1031 H=1031:00 +CPU= 19.594
  09:53:16 STEP 1032 H=1032:00 +CPU= 40.829
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_42.zip
Uploading the intermediate file: upload_file_42.zip
  09:53:38 STEP 1033 H=1033:00 +CPU= 22.106
  09:54:01 STEP 1034 H=1034:00 +CPU= 23.234
  09:54:39 STEP 1035 H=1035:00 +CPU= 37.482
  09:55:05 STEP 1036 H=1036:00 +CPU= 26.472
  09:55:28 STEP 1037 H=1037:00 +CPU= 22.686
  09:56:04 STEP 1038 H=1038:00 +CPU= 35.401
  09:56:24 STEP 1039 H=1039:00 +CPU= 20.205
  09:56:46 STEP 1040 H=1040:00 +CPU= 22.104
  09:57:20 STEP 1041 H=1041:00 +CPU= 33.315
  09:57:43 STEP 1042 H=1042:00 +CPU= 23.212
  09:58:06 STEP 1043 H=1043:00 +CPU= 22.570
  09:58:43 STEP 1044 H=1044:00 +CPU= 36.946
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001044
  09:59:04 STEP 1045 H=1045:00 +CPU= 21.513
  09:59:24 STEP 1046 H=1046:00 +CPU= 19.966
  09:59:56 STEP 1047 H=1047:00 +CPU= 31.443
  10:00:16 STEP 1048 H=1048:00 +CPU= 20.097
  10:00:35 STEP 1049 H=1049:00 +CPU= 18.828
  10:01:11 STEP 1050 H=1050:00 +CPU= 35.623
  10:01:33 STEP 1051 H=1051:00 +CPU= 21.408
  10:01:55 STEP 1052 H=1052:00 +CPU= 22.790
  10:02:30 STEP 1053 H=1053:00 +CPU= 34.087
  10:02:52 STEP 1054 H=1054:00 +CPU= 21.817
  10:03:14 STEP 1055 H=1055:00 +CPU= 22.881
  10:03:53 STEP 1056 H=1056:00 +CPU= 37.901
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_43.zip
Uploading the intermediate file: upload_file_43.zip
  10:04:13 STEP 1057 H=1057:00 +CPU= 20.509
  10:04:33 STEP 1058 H=1058:00 +CPU= 19.953
  10:05:05 STEP 1059 H=1059:00 +CPU= 31.455
  10:05:26 STEP 1060 H=1060:00 +CPU= 20.663
  10:05:45 STEP 1061 H=1061:00 +CPU= 19.109
  10:06:17 STEP 1062 H=1062:00 +CPU= 32.386
  10:06:38 STEP 1063 H=1063:00 +CPU= 20.360
  10:07:02 STEP 1064 H=1064:00 +CPU= 24.540
  10:07:42 STEP 1065 H=1065:00 +CPU= 39.242
  10:08:05 STEP 1066 H=1066:00 +CPU= 23.318
  10:08:25 STEP 1067 H=1067:00 +CPU= 20.473
  10:09:02 STEP 1068 H=1068:00 +CPU= 36.460
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001068
  10:09:23 STEP 1069 H=1069:00 +CPU= 20.519
  10:09:43 STEP 1070 H=1070:00 +CPU= 20.001
  10:10:14 STEP 1071 H=1071:00 +CPU= 31.061
  10:10:34 STEP 1072 H=1072:00 +CPU= 20.079
  10:10:53 STEP 1073 H=1073:00 +CPU= 19.248
  10:11:26 STEP 1074 H=1074:00 +CPU= 32.690
  10:11:47 STEP 1075 H=1075:00 +CPU= 20.613
  10:12:06 STEP 1076 H=1076:00 +CPU= 19.496
  10:12:38 STEP 1077 H=1077:00 +CPU= 32.168
  10:12:58 STEP 1078 H=1078:00 +CPU= 19.675
  10:13:18 STEP 1079 H=1079:00 +CPU= 19.865
  10:13:56 STEP 1080 H=1080:00 +CPU= 37.919
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_44.zip
Uploading the intermediate file: upload_file_44.zip
  10:14:16 STEP 1081 H=1081:00 +CPU= 19.561
Uploading trickle at timestep: 3888000
  10:14:37 STEP 1082 H=1082:00 +CPU= 20.827
  10:15:08 STEP 1083 H=1083:00 +CPU= 31.276
  10:15:28 STEP 1084 H=1084:00 +CPU= 20.141
  10:15:50 STEP 1085 H=1085:00 +CPU= 21.588
  10:16:24 STEP 1086 H=1086:00 +CPU= 34.698
  10:16:46 STEP 1087 H=1087:00 +CPU= 21.876
  10:17:08 STEP 1088 H=1088:00 +CPU= 21.140
  10:17:43 STEP 1089 H=1089:00 +CPU= 34.987
  10:18:04 STEP 1090 H=1090:00 +CPU= 21.545
  10:18:26 STEP 1091 H=1091:00 +CPU= 21.684
  10:19:04 STEP 1092 H=1092:00 +CPU= 37.997
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001092
  10:19:27 STEP 1093 H=1093:00 +CPU= 22.595
  10:19:50 STEP 1094 H=1094:00 +CPU= 22.998
  10:20:20 STEP 1095 H=1095:00 +CPU= 30.592
  10:20:41 STEP 1096 H=1096:00 +CPU= 20.869
  10:21:01 STEP 1097 H=1097:00 +CPU= 19.820
  10:21:36 STEP 1098 H=1098:00 +CPU= 35.048
  10:21:59 STEP 1099 H=1099:00 +CPU= 22.329
  10:22:23 STEP 1100 H=1100:00 +CPU= 22.160
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0860
(argv4) batchid: 974
(argv5) wuid: 12191504
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/10
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fb423cc1235f6a26832feb361525e6db to: /var/lib/boinc/slots/10/oifs_43r3_ps_0860_2005050100_123_974_12191504.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/10/oifs_43r3_ps_0860_2005050100_123_974_12191504.zip
ic_ancil_file: ic_ancil_12191504
ifsdata_file: ifsdata_12191504
climate_data_file: clim_data_12191504
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3f89e0f5e28a3478ecf91e0b3b4b4ad7 to: /var/lib/boinc/slots/10/ic_ancil_12191504.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/10/ic_ancil_12191504.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_253bf9d244527d94366e5ad2b93f72fd to: /var/lib/boinc/slots/10/ifsdata/ifsdata_12191504.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/10/ifsdata/ifsdata_12191504.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/10/159l_2/clim_data_12191504.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/10/159l_2/clim_data_12191504.zip
Checking for progress XML file: /var/lib/boinc/slots/10/progress_file_12191504.xml
Opened progress file ok : /var/lib/boinc/slots/10/progress_file_12191504.xml
-- Model is restarting --
Adjusting last_iter, 1101, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/10/progress_file_12191504.xml
last_cpu_time: 32276
upload_file_number: 45
last_iter: 1079
last_upload: 3888000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921
The child process has been launched with process id: 28543
Executing the command: /var/lib/boinc/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:28543:28543] [20230115:122820:1673785700:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:28:50 STEP 1080 H=1080:00 +CPU= 26.497
  12:29:13 STEP 1081 H=1081:00 +CPU= 21.397
  12:29:36 STEP 1082 H=1082:00 +CPU= 21.738
  12:30:10 STEP 1083 H=1083:00 +CPU= 32.098
  12:30:33 STEP 1084 H=1084:00 +CPU= 21.221
  12:30:55 STEP 1085 H=1085:00 +CPU= 20.490
  12:31:29 STEP 1086 H=1086:00 +CPU= 31.950
  12:31:51 STEP 1087 H=1087:00 +CPU= 20.449
  12:32:12 STEP 1088 H=1088:00 +CPU= 20.109
  12:32:47 STEP 1089 H=1089:00 +CPU= 32.482
  12:33:08 STEP 1090 H=1090:00 +CPU= 20.438
  12:33:30 STEP 1091 H=1091:00 +CPU= 20.431
  12:34:10 STEP 1092 H=1092:00 +CPU= 38.021
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001092
  12:34:32 STEP 1093 H=1093:00 +CPU= 19.919
  12:34:55 STEP 1094 H=1094:00 +CPU= 21.815
  12:35:28 STEP 1095 H=1095:00 +CPU= 32.044
  12:35:51 STEP 1096 H=1096:00 +CPU= 22.242
  12:36:13 STEP 1097 H=1097:00 +CPU= 21.264
  12:36:47 STEP 1098 H=1098:00 +CPU= 32.433
  12:37:08 STEP 1099 H=1099:00 +CPU= 20.805
  12:37:29 STEP 1100 H=1100:00 +CPU= 19.877
  12:38:02 STEP 1101 H=1101:00 +CPU= 32.641
  12:38:25 STEP 1102 H=1102:00 +CPU= 21.933
  12:38:46 STEP 1103 H=1103:00 +CPU= 19.790
  12:39:27 STEP 1104 H=1104:00 +CPU= 39.757
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_45.zip
Uploading the intermediate file: upload_file_45.zip
  12:39:45 STEP 1105 H=1105:00 +CPU= 18.203
  12:40:07 STEP 1106 H=1106:00 +CPU= 21.473
  12:40:40 STEP 1107 H=1107:00 +CPU= 32.120
  12:41:02 STEP 1108 H=1108:00 +CPU= 21.147
  12:41:24 STEP 1109 H=1109:00 +CPU= 21.014
  12:41:59 STEP 1110 H=1110:00 +CPU= 32.948
  12:42:21 STEP 1111 H=1111:00 +CPU= 21.281
  12:42:43 STEP 1112 H=1112:00 +CPU= 21.324
  12:43:17 STEP 1113 H=1113:00 +CPU= 32.900
  12:43:40 STEP 1114 H=1114:00 +CPU= 22.733
  12:44:00 STEP 1115 H=1115:00 +CPU= 18.561
  12:44:38 STEP 1116 H=1116:00 +CPU= 37.180
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001116
  12:44:58 STEP 1117 H=1117:00 +CPU= 19.715
  12:45:20 STEP 1118 H=1118:00 +CPU= 21.082
  12:45:54 STEP 1119 H=1119:00 +CPU= 32.892
  12:46:16 STEP 1120 H=1120:00 +CPU= 21.262
  12:46:38 STEP 1121 H=1121:00 +CPU= 21.148
  12:47:11 STEP 1122 H=1122:00 +CPU= 32.496
  12:47:33 STEP 1123 H=1123:00 +CPU= 20.924
  12:47:56 STEP 1124 H=1124:00 +CPU= 22.019
  12:48:29 STEP 1125 H=1125:00 +CPU= 31.966
  12:48:51 STEP 1126 H=1126:00 +CPU= 21.442
  12:49:13 STEP 1127 H=1127:00 +CPU= 20.681
  12:49:54 STEP 1128 H=1128:00 +CPU= 39.961
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_46.zip
Uploading the intermediate file: upload_file_46.zip
  12:50:15 STEP 1129 H=1129:00 +CPU= 20.040
  12:50:36 STEP 1130 H=1130:00 +CPU= 20.952
  12:51:10 STEP 1131 H=1131:00 +CPU= 32.652
  12:51:32 STEP 1132 H=1132:00 +CPU= 21.101
  12:51:54 STEP 1133 H=1133:00 +CPU= 21.461
  12:52:29 STEP 1134 H=1134:00 +CPU= 33.834
  12:52:50 STEP 1135 H=1135:00 +CPU= 19.965
  12:53:13 STEP 1136 H=1136:00 +CPU= 22.486
  12:53:46 STEP 1137 H=1137:00 +CPU= 32.244
  12:54:09 STEP 1138 H=1138:00 +CPU= 21.940
  12:54:29 STEP 1139 H=1139:00 +CPU= 19.169
  12:55:07 STEP 1140 H=1140:00 +CPU= 37.431
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001140
  12:55:29 STEP 1141 H=1141:00 +CPU= 20.732
  12:55:51 STEP 1142 H=1142:00 +CPU= 21.424
  12:56:24 STEP 1143 H=1143:00 +CPU= 32.461
  12:56:47 STEP 1144 H=1144:00 +CPU= 21.856
  12:57:08 STEP 1145 H=1145:00 +CPU= 20.610
  12:57:42 STEP 1146 H=1146:00 +CPU= 32.846
  12:58:04 STEP 1147 H=1147:00 +CPU= 21.374
  12:58:26 STEP 1148 H=1148:00 +CPU= 21.762
  12:59:00 STEP 1149 H=1149:00 +CPU= 32.253
  12:59:21 STEP 1150 H=1150:00 +CPU= 20.750
  12:59:44 STEP 1151 H=1151:00 +CPU= 21.935
  13:00:23 STEP 1152 H=1152:00 +CPU= 38.510
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_47.zip
Uploading the intermediate file: upload_file_47.zip
  13:00:45 STEP 1153 H=1153:00 +CPU= 21.248
  13:01:07 STEP 1154 H=1154:00 +CPU= 21.163
  13:01:40 STEP 1155 H=1155:00 +CPU= 31.871
  13:02:01 STEP 1156 H=1156:00 +CPU= 20.502
  13:02:23 STEP 1157 H=1157:00 +CPU= 21.120
  13:02:58 STEP 1158 H=1158:00 +CPU= 33.700
  13:03:20 STEP 1159 H=1159:00 +CPU= 21.451
  13:03:43 STEP 1160 H=1160:00 +CPU= 21.974
  13:04:16 STEP 1161 H=1161:00 +CPU= 32.767
  13:04:38 STEP 1162 H=1162:00 +CPU= 20.874
  13:04:59 STEP 1163 H=1163:00 +CPU= 20.794
  13:05:37 STEP 1164 H=1164:00 +CPU= 36.748
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001164
  13:05:58 STEP 1165 H=1165:00 +CPU= 20.871
  13:06:20 STEP 1166 H=1166:00 +CPU= 20.988
  13:06:54 STEP 1167 H=1167:00 +CPU= 32.507
  13:07:15 STEP 1168 H=1168:00 +CPU= 20.790
  13:07:36 STEP 1169 H=1169:00 +CPU= 20.332
  13:08:11 STEP 1170 H=1170:00 +CPU= 32.826
  13:08:34 STEP 1171 H=1171:00 +CPU= 21.952
  13:08:54 STEP 1172 H=1172:00 +CPU= 19.528
  13:09:29 STEP 1173 H=1173:00 +CPU= 32.395
  13:09:52 STEP 1174 H=1174:00 +CPU= 21.880
  13:10:15 STEP 1175 H=1175:00 +CPU= 21.401
  13:10:55 STEP 1176 H=1176:00 +CPU= 37.981
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_48.zip
Uploading the intermediate file: upload_file_48.zip
  13:11:16 STEP 1177 H=1177:00 +CPU= 20.620
  13:11:41 STEP 1178 H=1178:00 +CPU= 22.818
  13:12:15 STEP 1179 H=1179:00 +CPU= 32.632
  13:12:37 STEP 1180 H=1180:00 +CPU= 21.294
  13:13:02 STEP 1181 H=1181:00 +CPU= 21.548
  13:13:40 STEP 1182 H=1182:00 +CPU= 31.611
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:14:16 STEP 1183 H=1183:00 +CPU= 20.497
  13:14:41 STEP 1184 H=1184:00 +CPU= 21.760
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:15:30 STEP 1185 H=1185:00 +CPU= 30.691
  13:15:52 STEP 1186 H=1186:00 +CPU= 22.328
  13:16:13 STEP 1187 H=1187:00 +CPU= 20.067
  13:16:52 STEP 1188 H=1188:00 +CPU= 37.558
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001188
  13:17:14 STEP 1189 H=1189:00 +CPU= 20.002
  13:17:39 STEP 1190 H=1190:00 +CPU= 21.490
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:18:26 STEP 1191 H=1191:00 +CPU= 32.276
  13:18:47 STEP 1192 H=1192:00 +CPU= 21.252
  13:19:10 STEP 1193 H=1193:00 +CPU= 22.768
  13:19:44 STEP 1194 H=1194:00 +CPU= 32.904
  13:20:04 STEP 1195 H=1195:00 +CPU= 19.677
  13:20:27 STEP 1196 H=1196:00 +CPU= 20.922
  13:21:03 STEP 1197 H=1197:00 +CPU= 33.268
  13:21:26 STEP 1198 H=1198:00 +CPU= 21.520
  13:21:51 STEP 1199 H=1199:00 +CPU= 22.091
  13:22:34 STEP 1200 H=1200:00 +CPU= 39.629
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_49.zip
Uploading the intermediate file: upload_file_49.zip
  13:22:57 STEP 1201 H=1201:00 +CPU= 20.966
  13:23:21 STEP 1202 H=1202:00 +CPU= 21.802
  13:23:57 STEP 1203 H=1203:00 +CPU= 32.879
  13:24:21 STEP 1204 H=1204:00 +CPU= 22.353
  13:24:45 STEP 1205 H=1205:00 +CPU= 21.679
  13:25:22 STEP 1206 H=1206:00 +CPU= 33.865
  13:25:46 STEP 1207 H=1207:00 +CPU= 21.893
  13:26:08 STEP 1208 H=1208:00 +CPU= 20.954
  13:26:46 STEP 1209 H=1209:00 +CPU= 34.657
  13:27:09 STEP 1210 H=1210:00 +CPU= 20.987
  13:27:33 STEP 1211 H=1211:00 +CPU= 21.814
  13:28:13 STEP 1212 H=1212:00 +CPU= 37.388
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001212
  13:28:37 STEP 1213 H=1213:00 +CPU= 21.588
  13:29:01 STEP 1214 H=1214:00 +CPU= 22.502
  13:29:34 STEP 1215 H=1215:00 +CPU= 31.638
  13:29:58 STEP 1216 H=1216:00 +CPU= 23.763
  13:30:21 STEP 1217 H=1217:00 +CPU= 21.973
  13:30:57 STEP 1218 H=1218:00 +CPU= 34.644
  13:31:18 STEP 1219 H=1219:00 +CPU= 20.604
  13:31:40 STEP 1220 H=1220:00 +CPU= 20.817
  13:32:13 STEP 1221 H=1221:00 +CPU= 32.171
  13:32:34 STEP 1222 H=1222:00 +CPU= 20.685
  13:32:54 STEP 1223 H=1223:00 +CPU= 19.788
  13:33:31 STEP 1224 H=1224:00 +CPU= 36.498
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_50.zip
Uploading the intermediate file: upload_file_50.zip
  13:33:55 STEP 1225 H=1225:00 +CPU= 23.621
  13:34:18 STEP 1226 H=1226:00 +CPU= 21.547
  13:34:53 STEP 1227 H=1227:00 +CPU= 33.650
  13:35:17 STEP 1228 H=1228:00 +CPU= 22.025
  13:35:40 STEP 1229 H=1229:00 +CPU= 21.095
  13:36:17 STEP 1230 H=1230:00 +CPU= 35.187
  13:36:41 STEP 1231 H=1231:00 +CPU= 21.029
  13:37:04 STEP 1232 H=1232:00 +CPU= 21.876
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:37:58 STEP 1233 H=1233:00 +CPU= 31.028
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:38:32 STEP 1234 H=1234:00 +CPU= 21.726
  13:38:53 STEP 1235 H=1235:00 +CPU= 20.630
  13:39:32 STEP 1236 H=1236:00 +CPU= 37.795
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001236
  13:39:54 STEP 1237 H=1237:00 +CPU= 21.362
  13:40:16 STEP 1238 H=1238:00 +CPU= 20.368
  13:40:50 STEP 1239 H=1239:00 +CPU= 33.624
  13:41:12 STEP 1240 H=1240:00 +CPU= 20.899
  13:41:35 STEP 1241 H=1241:00 +CPU= 21.996
  13:42:11 STEP 1242 H=1242:00 +CPU= 34.595
  13:42:34 STEP 1243 H=1243:00 +CPU= 21.987
  13:42:57 STEP 1244 H=1244:00 +CPU= 22.044
  13:43:31 STEP 1245 H=1245:00 +CPU= 33.007
  13:43:54 STEP 1246 H=1246:00 +CPU= 22.152
  13:44:15 STEP 1247 H=1247:00 +CPU= 20.822
  13:44:58 STEP 1248 H=1248:00 +CPU= 41.157
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_51.zip
Uploading the intermediate file: upload_file_51.zip
  13:45:20 STEP 1249 H=1249:00 +CPU= 20.741
  13:45:43 STEP 1250 H=1250:00 +CPU= 22.110
  13:46:18 STEP 1251 H=1251:00 +CPU= 33.647
  13:46:42 STEP 1252 H=1252:00 +CPU= 21.807
  13:47:05 STEP 1253 H=1253:00 +CPU= 21.858
  13:47:38 STEP 1254 H=1254:00 +CPU= 32.357
  13:48:00 STEP 1255 H=1255:00 +CPU= 20.950
  13:48:22 STEP 1256 H=1256:00 +CPU= 21.120
  13:48:56 STEP 1257 H=1257:00 +CPU= 33.147
  13:49:20 STEP 1258 H=1258:00 +CPU= 21.731
  13:49:42 STEP 1259 H=1259:00 +CPU= 20.735
  13:50:22 STEP 1260 H=1260:00 +CPU= 37.387
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001260
  13:50:43 STEP 1261 H=1261:00 +CPU= 19.588
  13:51:06 STEP 1262 H=1262:00 +CPU= 22.479
  13:51:40 STEP 1263 H=1263:00 +CPU= 32.952
  13:52:04 STEP 1264 H=1264:00 +CPU= 22.862
  13:52:27 STEP 1265 H=1265:00 +CPU= 22.218
  13:52:58 STEP 1266 H=1266:00 +CPU= 31.430
  13:53:20 STEP 1267 H=1267:00 +CPU= 21.259
  13:53:42 STEP 1268 H=1268:00 +CPU= 21.800
  13:54:16 STEP 1269 H=1269:00 +CPU= 32.943
  13:54:37 STEP 1270 H=1270:00 +CPU= 20.540
  13:55:00 STEP 1271 H=1271:00 +CPU= 21.996
  13:55:39 STEP 1272 H=1272:00 +CPU= 38.522
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_52.zip
Uploading the intermediate file: upload_file_52.zip
  13:56:02 STEP 1273 H=1273:00 +CPU= 22.153
  13:56:24 STEP 1274 H=1274:00 +CPU= 20.915
  13:56:59 STEP 1275 H=1275:00 +CPU= 34.019
  13:57:21 STEP 1276 H=1276:00 +CPU= 21.589
  13:57:42 STEP 1277 H=1277:00 +CPU= 20.105
  13:58:20 STEP 1278 H=1278:00 +CPU= 35.014
  13:58:43 STEP 1279 H=1279:00 +CPU= 20.996
  13:59:07 STEP 1280 H=1280:00 +CPU= 21.776
  13:59:45 STEP 1281 H=1281:00 +CPU= 34.231
  14:00:08 STEP 1282 H=1282:00 +CPU= 20.680
  14:00:33 STEP 1283 H=1283:00 +CPU= 21.426
  14:01:13 STEP 1284 H=1284:00 +CPU= 36.192
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001284
  14:01:38 STEP 1285 H=1285:00 +CPU= 21.259
  14:02:03 STEP 1286 H=1286:00 +CPU= 21.358
  14:02:40 STEP 1287 H=1287:00 +CPU= 33.552
  14:03:03 STEP 1288 H=1288:00 +CPU= 23.029
  14:03:24 STEP 1289 H=1289:00 +CPU= 20.218
  14:04:01 STEP 1290 H=1290:00 +CPU= 35.896
  14:04:21 STEP 1291 H=1291:00 +CPU= 19.749
  14:04:44 STEP 1292 H=1292:00 +CPU= 22.373
  14:05:18 STEP 1293 H=1293:00 +CPU= 32.638
  14:05:41 STEP 1294 H=1294:00 +CPU= 22.576
  14:06:05 STEP 1295 H=1295:00 +CPU= 22.624
  14:06:45 STEP 1296 H=1296:00 +CPU= 38.533
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_53.zip
Uploading the intermediate file: upload_file_53.zip
  14:07:09 STEP 1297 H=1297:00 +CPU= 22.453
  14:07:30 STEP 1298 H=1298:00 +CPU= 20.358
  14:08:04 STEP 1299 H=1299:00 +CPU= 32.637
  14:08:26 STEP 1300 H=1300:00 +CPU= 20.859
  14:08:49 STEP 1301 H=1301:00 +CPU= 21.517
  14:09:23 STEP 1302 H=1302:00 +CPU= 32.543
  14:09:45 STEP 1303 H=1303:00 +CPU= 20.815
  14:10:08 STEP 1304 H=1304:00 +CPU= 21.798
  14:10:42 STEP 1305 H=1305:00 +CPU= 32.736
  14:11:05 STEP 1306 H=1306:00 +CPU= 21.758
  14:11:26 STEP 1307 H=1307:00 +CPU= 20.860
  14:12:06 STEP 1308 H=1308:00 +CPU= 37.591
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001308
  14:12:27 STEP 1309 H=1309:00 +CPU= 20.741
  14:12:51 STEP 1310 H=1310:00 +CPU= 22.159
  14:13:26 STEP 1311 H=1311:00 +CPU= 33.834
  14:13:49 STEP 1312 H=1312:00 +CPU= 21.680
  14:14:11 STEP 1313 H=1313:00 +CPU= 21.533
  14:14:48 STEP 1314 H=1314:00 +CPU= 34.023
  14:15:10 STEP 1315 H=1315:00 +CPU= 20.650
  14:15:34 STEP 1316 H=1316:00 +CPU= 22.232
  14:16:10 STEP 1317 H=1317:00 +CPU= 32.609
  14:16:34 STEP 1318 H=1318:00 +CPU= 21.492
  14:16:57 STEP 1319 H=1319:00 +CPU= 21.362
  14:17:40 STEP 1320 H=1320:00 +CPU= 39.295
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_54.zip
Uploading the intermediate file: upload_file_54.zip
  14:18:01 STEP 1321 H=1321:00 +CPU= 20.133
Uploading trickle at timestep: 4752000
  14:18:24 STEP 1322 H=1322:00 +CPU= 21.754
  14:18:58 STEP 1323 H=1323:00 +CPU= 32.258
  14:19:22 STEP 1324 H=1324:00 +CPU= 22.572
  14:19:45 STEP 1325 H=1325:00 +CPU= 20.905
  14:20:21 STEP 1326 H=1326:00 +CPU= 33.853
  14:20:44 STEP 1327 H=1327:00 +CPU= 21.665
  14:21:07 STEP 1328 H=1328:00 +CPU= 21.128
  14:21:44 STEP 1329 H=1329:00 +CPU= 34.423
  14:22:07 STEP 1330 H=1330:00 +CPU= 21.597
  14:22:31 STEP 1331 H=1331:00 +CPU= 21.673
  14:23:09 STEP 1332 H=1332:00 +CPU= 36.621
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001332
  14:23:31 STEP 1333 H=1333:00 +CPU= 21.214
  14:23:54 STEP 1334 H=1334:00 +CPU= 22.067
  14:24:29 STEP 1335 H=1335:00 +CPU= 33.564
  14:24:52 STEP 1336 H=1336:00 +CPU= 22.065
  14:25:13 STEP 1337 H=1337:00 +CPU= 19.938
  14:25:49 STEP 1338 H=1338:00 +CPU= 34.622
  14:26:11 STEP 1339 H=1339:00 +CPU= 21.524
  14:26:34 STEP 1340 H=1340:00 +CPU= 21.490
  14:27:10 STEP 1341 H=1341:00 +CPU= 34.315
  14:27:31 STEP 1342 H=1342:00 +CPU= 20.995
  14:27:55 STEP 1343 H=1343:00 +CPU= 22.160
  14:28:35 STEP 1344 H=1344:00 +CPU= 38.609
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_55.zip
Uploading the intermediate file: upload_file_55.zip
  14:28:57 STEP 1345 H=1345:00 +CPU= 20.601
  14:29:19 STEP 1346 H=1346:00 +CPU= 21.261
  14:29:53 STEP 1347 H=1347:00 +CPU= 32.374
  14:30:15 STEP 1348 H=1348:00 +CPU= 21.684
  14:30:37 STEP 1349 H=1349:00 +CPU= 20.671
  14:31:13 STEP 1350 H=1350:00 +CPU= 34.519
  14:31:34 STEP 1351 H=1351:00 +CPU= 20.199
  14:31:57 STEP 1352 H=1352:00 +CPU= 22.519
  14:32:31 STEP 1353 H=1353:00 +CPU= 32.571
  14:32:55 STEP 1354 H=1354:00 +CPU= 22.589
  14:33:17 STEP 1355 H=1355:00 +CPU= 21.065
  14:33:56 STEP 1356 H=1356:00 +CPU= 37.810
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001356
  14:34:18 STEP 1357 H=1357:00 +CPU= 21.365
  14:34:40 STEP 1358 H=1358:00 +CPU= 20.717
  14:35:17 STEP 1359 H=1359:00 +CPU= 34.123
  14:35:38 STEP 1360 H=1360:00 +CPU= 21.233
  14:35:59 STEP 1361 H=1361:00 +CPU= 20.313
  14:36:37 STEP 1362 H=1362:00 +CPU= 35.155
  14:36:59 STEP 1363 H=1363:00 +CPU= 19.866
  14:37:24 STEP 1364 H=1364:00 +CPU= 23.206
  14:38:00 STEP 1365 H=1365:00 +CPU= 33.726
  14:38:22 STEP 1366 H=1366:00 +CPU= 21.295
  14:38:46 STEP 1367 H=1367:00 +CPU= 21.846
  14:39:28 STEP 1368 H=1368:00 +CPU= 38.067
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_56.zip
Uploading the intermediate file: upload_file_56.zip
  14:39:52 STEP 1369 H=1369:00 +CPU= 21.890
  14:40:15 STEP 1370 H=1370:00 +CPU= 21.246
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2005050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0860
(argv4) batchid: 974
(argv5) wuid: 12191504
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/10
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fb423cc1235f6a26832feb361525e6db to: /var/lib/boinc/slots/10/oifs_43r3_ps_0860_2005050100_123_974_12191504.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/10/oifs_43r3_ps_0860_2005050100_123_974_12191504.zip
ic_ancil_file: ic_ancil_12191504
ifsdata_file: ifsdata_12191504
climate_data_file: clim_data_12191504
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3f89e0f5e28a3478ecf91e0b3b4b4ad7 to: /var/lib/boinc/slots/10/ic_ancil_12191504.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/10/ic_ancil_12191504.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_253bf9d244527d94366e5ad2b93f72fd to: /var/lib/boinc/slots/10/ifsdata/ifsdata_12191504.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/10/ifsdata/ifsdata_12191504.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/10/159l_2/clim_data_12191504.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/10/159l_2/clim_data_12191504.zip
Checking for progress XML file: /var/lib/boinc/slots/10/progress_file_12191504.xml
Opened progress file ok : /var/lib/boinc/slots/10/progress_file_12191504.xml
-- Model is restarting --
Adjusting last_iter, 1371, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/10/progress_file_12191504.xml
last_cpu_time: 39735
upload_file_number: 57
last_iter: 1367
last_upload: 4924800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921
The child process has been launched with process id: 6996
Executing the command: /var/lib/boinc/slots/10/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6996:6996] [20230115:154900:1673797740:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:49:30 STEP 1368 H=1368:00 +CPU= 27.362
  15:49:53 STEP 1369 H=1369:00 +CPU= 21.061
  15:50:17 STEP 1370 H=1370:00 +CPU= 21.490
  15:50:50 STEP 1371 H=1371:00 +CPU= 31.497
  15:51:14 STEP 1372 H=1372:00 +CPU= 21.836
  15:51:36 STEP 1373 H=1373:00 +CPU= 20.663
  15:52:12 STEP 1374 H=1374:00 +CPU= 33.269
  15:52:34 STEP 1375 H=1375:00 +CPU= 21.035
  15:52:56 STEP 1376 H=1376:00 +CPU= 19.933
  15:53:34 STEP 1377 H=1377:00 +CPU= 34.716
  15:53:56 STEP 1378 H=1378:00 +CPU= 20.803
  15:54:18 STEP 1379 H=1379:00 +CPU= 20.894
  15:54:56 STEP 1380 H=1380:00 +CPU= 36.035
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001380
  15:55:19 STEP 1381 H=1381:00 +CPU= 21.279
  15:55:42 STEP 1382 H=1382:00 +CPU= 21.801
  15:56:17 STEP 1383 H=1383:00 +CPU= 32.471
  15:56:40 STEP 1384 H=1384:00 +CPU= 21.777
  15:57:02 STEP 1385 H=1385:00 +CPU= 20.171
  15:57:37 STEP 1386 H=1386:00 +CPU= 32.695
  15:58:00 STEP 1387 H=1387:00 +CPU= 21.289
  15:58:22 STEP 1388 H=1388:00 +CPU= 20.467
  15:58:58 STEP 1389 H=1389:00 +CPU= 33.777
  15:59:20 STEP 1390 H=1390:00 +CPU= 20.257
  15:59:43 STEP 1391 H=1391:00 +CPU= 21.331
  16:00:24 STEP 1392 H=1392:00 +CPU= 38.478
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_57.zip
Uploading the intermediate file: upload_file_57.zip
  16:00:45 STEP 1393 H=1393:00 +CPU= 20.050
  16:01:09 STEP 1394 H=1394:00 +CPU= 22.157
  16:01:42 STEP 1395 H=1395:00 +CPU= 31.760
  16:02:05 STEP 1396 H=1396:00 +CPU= 21.402
  16:02:28 STEP 1397 H=1397:00 +CPU= 20.868
  16:03:03 STEP 1398 H=1398:00 +CPU= 32.732
  16:03:25 STEP 1399 H=1399:00 +CPU= 20.620
  16:03:47 STEP 1400 H=1400:00 +CPU= 21.261
  16:04:24 STEP 1401 H=1401:00 +CPU= 34.124
  16:04:46 STEP 1402 H=1402:00 +CPU= 20.499
  16:05:08 STEP 1403 H=1403:00 +CPU= 20.584
  16:05:49 STEP 1404 H=1404:00 +CPU= 37.413
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001404
  16:06:10 STEP 1405 H=1405:00 +CPU= 20.394
  16:06:34 STEP 1406 H=1406:00 +CPU= 22.201
  16:07:09 STEP 1407 H=1407:00 +CPU= 32.569
  16:07:32 STEP 1408 H=1408:00 +CPU= 22.110
  16:07:54 STEP 1409 H=1409:00 +CPU= 20.493
  16:08:29 STEP 1410 H=1410:00 +CPU= 33.322
  16:08:51 STEP 1411 H=1411:00 +CPU= 20.946
  16:09:14 STEP 1412 H=1412:00 +CPU= 20.757
  16:09:49 STEP 1413 H=1413:00 +CPU= 33.315
  16:10:13 STEP 1414 H=1414:00 +CPU= 21.999
  16:10:35 STEP 1415 H=1415:00 +CPU= 21.055
  16:11:17 STEP 1416 H=1416:00 +CPU= 39.431
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12191504/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0860_2005050100_123_974_12191504_0_r1034676921_58.zip
Uploading the intermediate file: upload_file_58.zip
  16:11:39 STEP 1417 H=1417:00 +CPU= 20.745
  16:12:02 STEP 1418 H=1418:00 +CPU= 21.627
  16:12:38 STEP 1419 H=1419:00 +CPU= 33.445
  16:13:02 STEP 1420 H=1420:00 +CPU= 22.310
  16:13:24 STEP 1421 H=1421:00 +CPU= 20.907
  16:14:01 STEP 1422 H=1422:00 +CPU= 34.447
  16:14:23 STEP 1423 H=1423:00 +CPU= 20.400
  16:14:46 STEP 1424 H=1424:00 +CPU= 21.172
  16:15:21 STEP 1425 H=1425:00 +CPU= 33.604
  16:15:44 STEP 1426 H=1426:00 +CPU= 21.658
  16:16:06 STEP 1427 H=1427:00 +CPU= 20.854
  16:16:47 STEP 1428 H=1428:00 +CPU= 37.811
Moving to projects directory: /var/lib/boinc/slots/10/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/10/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc/slots/10/ICMUAhq0f+001428
  16:17:09 STEP 1429 H=1429:00 +CPU= 20.682
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:20:50 1531398 22277189 oifs_43r3_ps_0860_2005050100_123_974_12191504_0 4,752,000 38,451 0.0081
09 May 2023 16:20:33 1531398 22277189 oifs_43r3_ps_0860_2005050100_123_974_12191504_0 3,888,000 31,768 0.0082
09 May 2023 16:20:27 1531398 22277189 oifs_43r3_ps_0860_2005050100_123_974_12191504_0 3,024,000 25,579 0.0085
09 May 2023 16:20:20 1531398 22277189 oifs_43r3_ps_0860_2005050100_123_974_12191504_0 2,160,000 19,359 0.0090
09 May 2023 16:20:12 1531398 22277189 oifs_43r3_ps_0860_2005050100_123_974_12191504_0 1,296,000 12,194 0.0094


©2024 climateprediction.net