climateprediction.net home page
Task 22273937

Task 22273937

Name oifs_43r3_ps_0804_2002050100_123_971_12188448_0
Workunit 12188448
Created 23 Dec 2022, 3:41:45 UTC
Sent 27 Dec 2022, 12:55:01 UTC
Report deadline 26 Jan 2023, 12:55:01 UTC
Received 28 Dec 2022, 18:08:56 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1524359
Run time 19 hours 13 min 16 sec
CPU time 10 hours 58 min 38 sec
Validate state Invalid
Credit 6,168.96
Device peak FLOPS 3.52 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,392.91 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,690.36 MB
Stderr 
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_43.zip
Uploading the intermediate file: upload_file_43.zip
  01:52:16 STEP 1057 H=1057:00 +CPU= 20.805
  01:52:44 STEP 1058 H=1058:00 +CPU= 21.130
  01:53:21 STEP 1059 H=1059:00 +CPU= 32.616
  01:53:42 STEP 1060 H=1060:00 +CPU= 21.158
  01:54:03 STEP 1061 H=1061:00 +CPU= 20.500
  01:54:51 STEP 1062 H=1062:00 +CPU= 33.101
  01:55:21 STEP 1063 H=1063:00 +CPU= 20.442
  01:55:48 STEP 1064 H=1064:00 +CPU= 20.939
  01:56:32 STEP 1065 H=1065:00 +CPU= 32.428
  01:56:56 STEP 1066 H=1066:00 +CPU= 20.819
  01:57:21 STEP 1067 H=1067:00 +CPU= 20.340
  01:58:28 STEP 1068 H=1068:00 +CPU= 35.786
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001068
  01:58:48 STEP 1069 H=1069:00 +CPU= 20.215
  01:59:09 STEP 1070 H=1070:00 +CPU= 20.746
  01:59:41 STEP 1071 H=1071:00 +CPU= 31.941
  02:00:04 STEP 1072 H=1072:00 +CPU= 20.661
  02:00:27 STEP 1073 H=1073:00 +CPU= 20.266
  02:01:15 STEP 1074 H=1074:00 +CPU= 32.617
  02:01:40 STEP 1075 H=1075:00 +CPU= 20.130
  02:02:02 STEP 1076 H=1076:00 +CPU= 20.737
  02:02:34 STEP 1077 H=1077:00 +CPU= 32.001
  02:02:56 STEP 1078 H=1078:00 +CPU= 20.672
  02:03:18 STEP 1079 H=1079:00 +CPU= 20.723
  02:04:12 STEP 1080 H=1080:00 +CPU= 37.880
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_44.zip
Uploading the intermediate file: upload_file_44.zip
  02:04:32 STEP 1081 H=1081:00 +CPU= 20.114
  02:04:54 STEP 1082 H=1082:00 +CPU= 20.641
  02:05:26 STEP 1083 H=1083:00 +CPU= 32.055
  02:05:56 STEP 1084 H=1084:00 +CPU= 20.515
  02:06:49 STEP 1085 H=1085:00 +CPU= 20.389
  02:12:35 STEP 1086 H=1086:00 +CPU= 36.793
  02:15:23 STEP 1087 H=1087:00 +CPU= 20.987
  02:15:47 STEP 1088 H=1088:00 +CPU= 21.155
  02:16:24 STEP 1089 H=1089:00 +CPU= 32.434
  02:16:54 STEP 1090 H=1090:00 +CPU= 20.918
  02:17:31 STEP 1091 H=1091:00 +CPU= 20.263
  02:18:46 STEP 1092 H=1092:00 +CPU= 35.882
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001092
  02:19:17 STEP 1093 H=1093:00 +CPU= 20.502
  02:19:43 STEP 1094 H=1094:00 +CPU= 20.971
  02:20:33 STEP 1095 H=1095:00 +CPU= 32.427
  02:20:54 STEP 1096 H=1096:00 +CPU= 20.884
  02:21:26 STEP 1097 H=1097:00 +CPU= 20.376
  02:22:13 STEP 1098 H=1098:00 +CPU= 34.010
  02:22:33 STEP 1099 H=1099:00 +CPU= 20.424
  02:22:55 STEP 1100 H=1100:00 +CPU= 20.952
  02:23:38 STEP 1101 H=1101:00 +CPU= 32.346
  02:23:59 STEP 1102 H=1102:00 +CPU= 20.879
  02:24:37 STEP 1103 H=1103:00 +CPU= 21.064
  02:26:33 STEP 1104 H=1104:00 +CPU= 37.304
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_45.zip
Uploading the intermediate file: upload_file_45.zip
  02:26:54 STEP 1105 H=1105:00 +CPU= 20.620
  02:27:15 STEP 1106 H=1106:00 +CPU= 20.853
  02:27:48 STEP 1107 H=1107:00 +CPU= 32.454
  02:28:09 STEP 1108 H=1108:00 +CPU= 20.829
  02:28:30 STEP 1109 H=1109:00 +CPU= 20.293
  02:29:03 STEP 1110 H=1110:00 +CPU= 32.669
  02:29:24 STEP 1111 H=1111:00 +CPU= 20.165
  02:29:45 STEP 1112 H=1112:00 +CPU= 20.837
  02:30:18 STEP 1113 H=1113:00 +CPU= 32.364
  02:30:39 STEP 1114 H=1114:00 +CPU= 20.773
  02:31:00 STEP 1115 H=1115:00 +CPU= 20.202
  02:31:35 STEP 1116 H=1116:00 +CPU= 35.446
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001116
  02:31:56 STEP 1117 H=1117:00 +CPU= 20.267
  02:32:16 STEP 1118 H=1118:00 +CPU= 20.892
  02:32:49 STEP 1119 H=1119:00 +CPU= 32.212
  02:33:27 STEP 1120 H=1120:00 +CPU= 21.326
  02:33:48 STEP 1121 H=1121:00 +CPU= 20.400
  02:34:37 STEP 1122 H=1122:00 +CPU= 32.930
  02:34:58 STEP 1123 H=1123:00 +CPU= 20.670
  02:35:57 STEP 1124 H=1124:00 +CPU= 20.983
  02:36:43 STEP 1125 H=1125:00 +CPU= 31.923
  02:37:07 STEP 1126 H=1126:00 +CPU= 20.794
  02:37:35 STEP 1127 H=1127:00 +CPU= 20.832
  02:39:01 STEP 1128 H=1128:00 +CPU= 37.083
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_46.zip
Uploading the intermediate file: upload_file_46.zip
  02:39:28 STEP 1129 H=1129:00 +CPU= 20.593
  02:39:58 STEP 1130 H=1130:00 +CPU= 20.761
  02:40:53 STEP 1131 H=1131:00 +CPU= 31.970
  02:41:25 STEP 1132 H=1132:00 +CPU= 20.820
  02:41:48 STEP 1133 H=1133:00 +CPU= 20.297
  02:42:50 STEP 1134 H=1134:00 +CPU= 33.274
  02:43:11 STEP 1135 H=1135:00 +CPU= 20.291
  02:43:36 STEP 1136 H=1136:00 +CPU= 20.991
  02:44:28 STEP 1137 H=1137:00 +CPU= 32.207
  02:44:50 STEP 1138 H=1138:00 +CPU= 20.977
  02:45:50 STEP 1139 H=1139:00 +CPU= 20.365
  02:47:02 STEP 1140 H=1140:00 +CPU= 35.740
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001140
  02:47:33 STEP 1141 H=1141:00 +CPU= 20.319
  02:48:01 STEP 1142 H=1142:00 +CPU= 21.095
  02:48:45 STEP 1143 H=1143:00 +CPU= 31.996
  02:49:14 STEP 1144 H=1144:00 +CPU= 20.899
  02:49:39 STEP 1145 H=1145:00 +CPU= 20.427
  02:51:24 STEP 1146 H=1146:00 +CPU= 32.755
  02:52:11 STEP 1147 H=1147:00 +CPU= 20.280
  02:53:26 STEP 1148 H=1148:00 +CPU= 21.061
  02:55:28 STEP 1149 H=1149:00 +CPU= 32.289
  02:55:56 STEP 1150 H=1150:00 +CPU= 22.125
  02:56:22 STEP 1151 H=1151:00 +CPU= 23.495
  02:57:55 STEP 1152 H=1152:00 +CPU= 37.338
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_47.zip
Uploading the intermediate file: upload_file_47.zip
  02:58:16 STEP 1153 H=1153:00 +CPU= 20.852
  02:58:37 STEP 1154 H=1154:00 +CPU= 20.923
  02:59:58 STEP 1155 H=1155:00 +CPU= 32.076
  03:01:21 STEP 1156 H=1156:00 +CPU= 21.029
  03:02:23 STEP 1157 H=1157:00 +CPU= 20.482
  03:03:01 STEP 1158 H=1158:00 +CPU= 33.606
  03:03:21 STEP 1159 H=1159:00 +CPU= 20.333
  03:03:42 STEP 1160 H=1160:00 +CPU= 20.981
  03:04:15 STEP 1161 H=1161:00 +CPU= 32.246
  03:04:36 STEP 1162 H=1162:00 +CPU= 20.882
  03:05:00 STEP 1163 H=1163:00 +CPU= 20.462
  03:05:42 STEP 1164 H=1164:00 +CPU= 35.562
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001164
  03:06:03 STEP 1165 H=1165:00 +CPU= 20.346
  03:06:31 STEP 1166 H=1166:00 +CPU= 20.891
  03:07:13 STEP 1167 H=1167:00 +CPU= 32.149
  03:07:34 STEP 1168 H=1168:00 +CPU= 20.996
  03:08:02 STEP 1169 H=1169:00 +CPU= 20.483
  03:08:45 STEP 1170 H=1170:00 +CPU= 32.959
  03:09:15 STEP 1171 H=1171:00 +CPU= 20.404
  03:09:42 STEP 1172 H=1172:00 +CPU= 21.017
  03:10:26 STEP 1173 H=1173:00 +CPU= 31.950
  03:10:48 STEP 1174 H=1174:00 +CPU= 20.942
  03:11:18 STEP 1175 H=1175:00 +CPU= 21.065
  03:14:49 STEP 1176 H=1176:00 +CPU= 37.836
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_48.zip
Uploading the intermediate file: upload_file_48.zip
  03:16:37 STEP 1177 H=1177:00 +CPU= 21.877
  03:19:40 STEP 1178 H=1178:00 +CPU= 21.174
  03:20:24 STEP 1179 H=1179:00 +CPU= 32.248
  03:20:56 STEP 1180 H=1180:00 +CPU= 21.081
  03:21:18 STEP 1181 H=1181:00 +CPU= 20.266
  03:22:07 STEP 1182 H=1182:00 +CPU= 32.752
  03:22:34 STEP 1183 H=1183:00 +CPU= 20.322
  03:23:02 STEP 1184 H=1184:00 +CPU= 20.976
  03:23:54 STEP 1185 H=1185:00 +CPU= 32.226
  03:24:23 STEP 1186 H=1186:00 +CPU= 21.003
  03:24:45 STEP 1187 H=1187:00 +CPU= 20.376
  03:25:51 STEP 1188 H=1188:00 +CPU= 35.915
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001188
  03:26:20 STEP 1189 H=1189:00 +CPU= 20.309
  03:26:42 STEP 1190 H=1190:00 +CPU= 21.523
  03:27:54 STEP 1191 H=1191:00 +CPU= 34.456
  03:28:19 STEP 1192 H=1192:00 +CPU= 21.195
  03:28:40 STEP 1193 H=1193:00 +CPU= 20.495
  03:29:14 STEP 1194 H=1194:00 +CPU= 33.076
  03:29:34 STEP 1195 H=1195:00 +CPU= 20.450
  03:30:22 STEP 1196 H=1196:00 +CPU= 21.250
  03:31:40 STEP 1197 H=1197:00 +CPU= 32.361
  03:32:39 STEP 1198 H=1198:00 +CPU= 20.915
  03:33:35 STEP 1199 H=1199:00 +CPU= 21.022
  03:38:45 STEP 1200 H=1200:00 +CPU= 38.374
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_49.zip
Uploading the intermediate file: upload_file_49.zip
  03:40:47 STEP 1201 H=1201:00 +CPU= 21.210
  03:41:28 STEP 1202 H=1202:00 +CPU= 21.066
  03:42:05 STEP 1203 H=1203:00 +CPU= 32.432
  03:42:34 STEP 1204 H=1204:00 +CPU= 20.830
  03:43:04 STEP 1205 H=1205:00 +CPU= 20.501
  03:43:51 STEP 1206 H=1206:00 +CPU= 32.933
  03:44:15 STEP 1207 H=1207:00 +CPU= 20.431
  03:44:45 STEP 1208 H=1208:00 +CPU= 21.132
  03:45:26 STEP 1209 H=1209:00 +CPU= 32.518
  03:45:47 STEP 1210 H=1210:00 +CPU= 20.994
  03:46:18 STEP 1211 H=1211:00 +CPU= 20.433
  03:47:24 STEP 1212 H=1212:00 +CPU= 35.739
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001212
  03:47:44 STEP 1213 H=1213:00 +CPU= 20.337
  03:48:06 STEP 1214 H=1214:00 +CPU= 21.003
  03:48:38 STEP 1215 H=1215:00 +CPU= 32.117
  03:48:59 STEP 1216 H=1216:00 +CPU= 21.048
  03:49:20 STEP 1217 H=1217:00 +CPU= 20.539
  03:49:53 STEP 1218 H=1218:00 +CPU= 32.953
  03:50:13 STEP 1219 H=1219:00 +CPU= 20.497
  03:50:35 STEP 1220 H=1220:00 +CPU= 21.238
  03:51:07 STEP 1221 H=1221:00 +CPU= 32.435
  03:51:28 STEP 1222 H=1222:00 +CPU= 21.122
  03:51:52 STEP 1223 H=1223:00 +CPU= 21.424
  03:53:02 STEP 1224 H=1224:00 +CPU= 37.843
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_50.zip
Uploading the intermediate file: upload_file_50.zip
  03:53:26 STEP 1225 H=1225:00 +CPU= 20.969
Uploading trickle at timestep: 4406400
  03:54:14 STEP 1226 H=1226:00 +CPU= 21.374
  03:55:09 STEP 1227 H=1227:00 +CPU= 32.619
  03:55:31 STEP 1228 H=1228:00 +CPU= 21.151
  03:55:52 STEP 1229 H=1229:00 +CPU= 20.764
  03:56:25 STEP 1230 H=1230:00 +CPU= 33.512
  03:56:47 STEP 1231 H=1231:00 +CPU= 20.498
  03:57:11 STEP 1232 H=1232:00 +CPU= 20.930
  03:57:44 STEP 1233 H=1233:00 +CPU= 32.305
  03:58:05 STEP 1234 H=1234:00 +CPU= 20.930
  03:58:25 STEP 1235 H=1235:00 +CPU= 20.410
  03:59:04 STEP 1236 H=1236:00 +CPU= 35.511
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001236
  03:59:24 STEP 1237 H=1237:00 +CPU= 20.570
  03:59:45 STEP 1238 H=1238:00 +CPU= 20.972
  04:00:17 STEP 1239 H=1239:00 +CPU= 32.149
  04:00:39 STEP 1240 H=1240:00 +CPU= 21.074
  04:00:59 STEP 1241 H=1241:00 +CPU= 20.318
  04:01:32 STEP 1242 H=1242:00 +CPU= 32.766
  04:01:52 STEP 1243 H=1243:00 +CPU= 20.412
  04:02:13 STEP 1244 H=1244:00 +CPU= 20.996
  04:02:45 STEP 1245 H=1245:00 +CPU= 32.109
  04:03:06 STEP 1246 H=1246:00 +CPU= 20.834
  04:03:29 STEP 1247 H=1247:00 +CPU= 21.004
  04:04:19 STEP 1248 H=1248:00 +CPU= 37.339
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_51.zip
Uploading the intermediate file: upload_file_51.zip
  04:04:39 STEP 1249 H=1249:00 +CPU= 20.463
  04:05:01 STEP 1250 H=1250:00 +CPU= 21.642
  04:05:34 STEP 1251 H=1251:00 +CPU= 33.325
  04:05:56 STEP 1252 H=1252:00 +CPU= 20.856
  04:06:16 STEP 1253 H=1253:00 +CPU= 20.323
  04:07:31 STEP 1254 H=1254:00 +CPU= 32.931
  04:07:54 STEP 1255 H=1255:00 +CPU= 20.272
  04:08:17 STEP 1256 H=1256:00 +CPU= 20.986
  04:09:10 STEP 1257 H=1257:00 +CPU= 32.343
  04:09:38 STEP 1258 H=1258:00 +CPU= 21.083
  04:10:02 STEP 1259 H=1259:00 +CPU= 20.268
  04:11:38 STEP 1260 H=1260:00 +CPU= 36.825
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001260
  04:12:31 STEP 1261 H=1261:00 +CPU= 20.627
  04:13:35 STEP 1262 H=1262:00 +CPU= 21.143
  04:15:02 STEP 1263 H=1263:00 +CPU= 32.569
  04:15:29 STEP 1264 H=1264:00 +CPU= 22.905
  04:16:01 STEP 1265 H=1265:00 +CPU= 31.502
  04:16:58 STEP 1266 H=1266:00 +CPU= 55.884
  04:17:29 STEP 1267 H=1267:00 +CPU= 31.105
  04:17:55 STEP 1268 H=1268:00 +CPU= 25.538
  04:18:44 STEP 1269 H=1269:00 +CPU= 49.326
  04:19:07 STEP 1270 H=1270:00 +CPU= 23.189
  04:19:44 STEP 1271 H=1271:00 +CPU= 35.815
  04:22:12 STEP 1272 H=1272:00 +CPU= 55.365
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_52.zip
Uploading the intermediate file: upload_file_52.zip
  04:22:48 STEP 1273 H=1273:00 +CPU= 35.547
  04:23:26 STEP 1274 H=1274:00 +CPU= 37.215
  04:24:20 STEP 1275 H=1275:00 +CPU= 53.867
  04:25:51 STEP 1276 H=1276:00 +CPU= 37.588
  04:26:32 STEP 1277 H=1277:00 +CPU= 36.103
  04:27:34 STEP 1278 H=1278:00 +CPU= 58.031
  04:28:08 STEP 1279 H=1279:00 +CPU= 34.100
  04:28:42 STEP 1280 H=1280:00 +CPU= 34.093
  04:29:38 STEP 1281 H=1281:00 +CPU= 53.408
  04:31:16 STEP 1282 H=1282:00 +CPU= 31.463
  04:31:51 STEP 1283 H=1283:00 +CPU= 33.594
  04:32:57 STEP 1284 H=1284:00 +CPU= 62.434
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001284
  04:33:33 STEP 1285 H=1285:00 +CPU= 35.799
  04:34:07 STEP 1286 H=1286:00 +CPU= 33.243
  04:34:58 STEP 1287 H=1287:00 +CPU= 50.427
  04:35:31 STEP 1288 H=1288:00 +CPU= 32.891
  04:36:04 STEP 1289 H=1289:00 +CPU= 32.335
  04:36:57 STEP 1290 H=1290:00 +CPU= 51.633
  04:37:39 STEP 1291 H=1291:00 +CPU= 31.800
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0804
(argv4) batchid: 971
(argv5) wuid: 12188448
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fa7a2b13d835f69697fe1951f78c29f5 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0804_2002050100_123_971_12188448.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0804_2002050100_123_971_12188448.zip
ic_ancil_file: ic_ancil_12188448
ifsdata_file: ifsdata_12188448
climate_data_file: clim_data_12188448
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e72a7c1c5dc7c60f28e3556f2b3dfbaf to: /var/lib/boinc-client/slots/3/ic_ancil_12188448.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12188448.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1067f813829bc0f1fd66f3ea6a8fbfaa to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12188448.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12188448.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12188448.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12188448.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12188448.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12188448.xml
-- Model is restarting --
Adjusting last_iter, 1292, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12188448.xml
last_cpu_time: 35657
upload_file_number: 53
last_iter: 1271
last_upload: 4579200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677
The child process has been launched with process id: 31334
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:31334:31334] [20221228:051547:1672233347:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:23:25 STEP 1272 H=1272:00 +CPU= 51.812
  05:24:00 STEP 1273 H=1273:00 +CPU= 34.036
  05:24:25 STEP 1274 H=1274:00 +CPU= 24.609
  05:25:10 STEP 1275 H=1275:00 +CPU= 45.594
  05:25:35 STEP 1276 H=1276:00 +CPU= 24.960
  05:25:58 STEP 1277 H=1277:00 +CPU= 22.394
  05:26:31 STEP 1278 H=1278:00 +CPU= 32.852
  05:27:01 STEP 1279 H=1279:00 +CPU= 30.380
  05:27:25 STEP 1280 H=1280:00 +CPU= 24.374
  05:28:14 STEP 1281 H=1281:00 +CPU= 48.217
  05:28:42 STEP 1282 H=1282:00 +CPU= 27.940
  05:29:08 STEP 1283 H=1283:00 +CPU= 26.847
  05:29:53 STEP 1284 H=1284:00 +CPU= 43.994
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001284
  05:30:13 STEP 1285 H=1285:00 +CPU= 20.401
  05:30:36 STEP 1286 H=1286:00 +CPU= 22.528
  05:31:30 STEP 1287 H=1287:00 +CPU= 53.976
  05:32:27 STEP 1288 H=1288:00 +CPU= 31.438
  05:33:20 STEP 1289 H=1289:00 +CPU= 35.524
  06:05:52 STEP 1290 H=1290:00 +CPU= 59.520
  06:10:35 STEP 1291 H=1291:00 +CPU= 38.302
  06:11:49 STEP 1292 H=1292:00 +CPU= 36.170
  06:13:07 STEP 1293 H=1293:00 +CPU= 50.263
  06:15:29 STEP 1294 H=1294:00 +CPU= 36.669
  06:16:07 STEP 1295 H=1295:00 +CPU= 36.368
  06:18:45 STEP 1296 H=1296:00 +CPU= 62.177
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_53.zip
Uploading the intermediate file: upload_file_53.zip
  06:19:40 STEP 1297 H=1297:00 +CPU= 35.007
  06:20:16 STEP 1298 H=1298:00 +CPU= 34.210
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0804
(argv4) batchid: 971
(argv5) wuid: 12188448
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fa7a2b13d835f69697fe1951f78c29f5 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0804_2002050100_123_971_12188448.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0804_2002050100_123_971_12188448.zip
ic_ancil_file: ic_ancil_12188448
ifsdata_file: ifsdata_12188448
climate_data_file: clim_data_12188448
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e72a7c1c5dc7c60f28e3556f2b3dfbaf to: /var/lib/boinc-client/slots/3/ic_ancil_12188448.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12188448.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1067f813829bc0f1fd66f3ea6a8fbfaa to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12188448.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12188448.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12188448.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12188448.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12188448.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12188448.xml
-- Model is restarting --
Adjusting last_iter, 1299, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12188448.xml
last_cpu_time: 36675
upload_file_number: 54
last_iter: 1295
last_upload: 4665600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677
The child process has been launched with process id: 4649
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:4649:4649] [20221228:070138:1672239698:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:07:56 STEP 1296 H=1296:00 +CPU= 37.959
  07:09:40 STEP 1297 H=1297:00 +CPU= 20.608
  07:10:11 STEP 1298 H=1298:00 +CPU= 21.394
  07:11:19 STEP 1299 H=1299:00 +CPU= 32.728
  07:12:27 STEP 1300 H=1300:00 +CPU= 21.149
  07:13:00 STEP 1301 H=1301:00 +CPU= 20.661
  07:13:50 STEP 1302 H=1302:00 +CPU= 33.142
  07:14:15 STEP 1303 H=1303:00 +CPU= 20.361
  07:14:41 STEP 1304 H=1304:00 +CPU= 21.078
  07:15:27 STEP 1305 H=1305:00 +CPU= 32.423
  07:16:12 STEP 1306 H=1306:00 +CPU= 21.362
  07:16:39 STEP 1307 H=1307:00 +CPU= 20.342
  07:17:37 STEP 1308 H=1308:00 +CPU= 35.158
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001308
  07:17:57 STEP 1309 H=1309:00 +CPU= 20.233
  07:18:18 STEP 1310 H=1310:00 +CPU= 20.937
  07:18:50 STEP 1311 H=1311:00 +CPU= 32.196
  07:19:11 STEP 1312 H=1312:00 +CPU= 20.591
  07:19:31 STEP 1313 H=1313:00 +CPU= 20.020
  07:20:03 STEP 1314 H=1314:00 +CPU= 32.189
  07:20:23 STEP 1315 H=1315:00 +CPU= 19.979
  07:20:44 STEP 1316 H=1316:00 +CPU= 20.555
  07:21:15 STEP 1317 H=1317:00 +CPU= 31.640
  07:21:36 STEP 1318 H=1318:00 +CPU= 20.665
  07:21:58 STEP 1319 H=1319:00 +CPU= 20.639
  07:23:58 STEP 1320 H=1320:00 +CPU= 37.215
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_54.zip
Uploading the intermediate file: upload_file_54.zip
  07:24:21 STEP 1321 H=1321:00 +CPU= 20.289
  07:24:43 STEP 1322 H=1322:00 +CPU= 20.600
  07:25:16 STEP 1323 H=1323:00 +CPU= 31.668
  07:25:37 STEP 1324 H=1324:00 +CPU= 20.590
  07:25:58 STEP 1325 H=1325:00 +CPU= 19.995
  07:26:33 STEP 1326 H=1326:00 +CPU= 32.384
  07:26:56 STEP 1327 H=1327:00 +CPU= 20.242
  07:27:17 STEP 1328 H=1328:00 +CPU= 20.618
  07:27:55 STEP 1329 H=1329:00 +CPU= 31.735
  07:28:16 STEP 1330 H=1330:00 +CPU= 20.653
  07:28:40 STEP 1331 H=1331:00 +CPU= 20.112
  07:29:21 STEP 1332 H=1332:00 +CPU= 35.021
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001332
  07:29:41 STEP 1333 H=1333:00 +CPU= 20.020
  07:30:08 STEP 1334 H=1334:00 +CPU= 20.650
  07:30:55 STEP 1335 H=1335:00 +CPU= 33.886
  07:31:17 STEP 1336 H=1336:00 +CPU= 20.608
  07:32:33 STEP 1337 H=1337:00 +CPU= 20.397
  07:34:01 STEP 1338 H=1338:00 +CPU= 32.784
  07:34:51 STEP 1339 H=1339:00 +CPU= 20.408
  07:35:44 STEP 1340 H=1340:00 +CPU= 20.867
  07:37:12 STEP 1341 H=1341:00 +CPU= 32.145
  07:38:45 STEP 1342 H=1342:00 +CPU= 21.242
  07:40:19 STEP 1343 H=1343:00 +CPU= 21.377
  07:43:19 STEP 1344 H=1344:00 +CPU= 37.393
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_55.zip
Uploading the intermediate file: upload_file_55.zip
  07:43:51 STEP 1345 H=1345:00 +CPU= 20.415
  07:44:17 STEP 1346 H=1346:00 +CPU= 20.806
  07:45:05 STEP 1347 H=1347:00 +CPU= 31.874
  07:45:53 STEP 1348 H=1348:00 +CPU= 20.931
  07:46:24 STEP 1349 H=1349:00 +CPU= 20.032
  07:47:12 STEP 1350 H=1350:00 +CPU= 32.457
  07:47:36 STEP 1351 H=1351:00 +CPU= 20.196
  07:47:57 STEP 1352 H=1352:00 +CPU= 20.772
  07:48:29 STEP 1353 H=1353:00 +CPU= 31.807
  07:48:51 STEP 1354 H=1354:00 +CPU= 20.770
  07:49:11 STEP 1355 H=1355:00 +CPU= 20.358
  07:49:51 STEP 1356 H=1356:00 +CPU= 35.753
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001356
  07:50:12 STEP 1357 H=1357:00 +CPU= 20.398
  07:50:33 STEP 1358 H=1358:00 +CPU= 21.012
  07:51:24 STEP 1359 H=1359:00 +CPU= 32.046
  07:51:59 STEP 1360 H=1360:00 +CPU= 20.578
  07:52:37 STEP 1361 H=1361:00 +CPU= 20.366
  07:53:21 STEP 1362 H=1362:00 +CPU= 32.485
  07:53:46 STEP 1363 H=1363:00 +CPU= 20.290
  07:54:10 STEP 1364 H=1364:00 +CPU= 20.779
  07:55:01 STEP 1365 H=1365:00 +CPU= 31.908
  07:55:57 STEP 1366 H=1366:00 +CPU= 20.954
  07:57:00 STEP 1367 H=1367:00 +CPU= 21.043
  08:01:04 STEP 1368 H=1368:00 +CPU= 37.466
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_56.zip
Uploading the intermediate file: upload_file_56.zip
  08:02:58 STEP 1369 H=1369:00 +CPU= 21.128
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0804
(argv4) batchid: 971
(argv5) wuid: 12188448
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fa7a2b13d835f69697fe1951f78c29f5 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0804_2002050100_123_971_12188448.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0804_2002050100_123_971_12188448.zip
ic_ancil_file: ic_ancil_12188448
ifsdata_file: ifsdata_12188448
climate_data_file: clim_data_12188448
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e72a7c1c5dc7c60f28e3556f2b3dfbaf to: /var/lib/boinc-client/slots/3/ic_ancil_12188448.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12188448.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1067f813829bc0f1fd66f3ea6a8fbfaa to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12188448.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12188448.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12188448.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12188448.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12188448.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12188448.xml
-- Model is restarting --
Adjusting last_iter, 1370, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12188448.xml
last_cpu_time: 38544
upload_file_number: 57
last_iter: 1367
last_upload: 4924800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677
The child process has been launched with process id: 8639
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:8639:8639] [20221228:083039:1672245039:0.031] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:35:36 STEP 1368 H=1368:00 +CPU= 44.168
  08:36:44 STEP 1369 H=1369:00 +CPU= 28.059
  08:37:21 STEP 1370 H=1370:00 +CPU= 35.660
  08:38:03 STEP 1371 H=1371:00 +CPU= 40.861
  08:38:24 STEP 1372 H=1372:00 +CPU= 20.829
  08:38:44 STEP 1373 H=1373:00 +CPU= 20.323
  08:39:20 STEP 1374 H=1374:00 +CPU= 35.413
  08:39:40 STEP 1375 H=1375:00 +CPU= 20.275
  08:40:01 STEP 1376 H=1376:00 +CPU= 20.871
  08:40:55 STEP 1377 H=1377:00 +CPU= 53.706
  08:41:30 STEP 1378 H=1378:00 +CPU= 35.673
  08:42:04 STEP 1379 H=1379:00 +CPU= 33.674
  08:43:36 STEP 1380 H=1380:00 +CPU= 57.131
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001380
  08:44:00 STEP 1381 H=1381:00 +CPU= 23.462
  08:45:25 STEP 1382 H=1382:00 +CPU= 36.498
  08:48:49 STEP 1383 H=1383:00 +CPU= 52.394
  08:49:25 STEP 1384 H=1384:00 +CPU= 35.266
  08:58:43 STEP 1385 H=1385:00 +CPU= 39.363
  09:01:10 STEP 1386 H=1386:00 +CPU= 53.351
  09:01:57 STEP 1387 H=1387:00 +CPU= 32.699
  09:02:52 STEP 1388 H=1388:00 +CPU= 35.392
  09:03:53 STEP 1389 H=1389:00 +CPU= 55.625
  09:04:31 STEP 1390 H=1390:00 +CPU= 36.683
  09:05:09 STEP 1391 H=1391:00 +CPU= 36.657
  09:07:27 STEP 1392 H=1392:00 +CPU= 43.591
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188448/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0804_2002050100_123_971_12188448_0_r2122311677_57.zip
Uploading the intermediate file: upload_file_57.zip
  09:07:49 STEP 1393 H=1393:00 +CPU= 21.097
  09:08:11 STEP 1394 H=1394:00 +CPU= 21.338
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:00:03 1524359 22273937 oifs_43r3_ps_0804_2002050100_123_971_12188448_0 4,406,400 33,575 0.0076
09 May 2023 15:59:31 1524359 22273937 oifs_43r3_ps_0804_2002050100_123_971_12188448_0 2,592,000 18,593 0.0072
09 May 2023 15:59:21 1524359 22273937 oifs_43r3_ps_0804_2002050100_123_971_12188448_0 1,728,000 12,431 0.0072
09 May 2023 15:58:57 1524359 22273937 oifs_43r3_ps_0804_2002050100_123_971_12188448_0 864,000 6,199 0.0072


©2024 cpdn.org