climateprediction.net home page
Task 22437486

Task 22437486

Name oifs_43r3_bl_a34f_2016092300_20_1017_12285874_0
Workunit 12285874
Created 7 Jun 2024, 16:41:46 UTC
Sent 7 Jun 2024, 16:48:51 UTC
Report deadline 6 Aug 2024, 16:48:51 UTC
Received 8 Jun 2024, 22:37:41 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1469580
Run time 12 hours 48 min 14 sec
CPU time 12 hours 48 min 14 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 5.23 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,013.57 MB
Peak swap size 5,980.80 MB
Peak disk usage 2,278.19 MB
Stderr 
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:8267:8267] [20240608:202141:1717870901:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:8267:8267] [20240608:202141:1717870901:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:8267:8267] [20240608:202141:1717870901:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:8267:8267] [20240608:202141:1717870901:0.001] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:8267:8267] [20240608:202141:1717870901:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:8267:8267] [20240608:202141:1717870901:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:8267:8267] [20240608:202141:1717870901:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:8267:8267] [20240608:202141:1717870901:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:8267:8267] [20240608:202141:1717870901:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:8267:8267] [20240608:202141:1717870901:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:8267:8267] [20240608:202141:1717870901:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:8267:8267] [20240608:202141:1717870901:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:8267:8267] [20240608:202141:1717870901:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:8267:8267] [20240608:202141:1717870901:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:22:54 STEP 1104 H= 276:00 +CPU= 26.285
  20:23:19 STEP 1105 H= 276:15 +CPU= 24.236
  20:23:56 STEP 1106 H= 276:30 +CPU= 24.134
  20:24:25 STEP 1107 H= 276:45 +CPU= 26.029
  20:24:54 STEP 1108 H= 277:00 +CPU= 29.725
  20:25:22 STEP 1109 H= 277:15 +CPU= 27.725
  20:25:51 STEP 1110 H= 277:30 +CPU= 28.595
  20:26:17 STEP 1111 H= 277:45 +CPU= 26.363
  20:26:53 STEP 1112 H= 278:00 +CPU= 26.951
  20:27:23 STEP 1113 H= 278:15 +CPU= 23.486
  20:27:54 STEP 1114 H= 278:30 +CPU= 30.659
  20:28:20 STEP 1115 H= 278:45 +CPU= 26.123
  20:29:01 STEP 1116 H= 279:00 +CPU= 37.604
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001116
  20:29:30 STEP 1117 H= 279:15 +CPU= 28.994
  20:30:00 STEP 1118 H= 279:30 +CPU= 29.268
  20:30:30 STEP 1119 H= 279:45 +CPU= 30.501
  20:31:02 STEP 1120 H= 280:00 +CPU= 31.366
  20:31:32 STEP 1121 H= 280:15 +CPU= 29.689
  20:32:01 STEP 1122 H= 280:30 +CPU= 28.970
  20:32:31 STEP 1123 H= 280:45 +CPU= 29.719
  20:33:03 STEP 1124 H= 281:00 +CPU= 31.589
  20:33:34 STEP 1125 H= 281:15 +CPU= 31.330
  20:34:02 STEP 1126 H= 281:30 +CPU= 27.681
  20:34:32 STEP 1127 H= 281:45 +CPU= 30.205
  20:35:12 STEP 1128 H= 282:00 +CPU= 38.821
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001128
  20:35:43 STEP 1129 H= 282:15 +CPU= 30.955
  20:36:12 STEP 1130 H= 282:30 +CPU= 28.717
  20:36:43 STEP 1131 H= 282:45 +CPU= 30.442
  20:37:14 STEP 1132 H= 283:00 +CPU= 30.873
  20:37:43 STEP 1133 H= 283:15 +CPU= 28.917
  20:38:13 STEP 1134 H= 283:30 +CPU= 29.847
  20:38:43 STEP 1135 H= 283:45 +CPU= 29.142
  20:39:11 STEP 1136 H= 284:00 +CPU= 27.812
  20:39:36 STEP 1137 H= 284:15 +CPU= 25.490
  20:40:07 STEP 1138 H= 284:30 +CPU= 30.239
  20:40:39 STEP 1139 H= 284:45 +CPU= 31.555
  20:41:18 STEP 1140 H= 285:00 +CPU= 36.967
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001140
  20:41:49 STEP 1141 H= 285:15 +CPU= 30.833
  20:42:20 STEP 1142 H= 285:30 +CPU= 30.859
  20:42:50 STEP 1143 H= 285:45 +CPU= 29.139
  20:43:19 STEP 1144 H= 286:00 +CPU= 29.562
  20:43:50 STEP 1145 H= 286:15 +CPU= 28.573
  20:44:18 STEP 1146 H= 286:30 +CPU= 27.189
  20:44:47 STEP 1147 H= 286:45 +CPU= 29.095
  20:45:30 STEP 1148 H= 287:00 +CPU= 23.530
  20:45:54 STEP 1149 H= 287:15 +CPU= 23.284
  20:46:21 STEP 1150 H= 287:30 +CPU= 26.671
  20:46:47 STEP 1151 H= 287:45 +CPU= 26.704
  20:47:40 STEP 1152 H= 288:00 +CPU= 42.011
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a34f_2016092300_20_1017_12285874_0_r36598283_11.zip
  20:48:06 STEP 1153 H= 288:15 +CPU= 26.169
Uploading the intermediate file: upload_file_11.zip
  20:48:37 STEP 1154 H= 288:30 +CPU= 30.298
  20:49:04 STEP 1155 H= 288:45 +CPU= 27.163
  20:49:35 STEP 1156 H= 289:00 +CPU= 30.978
  20:50:04 STEP 1157 H= 289:15 +CPU= 25.853
  20:50:35 STEP 1158 H= 289:30 +CPU= 31.449
  20:51:05 STEP 1159 H= 289:45 +CPU= 29.730
  20:51:36 STEP 1160 H= 290:00 +CPU= 30.453
  20:52:07 STEP 1161 H= 290:15 +CPU= 31.207
  20:52:37 STEP 1162 H= 290:30 +CPU= 28.883
  20:53:06 STEP 1163 H= 290:45 +CPU= 29.777
  20:53:48 STEP 1164 H= 291:00 +CPU= 36.540
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001164
  20:54:18 STEP 1165 H= 291:15 +CPU= 29.791
  20:54:46 STEP 1166 H= 291:30 +CPU= 28.476
  20:55:21 STEP 1167 H= 291:45 +CPU= 30.285
  20:55:52 STEP 1168 H= 292:00 +CPU= 30.954
  20:56:21 STEP 1169 H= 292:15 +CPU= 28.965
  20:56:49 STEP 1170 H= 292:30 +CPU= 26.988
  20:57:31 STEP 1171 H= 292:45 +CPU= 28.148
  20:57:59 STEP 1172 H= 293:00 +CPU= 26.001
  20:58:28 STEP 1173 H= 293:15 +CPU= 28.854
  20:58:56 STEP 1174 H= 293:30 +CPU= 27.024
  20:59:29 STEP 1175 H= 293:45 +CPU= 29.002
  21:00:16 STEP 1176 H= 294:00 +CPU= 35.389
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001176
  21:00:45 STEP 1177 H= 294:15 +CPU= 29.163
  21:01:16 STEP 1178 H= 294:30 +CPU= 30.553
  21:01:47 STEP 1179 H= 294:45 +CPU= 31.070
  21:02:18 STEP 1180 H= 295:00 +CPU= 30.665
  21:02:47 STEP 1181 H= 295:15 +CPU= 28.329
  21:03:16 STEP 1182 H= 295:30 +CPU= 29.019
  21:03:45 STEP 1183 H= 295:45 +CPU= 29.440
  21:04:37 STEP 1184 H= 296:00 +CPU= 30.943
  21:05:07 STEP 1185 H= 296:15 +CPU= 25.643
  21:05:33 STEP 1186 H= 296:30 +CPU= 26.746
  21:06:01 STEP 1187 H= 296:45 +CPU= 26.973
  21:06:46 STEP 1188 H= 297:00 +CPU= 37.085
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001188
  21:07:32 STEP 1189 H= 297:15 +CPU= 31.889
  21:08:02 STEP 1190 H= 297:30 +CPU= 26.703
  21:08:26 STEP 1191 H= 297:45 +CPU= 24.170
  21:08:51 STEP 1192 H= 298:00 +CPU= 25.273
  21:09:19 STEP 1193 H= 298:15 +CPU= 26.783
  21:09:43 STEP 1194 H= 298:30 +CPU= 24.753
  21:10:09 STEP 1195 H= 298:45 +CPU= 25.288
  21:10:54 STEP 1196 H= 299:00 +CPU= 29.057
  21:11:22 STEP 1197 H= 299:15 +CPU= 21.967
  21:11:47 STEP 1198 H= 299:30 +CPU= 23.690
  21:12:13 STEP 1199 H= 299:45 +CPU= 25.933
  21:13:10 STEP 1200 H= 300:00 +CPU= 36.075
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001200
  21:13:38 STEP 1201 H= 300:15 +CPU= 26.623
  21:14:14 STEP 1202 H= 300:30 +CPU= 26.006
  21:14:46 STEP 1203 H= 300:45 +CPU= 27.765
  21:15:15 STEP 1204 H= 301:00 +CPU= 28.412
  21:15:44 STEP 1205 H= 301:15 +CPU= 29.569
  21:16:43 STEP 1206 H= 301:30 +CPU= 26.976
  21:17:12 STEP 1207 H= 301:45 +CPU= 27.367
  21:17:42 STEP 1208 H= 302:00 +CPU= 29.585
  21:18:11 STEP 1209 H= 302:15 +CPU= 29.822
  21:18:41 STEP 1210 H= 302:30 +CPU= 29.839
  21:19:10 STEP 1211 H= 302:45 +CPU= 28.736
  21:19:54 STEP 1212 H= 303:00 +CPU= 35.073
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001212
  21:20:24 STEP 1213 H= 303:15 +CPU= 30.685
  21:20:51 STEP 1214 H= 303:30 +CPU= 26.272
  21:21:22 STEP 1215 H= 303:45 +CPU= 31.438
  21:21:52 STEP 1216 H= 304:00 +CPU= 28.312
  21:22:20 STEP 1217 H= 304:15 +CPU= 27.353
  21:22:50 STEP 1218 H= 304:30 +CPU= 30.085
  21:23:20 STEP 1219 H= 304:45 +CPU= 29.958
  21:23:53 STEP 1220 H= 305:00 +CPU= 30.786
  21:24:22 STEP 1221 H= 305:15 +CPU= 29.780
  21:24:49 STEP 1222 H= 305:30 +CPU= 25.998
  21:25:17 STEP 1223 H= 305:45 +CPU= 28.353
  21:25:56 STEP 1224 H= 306:00 +CPU= 37.949
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001224
  21:26:26 STEP 1225 H= 306:15 +CPU= 29.985
  21:26:53 STEP 1226 H= 306:30 +CPU= 26.635
  21:27:22 STEP 1227 H= 306:45 +CPU= 28.641
  21:27:52 STEP 1228 H= 307:00 +CPU= 29.765
  21:28:20 STEP 1229 H= 307:15 +CPU= 27.446
  21:28:47 STEP 1230 H= 307:30 +CPU= 26.840
  21:29:16 STEP 1231 H= 307:45 +CPU= 29.337
  21:29:44 STEP 1232 H= 308:00 +CPU= 27.908
  21:30:15 STEP 1233 H= 308:15 +CPU= 30.301
  21:30:40 STEP 1234 H= 308:30 +CPU= 25.363
  21:31:08 STEP 1235 H= 308:45 +CPU= 27.041
  21:31:47 STEP 1236 H= 309:00 +CPU= 35.778
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001236
  21:32:18 STEP 1237 H= 309:15 +CPU= 30.765
  21:32:48 STEP 1238 H= 309:30 +CPU= 30.339
  21:33:23 STEP 1239 H= 309:45 +CPU= 29.387
  21:33:51 STEP 1240 H= 310:00 +CPU= 27.581
  21:34:14 STEP 1241 H= 310:15 +CPU= 23.574
  21:34:41 STEP 1242 H= 310:30 +CPU= 26.937
  21:35:08 STEP 1243 H= 310:45 +CPU= 25.801
  21:35:35 STEP 1244 H= 311:00 +CPU= 27.394
  21:36:01 STEP 1245 H= 311:15 +CPU= 23.376
  21:36:28 STEP 1246 H= 311:30 +CPU= 26.829
  21:36:58 STEP 1247 H= 311:45 +CPU= 27.605
  21:38:19 STEP 1248 H= 312:00 +CPU= 39.380
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a34f_2016092300_20_1017_12285874_0_r36598283_12.zip
  21:38:48 STEP 1249 H= 312:15 +CPU= 25.805
  21:39:18 STEP 1250 H= 312:30 +CPU= 29.915
Uploading the intermediate file: upload_file_12.zip
  21:39:48 STEP 1251 H= 312:45 +CPU= 29.655
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a34f
(argv4) batchid: 1017
(argv5) wuid: 12285874
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/17
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/17/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/17/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d26c9eddddbb78a0584a09be110c4068 to: /var/lib/boinc-client/slots/17/oifs_43r3_bl_a34f_2016092300_20_1017_12285874.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/17/oifs_43r3_bl_a34f_2016092300_20_1017_12285874.zip
ic_ancil_file: ic_ancil_12285874
ifsdata_file: ifsdata_12285874
climate_data_file: clim_data_12285874
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/17/ic_ancil_12285874.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/17/ic_ancil_12285874.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8f231f1887df997466659c61938fd32a to: /var/lib/boinc-client/slots/17/ifsdata/ifsdata_12285874.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/17/ifsdata/ifsdata_12285874.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/17/159l_2/clim_data_12285874.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/17/159l_2/clim_data_12285874.zip
Checking for progress XML file: /var/lib/boinc-client/slots/17/progress_file_12285874.xml
Opened progress file ok : /var/lib/boinc-client/slots/17/progress_file_12285874.xml
-- Model is restarting --
Adjusting last_iter, 1249, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/17/progress_file_12285874.xml
last_cpu_time: 40270
upload_file_number: 13
last_iter: 1249
last_upload: 1123200
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a34f_2016092300_20_1017_12285874_0_r36598283
The child process has been launched with process id: 10228
Executing the command: /var/lib/boinc-client/slots/17/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:10228:10228] [20240608:214146:1717875706:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:44:31 STEP 1248 H= 312:00 +CPU= 31.847
  21:45:05 STEP 1249 H= 312:15 +CPU= 28.205
  21:45:35 STEP 1250 H= 312:30 +CPU= 29.516
  21:46:02 STEP 1251 H= 312:45 +CPU= 27.401
  21:46:32 STEP 1252 H= 313:00 +CPU= 29.656
  21:47:01 STEP 1253 H= 313:15 +CPU= 28.678
  21:47:33 STEP 1254 H= 313:30 +CPU= 31.065
  21:48:03 STEP 1255 H= 313:45 +CPU= 30.019
  21:48:32 STEP 1256 H= 314:00 +CPU= 28.425
  21:49:00 STEP 1257 H= 314:15 +CPU= 28.429
  21:49:27 STEP 1258 H= 314:30 +CPU= 26.924
  21:49:53 STEP 1259 H= 314:45 +CPU= 25.339
  21:50:41 STEP 1260 H= 315:00 +CPU= 38.270
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001260
  21:51:10 STEP 1261 H= 315:15 +CPU= 28.481
  21:51:36 STEP 1262 H= 315:30 +CPU= 26.233
  21:52:05 STEP 1263 H= 315:45 +CPU= 28.649
  21:52:43 STEP 1264 H= 316:00 +CPU= 26.510
  21:53:10 STEP 1265 H= 316:15 +CPU= 25.865
  21:53:35 STEP 1266 H= 316:30 +CPU= 24.344
  21:54:00 STEP 1267 H= 316:45 +CPU= 25.048
  21:54:32 STEP 1268 H= 317:00 +CPU= 31.798
  21:55:01 STEP 1269 H= 317:15 +CPU= 28.696
  21:55:30 STEP 1270 H= 317:30 +CPU= 28.377
  21:55:58 STEP 1271 H= 317:45 +CPU= 27.523
  21:56:43 STEP 1272 H= 318:00 +CPU= 33.731
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001272
  21:57:10 STEP 1273 H= 318:15 +CPU= 27.579
  21:57:40 STEP 1274 H= 318:30 +CPU= 29.727
  21:58:10 STEP 1275 H= 318:45 +CPU= 28.731
  21:58:38 STEP 1276 H= 319:00 +CPU= 28.362
  21:59:06 STEP 1277 H= 319:15 +CPU= 27.280
  21:59:36 STEP 1278 H= 319:30 +CPU= 29.836
  22:00:23 STEP 1279 H= 319:45 +CPU= 26.778
  22:00:51 STEP 1280 H= 320:00 +CPU= 26.183
  22:01:19 STEP 1281 H= 320:15 +CPU= 26.412
  22:01:47 STEP 1282 H= 320:30 +CPU= 28.079
  22:02:18 STEP 1283 H= 320:45 +CPU= 29.965
  22:02:58 STEP 1284 H= 321:00 +CPU= 36.279
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001284
  22:03:25 STEP 1285 H= 321:15 +CPU= 26.145
  22:03:51 STEP 1286 H= 321:30 +CPU= 26.356
  22:04:22 STEP 1287 H= 321:45 +CPU= 30.552
  22:04:51 STEP 1288 H= 322:00 +CPU= 29.373
  22:05:17 STEP 1289 H= 322:15 +CPU= 25.361
  22:05:45 STEP 1290 H= 322:30 +CPU= 28.051
  22:06:15 STEP 1291 H= 322:45 +CPU= 29.548
  22:06:43 STEP 1292 H= 323:00 +CPU= 28.345
  22:07:11 STEP 1293 H= 323:15 +CPU= 27.296
  22:07:41 STEP 1294 H= 323:30 +CPU= 30.671
  22:08:09 STEP 1295 H= 323:45 +CPU= 27.512
  22:09:32 STEP 1296 H= 324:00 +CPU= 35.370
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001296
  22:09:58 STEP 1297 H= 324:15 +CPU= 25.933
  22:10:25 STEP 1298 H= 324:30 +CPU= 26.099
  22:10:53 STEP 1299 H= 324:45 +CPU= 28.510
  22:11:24 STEP 1300 H= 325:00 +CPU= 30.753
  22:11:56 STEP 1301 H= 325:15 +CPU= 30.948
  22:12:25 STEP 1302 H= 325:30 +CPU= 29.473
  22:12:54 STEP 1303 H= 325:45 +CPU= 27.997
  22:13:23 STEP 1304 H= 326:00 +CPU= 29.260
  22:13:57 STEP 1305 H= 326:15 +CPU= 29.038
  22:14:27 STEP 1306 H= 326:30 +CPU= 29.060
  22:14:57 STEP 1307 H= 326:45 +CPU= 27.778
  22:15:35 STEP 1308 H= 327:00 +CPU= 36.630
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001308
  22:16:02 STEP 1309 H= 327:15 +CPU= 26.248
  22:16:32 STEP 1310 H= 327:30 +CPU= 30.146
  22:17:03 STEP 1311 H= 327:45 +CPU= 30.316
  22:17:31 STEP 1312 H= 328:00 +CPU= 27.805
  22:17:59 STEP 1313 H= 328:15 +CPU= 28.243
  22:18:30 STEP 1314 H= 328:30 +CPU= 30.591
  22:18:56 STEP 1315 H= 328:45 +CPU= 25.680
  22:19:27 STEP 1316 H= 329:00 +CPU= 30.549
  22:19:55 STEP 1317 H= 329:15 +CPU= 26.853
  22:20:26 STEP 1318 H= 329:30 +CPU= 29.978
  22:20:56 STEP 1319 H= 329:45 +CPU= 30.287
  22:21:38 STEP 1320 H= 330:00 +CPU= 38.669
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001320
  22:22:10 STEP 1321 H= 330:15 +CPU= 31.107
  22:22:39 STEP 1322 H= 330:30 +CPU= 29.376
  22:23:09 STEP 1323 H= 330:45 +CPU= 29.080
  22:23:39 STEP 1324 H= 331:00 +CPU= 30.311
  22:24:07 STEP 1325 H= 331:15 +CPU= 27.489
  22:24:35 STEP 1326 H= 331:30 +CPU= 26.798
  22:25:05 STEP 1327 H= 331:45 +CPU= 26.707
  22:25:33 STEP 1328 H= 332:00 +CPU= 27.394
  22:25:57 STEP 1329 H= 332:15 +CPU= 24.260
  22:26:29 STEP 1330 H= 332:30 +CPU= 29.748
  22:26:58 STEP 1331 H= 332:45 +CPU= 28.052
  22:27:39 STEP 1332 H= 333:00 +CPU= 35.025
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001332
  22:28:07 STEP 1333 H= 333:15 +CPU= 28.662
  22:28:35 STEP 1334 H= 333:30 +CPU= 27.759
  22:29:05 STEP 1335 H= 333:45 +CPU= 29.665
  22:29:35 STEP 1336 H= 334:00 +CPU= 29.494
  22:30:02 STEP 1337 H= 334:15 +CPU= 26.468
  22:30:32 STEP 1338 H= 334:30 +CPU= 29.782
  22:30:57 STEP 1339 H= 334:45 +CPU= 25.182
  22:31:28 STEP 1340 H= 335:00 +CPU= 30.617
  22:31:55 STEP 1341 H= 335:15 +CPU= 26.810
  22:32:21 STEP 1342 H= 335:30 +CPU= 25.433
  22:32:48 STEP 1343 H= 335:45 +CPU= 27.001
  22:33:41 STEP 1344 H= 336:00 +CPU= 35.209
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a34f_2016092300_20_1017_12285874_0_r36598283_13.zip
  22:34:08 STEP 1345 H= 336:15 +CPU= 24.361
  22:34:34 STEP 1346 H= 336:30 +CPU= 25.411
  22:35:00 STEP 1347 H= 336:45 +CPU= 26.608
Uploading the intermediate file: upload_file_13.zip
  22:35:31 STEP 1348 H= 337:00 +CPU= 29.996
  22:36:00 STEP 1349 H= 337:15 +CPU= 29.308
  22:36:26 STEP 1350 H= 337:30 +CPU= 26.225
  22:36:54 STEP 1351 H= 337:45 +CPU= 27.018
  22:37:36 STEP 1352 H= 338:00 +CPU= 29.149
  22:38:08 STEP 1353 H= 338:15 +CPU= 29.525
  22:38:40 STEP 1354 H= 338:30 +CPU= 31.011
  22:39:12 STEP 1355 H= 338:45 +CPU= 31.410
  22:39:50 STEP 1356 H= 339:00 +CPU= 34.229
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001356
  22:40:19 STEP 1357 H= 339:15 +CPU= 28.916
  22:40:48 STEP 1358 H= 339:30 +CPU= 29.518
  22:41:20 STEP 1359 H= 339:45 +CPU= 31.012
  22:41:52 STEP 1360 H= 340:00 +CPU= 30.424
  22:42:20 STEP 1361 H= 340:15 +CPU= 27.873
  22:42:48 STEP 1362 H= 340:30 +CPU= 27.862
  22:43:17 STEP 1363 H= 340:45 +CPU= 28.357
  22:43:50 STEP 1364 H= 341:00 +CPU= 30.466
  22:44:19 STEP 1365 H= 341:15 +CPU= 28.767
  22:44:49 STEP 1366 H= 341:30 +CPU= 29.601
  22:45:14 STEP 1367 H= 341:45 +CPU= 25.292
  22:45:57 STEP 1368 H= 342:00 +CPU= 33.016
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001368
  22:46:25 STEP 1369 H= 342:15 +CPU= 27.208
  22:46:50 STEP 1370 H= 342:30 +CPU= 25.465
  22:47:18 STEP 1371 H= 342:45 +CPU= 27.215
  22:48:00 STEP 1372 H= 343:00 +CPU= 31.585
  22:48:35 STEP 1373 H= 343:15 +CPU= 27.225
  22:49:10 STEP 1374 H= 343:30 +CPU= 32.318
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:49:54 STEP 1375 H= 343:45 +CPU= 31.270
  22:50:23 STEP 1376 H= 344:00 +CPU= 28.831
  22:50:49 STEP 1377 H= 344:15 +CPU= 25.756
  22:51:15 STEP 1378 H= 344:30 +CPU= 26.024
  22:51:46 STEP 1379 H= 344:45 +CPU= 29.638
  22:52:48 STEP 1380 H= 345:00 +CPU= 33.810
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001380
  22:53:18 STEP 1381 H= 345:15 +CPU= 26.870
  22:53:49 STEP 1382 H= 345:30 +CPU= 31.125
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:54:28 STEP 1383 H= 345:45 +CPU= 28.099
  22:54:56 STEP 1384 H= 346:00 +CPU= 27.317
  22:55:26 STEP 1385 H= 346:15 +CPU= 29.951
  22:55:54 STEP 1386 H= 346:30 +CPU= 25.670
  22:56:21 STEP 1387 H= 346:45 +CPU= 27.164
  22:56:50 STEP 1388 H= 347:00 +CPU= 27.511
  22:57:44 STEP 1389 H= 347:15 +CPU= 31.788
  22:58:20 STEP 1390 H= 347:30 +CPU= 24.239
  22:59:01 STEP 1391 H= 347:45 +CPU= 32.202
  23:00:32 STEP 1392 H= 348:00 +CPU= 40.061
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001392
  23:01:23 STEP 1393 H= 348:15 +CPU= 29.181
  23:02:21 STEP 1394 H= 348:30 +CPU= 30.432
  23:03:14 STEP 1395 H= 348:45 +CPU= 29.589
  23:03:50 STEP 1396 H= 349:00 +CPU= 30.117
  23:04:24 STEP 1397 H= 349:15 +CPU= 32.805
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:05:05 STEP 1398 H= 349:30 +CPU= 29.428
  23:05:32 STEP 1399 H= 349:45 +CPU= 27.147
  23:06:01 STEP 1400 H= 350:00 +CPU= 27.880
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a34f
(argv4) batchid: 1017
(argv5) wuid: 12285874
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/17
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/17/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/17/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d26c9eddddbb78a0584a09be110c4068 to: /var/lib/boinc-client/slots/17/oifs_43r3_bl_a34f_2016092300_20_1017_12285874.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/17/oifs_43r3_bl_a34f_2016092300_20_1017_12285874.zip
ic_ancil_file: ic_ancil_12285874
ifsdata_file: ifsdata_12285874
climate_data_file: clim_data_12285874
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/17/ic_ancil_12285874.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/17/ic_ancil_12285874.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8f231f1887df997466659c61938fd32a to: /var/lib/boinc-client/slots/17/ifsdata/ifsdata_12285874.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/17/ifsdata/ifsdata_12285874.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/17/159l_2/clim_data_12285874.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/17/159l_2/clim_data_12285874.zip
Checking for progress XML file: /var/lib/boinc-client/slots/17/progress_file_12285874.xml
Opened progress file ok : /var/lib/boinc-client/slots/17/progress_file_12285874.xml
-- Model is restarting --
Adjusting last_iter, 1401, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/17/progress_file_12285874.xml
last_cpu_time: 44714
upload_file_number: 14
last_iter: 1393
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a34f_2016092300_20_1017_12285874_0_r36598283
The child process has been launched with process id: 12054
Executing the command: /var/lib/boinc-client/slots/17/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:server242:1:1:12054:12054] [20240608:230812:1717880892:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:09:33 STEP 1392 H= 348:00 +CPU= 25.889
  23:09:58 STEP 1393 H= 348:15 +CPU= 24.491
  23:10:26 STEP 1394 H= 348:30 +CPU= 27.914
  23:10:52 STEP 1395 H= 348:45 +CPU= 25.421
  23:11:21 STEP 1396 H= 349:00 +CPU= 28.393
  23:11:47 STEP 1397 H= 349:15 +CPU= 26.153
  23:12:14 STEP 1398 H= 349:30 +CPU= 27.359
  23:12:42 STEP 1399 H= 349:45 +CPU= 27.672
  23:13:09 STEP 1400 H= 350:00 +CPU= 26.866
  23:13:36 STEP 1401 H= 350:15 +CPU= 26.150
  23:14:00 STEP 1402 H= 350:30 +CPU= 24.410
  23:14:30 STEP 1403 H= 350:45 +CPU= 29.624
  23:15:20 STEP 1404 H= 351:00 +CPU= 34.422
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001404
  23:15:46 STEP 1405 H= 351:15 +CPU= 25.487
  23:16:09 STEP 1406 H= 351:30 +CPU= 22.350
  23:16:34 STEP 1407 H= 351:45 +CPU= 24.958
  23:17:04 STEP 1408 H= 352:00 +CPU= 30.469
  23:17:39 STEP 1409 H= 352:15 +CPU= 25.849
  23:18:06 STEP 1410 H= 352:30 +CPU= 23.884
  23:18:32 STEP 1411 H= 352:45 +CPU= 25.502
  23:18:58 STEP 1412 H= 353:00 +CPU= 25.262
  23:19:22 STEP 1413 H= 353:15 +CPU= 22.623
  23:20:03 STEP 1414 H= 353:30 +CPU= 29.514
  23:20:44 STEP 1415 H= 353:45 +CPU= 28.148
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:21:43 STEP 1416 H= 354:00 +CPU= 38.423
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001416
  23:22:09 STEP 1417 H= 354:15 +CPU= 25.723
  23:22:35 STEP 1418 H= 354:30 +CPU= 25.786
  23:23:04 STEP 1419 H= 354:45 +CPU= 26.858
  23:23:34 STEP 1420 H= 355:00 +CPU= 28.141
  23:24:05 STEP 1421 H= 355:15 +CPU= 31.073
  23:24:36 STEP 1422 H= 355:30 +CPU= 30.720
  23:25:05 STEP 1423 H= 355:45 +CPU= 28.590
  23:25:33 STEP 1424 H= 356:00 +CPU= 27.065
  23:25:58 STEP 1425 H= 356:15 +CPU= 25.005
  23:26:27 STEP 1426 H= 356:30 +CPU= 29.391
  23:26:56 STEP 1427 H= 356:45 +CPU= 26.735
  23:27:43 STEP 1428 H= 357:00 +CPU= 37.121
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001428
  23:28:11 STEP 1429 H= 357:15 +CPU= 27.329
  23:28:38 STEP 1430 H= 357:30 +CPU= 26.935
  23:29:07 STEP 1431 H= 357:45 +CPU= 29.094
  23:29:40 STEP 1432 H= 358:00 +CPU= 31.868
  23:30:10 STEP 1433 H= 358:15 +CPU= 29.518
  23:30:41 STEP 1434 H= 358:30 +CPU= 31.301
  23:31:11 STEP 1435 H= 358:45 +CPU= 29.760
  23:31:38 STEP 1436 H= 359:00 +CPU= 26.185
  23:32:04 STEP 1437 H= 359:15 +CPU= 26.280
  23:32:33 STEP 1438 H= 359:30 +CPU= 29.010
  23:33:03 STEP 1439 H= 359:45 +CPU= 29.473
  23:34:01 STEP 1440 H= 360:00 +CPU= 38.754
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.3847180176E-21      I       0.4543297749E-05      I
ICloud fractionI      0.0000000000E+00      I       0.3499707310E-01      I
I  Relat. Hum. I      0.3081178074E+00      I       0.3753157446E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.5754571883E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    300044692 BYTES IN      7 RECORDS TRANSFERRED IN    0.0007 SECONDS428635.3 Mbytes/s, TOTAL TIME=    0.0149( 4.7%)
 23:34:01 STEP 1440 H= 360:00 +CPU= 38.754
 END CNT3
 NSTEP =  1440 CNT0   000000000  27.251  27.251  28.177   0.967               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         1430/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     28    0       46      1044068     1430 ************    45.4    39.6     5.8    40.6    29.2
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 1549575.1 1549575.1    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 1505815.5 1505815.5     97.18      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                   53    5060.7    5060.7     17.31      0.00
   9 SPCM     - SPECTRAL COMP.                      48     819.2     819.2      2.54      0.00
  10 SCAN2M - PHYSICS                               49   13149.0   13149.0     41.58      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  4    2849.9    2849.9      0.74      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  3     460.2     460.2      0.09      0.00
  17 GRIDFPOS IN CNT4                                4     224.5     224.5      0.06      0.00
  27 RESTART FILES - WRITING                         1   10731.9   10731.9      0.69      0.00
  28 RESTART FILES - READING                         1   41539.6   41539.6      2.68      0.00
  29 SU4FPOS IN CNT4                                 4     173.7     173.7      0.04      0.00
  30 DYNFPOS IN CNT4                                 4   10933.9   10933.9      2.82      0.00
  31 POSDDH IN STEPO                                 1     276.2     276.2      0.02      0.00
  37 CPGLAG   - SL COMPUTATIONS                     49    6320.8    6320.8     19.99      0.00
  39 SU0YOMB                                         1     497.7     497.7      0.03      0.00
  51 SCAN2M   - SL COMM. PART 1                     49      30.0      30.0      0.09      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               48     365.1     365.1      1.13      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               48      63.1      63.1      0.20      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  3     314.3     314.3      0.06      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      108     516.4     516.4      3.60      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       100     235.9     235.9      1.52      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        100      90.9      90.9      0.59      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       108     111.6     111.6      0.78      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      56.5      56.5      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            108      69.3      69.3      0.48      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            100      34.8      34.8      0.22      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            108      71.7      71.7      0.50      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            100      36.7      36.7      0.24      0.00
 400 GSTATS                                       6667       0.3       0.3      0.13      0.00
 401 GSTATS HOOK                                  6149       0.2       0.2      0.08      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     1549.6 CPU TIME    1380.4 VECTOR TIME     1380.4


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      860.7
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/17/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/17/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285874/ICMSHh7zg+001380
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a34f_2016092300_20_1017_12285874_0_r36598283_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
23:38:25 (12039): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a34f_2016092300_20_1017_12285874_0_r36598283_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a34f_2016092300_20_1017_12285874_0_r36598283_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a34f_2016092300_20_1017_12285874_0_r36598283_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a34f_2016092300_20_1017_12285874_0_r36598283_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a34f_2016092300_20_1017_12285874_0_r36598283_19.zip</file_name>
  <error_code>-161 (not found)</erro
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 21:38:03 1469580 22437486 oifs_43r3_bl_a34f_2016092300_20_1017_12285874_0 1,295,100 46,094 0.0356


©2024 cpdn.org