climateprediction.net home page
Task 22309532

Task 22309532

Name oifs_43r3_bl_a1yt_2016092300_15_991_12209336_0
Workunit 12209336
Created 13 Feb 2023, 16:42:15 UTC
Sent 13 Feb 2023, 16:43:23 UTC
Report deadline 14 Apr 2023, 16:43:23 UTC
Received 23 Feb 2023, 13:56:16 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1515394
Run time 1 days 1 hours 35 min 25 sec
CPU time 23 hours 23 min 41 sec
Validate state Valid
Credit 1,815.66
Device peak FLOPS 5.76 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 4,953.73 MB
Peak swap size 5,930.79 MB
Peak disk usage 2,360.05 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
] [20230223:015433:1677146073:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:841780:841780] [20230223:015433:1677146073:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:AHogben-P1:1:1:841780:841780] [20230223:015433:1677146073:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:841780:841780] [20230223:015433:1677146073:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:841780:841780] [20230223:015433:1677146073:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:841780:841780] [20230223:015433:1677146073:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:841780:841780] [20230223:015433:1677146073:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:841780:841780] [20230223:015433:1677146073:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:841780:841780] [20230223:015433:1677146073:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:841780:841780] [20230223:015433:1677146073:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:841780:841780] [20230223:015433:1677146073:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:841780:841780] [20230223:015433:1677146073:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:55:38 STEP 1248 H= 312:00 +CPU= 55.753
  01:56:37 STEP 1249 H= 312:15 +CPU= 53.905
  01:57:27 STEP 1250 H= 312:30 +CPU= 49.599
  01:58:22 STEP 1251 H= 312:45 +CPU= 54.209
  01:59:15 STEP 1252 H= 313:00 +CPU= 52.715
  02:00:06 STEP 1253 H= 313:15 +CPU= 50.386
  02:01:03 STEP 1254 H= 313:30 +CPU= 52.433
  02:01:59 STEP 1255 H= 313:45 +CPU= 54.094
  02:02:52 STEP 1256 H= 314:00 +CPU= 52.495
  02:03:43 STEP 1257 H= 314:15 +CPU= 50.287
  02:04:33 STEP 1258 H= 314:30 +CPU= 49.871
  02:05:26 STEP 1259 H= 314:45 +CPU= 51.827
  02:06:31 STEP 1260 H= 315:00 +CPU= 63.862
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001260
  02:07:25 STEP 1261 H= 315:15 +CPU= 52.636
  02:08:18 STEP 1262 H= 315:30 +CPU= 52.239
  02:09:07 STEP 1263 H= 315:45 +CPU= 49.275
  02:10:03 STEP 1264 H= 316:00 +CPU= 55.267
  02:10:54 STEP 1265 H= 316:15 +CPU= 49.781
  02:11:45 STEP 1266 H= 316:30 +CPU= 50.652
  02:12:37 STEP 1267 H= 316:45 +CPU= 52.090
  02:13:31 STEP 1268 H= 317:00 +CPU= 53.294
  02:14:21 STEP 1269 H= 317:15 +CPU= 49.189
  02:15:13 STEP 1270 H= 317:30 +CPU= 51.634
  02:16:04 STEP 1271 H= 317:45 +CPU= 50.122
  02:17:08 STEP 1272 H= 318:00 +CPU= 62.720
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001272
  02:18:00 STEP 1273 H= 318:15 +CPU= 52.065
  02:18:53 STEP 1274 H= 318:30 +CPU= 51.926
  02:19:49 STEP 1275 H= 318:45 +CPU= 55.344
  02:20:43 STEP 1276 H= 319:00 +CPU= 53.792
  02:21:36 STEP 1277 H= 319:15 +CPU= 51.552
  02:22:29 STEP 1278 H= 319:30 +CPU= 53.112
  02:23:20 STEP 1279 H= 319:45 +CPU= 49.931
  02:24:11 STEP 1280 H= 320:00 +CPU= 49.906
  02:25:01 STEP 1281 H= 320:15 +CPU= 50.355
  02:25:54 STEP 1282 H= 320:30 +CPU= 51.737
  02:26:44 STEP 1283 H= 320:45 +CPU= 50.118
  02:27:51 STEP 1284 H= 321:00 +CPU= 65.183
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001284
  02:28:42 STEP 1285 H= 321:15 +CPU= 50.608
  02:29:35 STEP 1286 H= 321:30 +CPU= 52.554
  02:30:27 STEP 1287 H= 321:45 +CPU= 51.214
  02:31:25 STEP 1288 H= 322:00 +CPU= 57.569
  02:32:14 STEP 1289 H= 322:15 +CPU= 49.144
  02:33:08 STEP 1290 H= 322:30 +CPU= 52.775
  02:34:00 STEP 1291 H= 322:45 +CPU= 52.285
  02:34:54 STEP 1292 H= 323:00 +CPU= 52.723
  02:35:46 STEP 1293 H= 323:15 +CPU= 52.193
  02:36:38 STEP 1294 H= 323:30 +CPU= 50.677
  02:37:30 STEP 1295 H= 323:45 +CPU= 51.684
  02:38:50 STEP 1296 H= 324:00 +CPU= 70.647
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001296
  02:39:46 STEP 1297 H= 324:15 +CPU= 54.569
  02:40:37 STEP 1298 H= 324:30 +CPU= 49.086
  02:41:33 STEP 1299 H= 324:45 +CPU= 54.902
  02:42:30 STEP 1300 H= 325:00 +CPU= 56.000
  02:43:20 STEP 1301 H= 325:15 +CPU= 48.541
  02:44:09 STEP 1302 H= 325:30 +CPU= 48.985
  02:45:01 STEP 1303 H= 325:45 +CPU= 50.837
  02:45:52 STEP 1304 H= 326:00 +CPU= 51.048
  02:46:45 STEP 1305 H= 326:15 +CPU= 52.619
  02:47:35 STEP 1306 H= 326:30 +CPU= 49.462
  02:48:25 STEP 1307 H= 326:45 +CPU= 48.735
  02:49:37 STEP 1308 H= 327:00 +CPU= 66.102
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001308
  02:50:29 STEP 1309 H= 327:15 +CPU= 50.950
  02:51:21 STEP 1310 H= 327:30 +CPU= 51.032
  02:52:29 STEP 1311 H= 327:45 +CPU= 54.437
  02:53:34 STEP 1312 H= 328:00 +CPU= 58.969
  02:54:46 STEP 1313 H= 328:15 +CPU= 61.565
  02:55:40 STEP 1314 H= 328:30 +CPU= 53.120
  02:56:35 STEP 1315 H= 328:45 +CPU= 53.884
  02:57:29 STEP 1316 H= 329:00 +CPU= 54.406
  02:58:28 STEP 1317 H= 329:15 +CPU= 55.838
  02:59:20 STEP 1318 H= 329:30 +CPU= 50.477
  03:00:13 STEP 1319 H= 329:45 +CPU= 53.145
  03:01:23 STEP 1320 H= 330:00 +CPU= 66.439
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001320
  03:02:12 STEP 1321 H= 330:15 +CPU= 48.625
  03:03:05 STEP 1322 H= 330:30 +CPU= 52.358
  03:03:58 STEP 1323 H= 330:45 +CPU= 51.841
  03:04:49 STEP 1324 H= 331:00 +CPU= 51.395
  03:05:38 STEP 1325 H= 331:15 +CPU= 48.762
  03:06:32 STEP 1326 H= 331:30 +CPU= 53.188
  03:07:25 STEP 1327 H= 331:45 +CPU= 52.848
  03:08:21 STEP 1328 H= 332:00 +CPU= 55.078
  03:09:13 STEP 1329 H= 332:15 +CPU= 50.073
  03:10:08 STEP 1330 H= 332:30 +CPU= 54.319
  03:10:58 STEP 1331 H= 332:45 +CPU= 48.964
  03:12:03 STEP 1332 H= 333:00 +CPU= 65.108
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001332
  03:12:56 STEP 1333 H= 333:15 +CPU= 52.462
  03:13:52 STEP 1334 H= 333:30 +CPU= 53.748
  03:14:43 STEP 1335 H= 333:45 +CPU= 49.907
  03:15:36 STEP 1336 H= 334:00 +CPU= 53.155
  03:16:28 STEP 1337 H= 334:15 +CPU= 50.833
  03:17:21 STEP 1338 H= 334:30 +CPU= 52.899
  03:18:14 STEP 1339 H= 334:45 +CPU= 52.237
  03:19:10 STEP 1340 H= 335:00 +CPU= 54.907
  03:20:07 STEP 1341 H= 335:15 +CPU= 55.607
  03:20:57 STEP 1342 H= 335:30 +CPU= 48.730
  03:21:51 STEP 1343 H= 335:45 +CPU= 52.701
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

 03:18:14 STEP 1339 H= 334:45 +CPU= 52.237
ISTEP=    1340 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1340 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1340 STEPO   0AAA00AAA
 03:19:10 STEP 1340 H= 335:00 +CPU= 54.907
 NSTEP =  1341 STEPO   0AAA00AAA
 03:20:07 STEP 1341 H= 335:15 +CPU= 55.607
 NSTEP =  1342 STEPO   0AAA00AAA
 03:20:57 STEP 1342 H= 335:30 +CPU= 48.730
 NSTEP =  1343 STEPO   0AAA00AAA
 03:21:51 STEP 1343 H= 335:45 +CPU= 52.701
  DATE= 2016   10    6
ISTEP=    1344: Reset RSOLINC= 1361.4482 RIP0= 1361.4482
 SU_GHGCLIM, RCARDI,RCH4,RN2O,RCFC11,RCFC12,RNO2:   6.179466599480000E-004
  1.088035170184667E-006  4.990317592110001E-007  3.778213382954999E-009
  2.097581753513333E-009  7.940283000000001E-011
 suecaec NAERMACC =  1
 UPDT RAESHDU =  2000.0
 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT)     20160923
          90  9.999999999999998E-005
  DATE= 2016   10    6
 NORMS AT NSTEP CNT4 1344
 SPECTRAL NORMS -   LOG(PREHYDS)    0.115129225945924E+02
 LEV     VORTICITY             DIVERGENCE           TEMPERATURE           KINETIC ENERGY    
 AVE 0.139376476636926E-04 0.273833490346185E-05 0.213867092726018E+03 0.205730472559323E+03
 GPNORM  HUMIDITY            AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.842133799095334E-03 0.999999999990477E-08 0.153814872023025E-01
 GPNORM  LIQUID WATER        AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.111202051722143E-05 0.000000000000000E+00 0.739474335668744E-03
 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.270496092292918E-06 0.000000000000000E+00 0.165339102752436E-03
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.279500136390044E-06 0.000000000000000E+00 0.606265150440975E-03
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.246602311893381E-07 0.000000000000000E+00 0.952570653791225E-04
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.230905846164741E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.111190241753447E-05 0.000000000000000E+00 0.737574281951732E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.270369391151779E-06 0.000000000000000E+00 0.171232528904474E-03
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.230741440985252E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.112960312357476E-06 0.000000000000000E+00 0.425041302363012E-03
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.359010110932209E-06 0.000000000000000E+00 0.651685055665716E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.806261872915088E-07 0.000000000000000E+00 0.228668172394126E-03
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.785337133161726E-08 0.000000000000000E+00 0.264659129668243E-03
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.489303117742844E-01 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1344 STEPO   F00000000
 CREATE RESTART FILE ./srf00140000.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00140000.0001 MODE=w
 IO-STREAM CLOSED - ./srf00140000.0001
 MPI-TASK:   1 - ************ BYTES IN     34 RECORDS TRANSFERRED IN    5.1944 SECONDS******** Mbytes/s, TOTAL TIME=   16.9212(30.7%)
 CREATE RESTART CONTROL FILE ./rcf                                                                                                                   
 WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00131200.0001
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1344 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001344 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001344
 MPI-TASK:   1 -    304559818 BYTES IN      6 RECORDS TRANSFERRED IN    0.0008 SECONDS380699.8 Mbytes/s, TOTAL TIME=    0.0525( 1.5%)
 NSTEP =  1344 STEPO   0AA000000
------------------------------------------------
CNT0 not found; string returned was: 'DYNFPOS'
>>> Printing last 8 lines from file: ifs.stat

 03:18:14 0AAA00AAA STEPO     1340    52.237   52.237   52.968     81:40     83:41 0.26904818808853E-05       2GB       0MB
 03:19:10 0AAA00AAA STEPO     1341    54.907   54.907   55.438     82:34     84:36 0.26904818808853E-05       2GB       0MB
 03:20:07 0AAA00AAA STEPO     1342    55.609   55.609   57.364     83:30     85:34 0.26904818808853E-05       2GB       0MB
 03:20:57 0AAA00AAA STEPO     1343    48.730   48.730   49.662     84:19     86:23 0.26904818808853E-05       2GB       0MB
 03:21:51 F00000000 STEPO     1344    53.058   53.058   54.330     85:12     87:18 0.27383349034618E-05       2GB       0MB
 03:22:09 0AA000000 STEPO     1344     7.551    7.551   17.640     85:19     87:35 0.27383349034618E-05       2GB       0MB
 03:22:12 FULLPOS-B DYNFPOS   1344     3.312    3.312    3.331     85:23     87:39 0.27383349034618E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1344",
 CTIME   = "00140000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1332, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/3/progress_file_12209336.xml

<running_values>
  <last_cpu_time>79011.910000</last_cpu_time>
  <upload_file_number>13</upload_file_number>
  <last_iter>1344</last_iter>
  <last_upload>1124100</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1yt
(argv4) batchid: 991
(argv5) wuid: 12209336
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_41896e9817ecaa389c1209bd2d42c47d to: /var/lib/boinc-client/slots/3/oifs_43r3_bl_a1yt_2016092300_15_991_12209336.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_bl_a1yt_2016092300_15_991_12209336.zip
ic_ancil_file: ic_ancil_12209336
ifsdata_file: ifsdata_12209336
climate_data_file: clim_data_12209336
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/3/ic_ancil_12209336.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12209336.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2c37f3143c42d52f5ba12e0acc4a128b to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12209336.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12209336.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12209336.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12209336.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12209336.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12209336.xml
-- Model is restarting --
Adjusting last_iter, 1344, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12209336.xml
last_cpu_time: 79011
upload_file_number: 13
last_iter: 1345
last_upload: 1124100
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a1yt_2016092300_15_991_12209336_0_r347271434
The child process has been launched with process id: 844188
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:844188:844188] [20230223:032326:1677151406:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1yt_2016092300_15_991_12209336_0_r347271434_13.zip
Uploading the intermediate file: upload_file_13.zip
  03:24:33 STEP 1344 H= 336:00 +CPU= 54.797
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001344
  03:25:25 STEP 1345 H= 336:15 +CPU= 51.117
  03:26:19 STEP 1346 H= 336:30 +CPU= 53.360
  03:27:17 STEP 1347 H= 336:45 +CPU= 56.795
  03:28:14 STEP 1348 H= 337:00 +CPU= 55.319
  03:29:06 STEP 1349 H= 337:15 +CPU= 51.695
  03:30:00 STEP 1350 H= 337:30 +CPU= 54.501
  03:30:51 STEP 1351 H= 337:45 +CPU= 49.760
  03:31:44 STEP 1352 H= 338:00 +CPU= 51.607
  03:32:38 STEP 1353 H= 338:15 +CPU= 52.921
  03:33:30 STEP 1354 H= 338:30 +CPU= 51.198
  03:34:24 STEP 1355 H= 338:45 +CPU= 51.658
  03:35:29 STEP 1356 H= 339:00 +CPU= 64.337
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001356
  03:36:22 STEP 1357 H= 339:15 +CPU= 52.648
  03:37:15 STEP 1358 H= 339:30 +CPU= 51.911
  03:38:07 STEP 1359 H= 339:45 +CPU= 51.660
  03:38:57 STEP 1360 H= 340:00 +CPU= 50.063
  03:39:53 STEP 1361 H= 340:15 +CPU= 54.833
  03:40:45 STEP 1362 H= 340:30 +CPU= 51.587
  03:41:40 STEP 1363 H= 340:45 +CPU= 53.835
  03:42:32 STEP 1364 H= 341:00 +CPU= 52.261
  03:43:31 STEP 1365 H= 341:15 +CPU= 56.497
  03:44:20 STEP 1366 H= 341:30 +CPU= 49.397
  03:45:14 STEP 1367 H= 341:45 +CPU= 52.235
  03:46:19 STEP 1368 H= 342:00 +CPU= 63.278
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001368
  03:47:13 STEP 1369 H= 342:15 +CPU= 52.902
  03:48:02 STEP 1370 H= 342:30 +CPU= 48.806
  03:48:57 STEP 1371 H= 342:45 +CPU= 53.723
  03:49:52 STEP 1372 H= 343:00 +CPU= 54.455
  03:50:45 STEP 1373 H= 343:15 +CPU= 51.797
  03:51:36 STEP 1374 H= 343:30 +CPU= 50.197
  03:52:30 STEP 1375 H= 343:45 +CPU= 53.209
  03:53:24 STEP 1376 H= 344:00 +CPU= 53.013
  03:54:18 STEP 1377 H= 344:15 +CPU= 53.684
  03:55:12 STEP 1378 H= 344:30 +CPU= 53.530
  03:56:04 STEP 1379 H= 344:45 +CPU= 51.036
  03:57:13 STEP 1380 H= 345:00 +CPU= 67.840
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001380
  03:58:05 STEP 1381 H= 345:15 +CPU= 51.674
  03:58:57 STEP 1382 H= 345:30 +CPU= 51.663
  03:59:51 STEP 1383 H= 345:45 +CPU= 53.077
  04:00:46 STEP 1384 H= 346:00 +CPU= 54.250
  04:01:42 STEP 1385 H= 346:15 +CPU= 54.910
  04:02:33 STEP 1386 H= 346:30 +CPU= 51.416
  04:03:30 STEP 1387 H= 346:45 +CPU= 54.277
  04:04:22 STEP 1388 H= 347:00 +CPU= 51.497
  04:05:14 STEP 1389 H= 347:15 +CPU= 51.902
  04:06:04 STEP 1390 H= 347:30 +CPU= 48.906
  04:06:58 STEP 1391 H= 347:45 +CPU= 52.319
  04:08:16 STEP 1392 H= 348:00 +CPU= 71.120
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001392
  04:09:07 STEP 1393 H= 348:15 +CPU= 51.058
  04:10:03 STEP 1394 H= 348:30 +CPU= 54.625
  04:10:57 STEP 1395 H= 348:45 +CPU= 52.726
  04:11:54 STEP 1396 H= 349:00 +CPU= 53.469
  04:12:50 STEP 1397 H= 349:15 +CPU= 54.425
  04:13:43 STEP 1398 H= 349:30 +CPU= 52.835
  04:14:37 STEP 1399 H= 349:45 +CPU= 53.398
  04:15:34 STEP 1400 H= 350:00 +CPU= 54.205
  04:16:29 STEP 1401 H= 350:15 +CPU= 53.741
  04:17:22 STEP 1402 H= 350:30 +CPU= 52.029
  04:18:17 STEP 1403 H= 350:45 +CPU= 55.004
  04:19:26 STEP 1404 H= 351:00 +CPU= 66.986
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001404
  04:20:16 STEP 1405 H= 351:15 +CPU= 49.910
  04:21:07 STEP 1406 H= 351:30 +CPU= 50.777
  04:21:59 STEP 1407 H= 351:45 +CPU= 51.077
  04:22:57 STEP 1408 H= 352:00 +CPU= 54.870
  04:23:48 STEP 1409 H= 352:15 +CPU= 50.944
  04:24:41 STEP 1410 H= 352:30 +CPU= 52.118
  04:25:34 STEP 1411 H= 352:45 +CPU= 52.282
  04:26:28 STEP 1412 H= 353:00 +CPU= 54.312
  04:27:21 STEP 1413 H= 353:15 +CPU= 50.874
  04:28:16 STEP 1414 H= 353:30 +CPU= 54.153
  04:29:02 STEP 1415 H= 353:45 +CPU= 46.560
  04:30:10 STEP 1416 H= 354:00 +CPU= 66.075
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001416
  04:31:03 STEP 1417 H= 354:15 +CPU= 52.345
  04:31:59 STEP 1418 H= 354:30 +CPU= 55.011
  04:32:54 STEP 1419 H= 354:45 +CPU= 55.356
  04:33:50 STEP 1420 H= 355:00 +CPU= 54.825
  04:34:42 STEP 1421 H= 355:15 +CPU= 51.403
  04:35:30 STEP 1422 H= 355:30 +CPU= 48.036
  04:36:24 STEP 1423 H= 355:45 +CPU= 52.607
  04:37:19 STEP 1424 H= 356:00 +CPU= 54.734
  04:38:12 STEP 1425 H= 356:15 +CPU= 51.937
  04:39:02 STEP 1426 H= 356:30 +CPU= 48.323
  04:39:59 STEP 1427 H= 356:45 +CPU= 55.700
  04:41:05 STEP 1428 H= 357:00 +CPU= 62.426
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001428
  04:41:58 STEP 1429 H= 357:15 +CPU= 52.580
  04:42:48 STEP 1430 H= 357:30 +CPU= 49.929
  04:43:42 STEP 1431 H= 357:45 +CPU= 53.981
  04:44:33 STEP 1432 H= 358:00 +CPU= 50.354
  04:45:27 STEP 1433 H= 358:15 +CPU= 53.098
  04:46:19 STEP 1434 H= 358:30 +CPU= 50.897
  04:47:12 STEP 1435 H= 358:45 +CPU= 51.993
  04:48:06 STEP 1436 H= 359:00 +CPU= 53.841
  04:49:04 STEP 1437 H= 359:15 +CPU= 53.500
  04:49:57 STEP 1438 H= 359:30 +CPU= 51.591
  04:50:48 STEP 1439 H= 359:45 +CPU= 51.023
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

 04:49:57 STEP 1438 H= 359:30 +CPU= 51.591
 NSTEP =  1439 STEPO   0AAA00AAA
 04:50:48 STEP 1439 H= 359:45 +CPU= 51.023
ISTEP=    1440 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1440 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
  DATE= 2016   10    7
ISTEP=    1440: Reset RSOLINC= 1361.4482 RIP0= 1361.4482
 SU_GHGCLIM, RCARDI,RCH4,RN2O,RCFC11,RCFC12,RNO2:   6.179466599480000E-004
  1.088035170184667E-006  4.990317592110001E-007  3.778213382954999E-009
  2.097581753513333E-009  7.940283000000001E-011
 suecaec NAERMACC =  1
 UPDT RAESHDU =  2000.0
 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT)     20160923
          90  9.999999999999998E-005
  DATE= 2016   10    7
 NORMS AT NSTEP CNT4 1440
 SPECTRAL NORMS -   LOG(PREHYDS)    0.115129235759867E+02
 LEV     VORTICITY             DIVERGENCE           TEMPERATURE           KINETIC ENERGY    
 AVE 0.152440668063688E-04 0.308785278971483E-05 0.213899521710470E+03 0.209667429234419E+03
 GPNORM  HUMIDITY            AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.867349593928968E-03 0.999999999981102E-08 0.154951196991593E-01
 GPNORM  LIQUID WATER        AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.109708006259606E-05 0.000000000000000E+00 0.728312087607406E-03
 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.380878915756387E-06 0.000000000000000E+00 0.138961107295582E-03
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.257429618399232E-06 0.000000000000000E+00 0.331905864437736E-03
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.191604287975402E-07 0.000000000000000E+00 0.514052262006419E-04
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.302630382214583E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.109656796610854E-05 0.000000000000000E+00 0.744913044049091E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.378759800274126E-06 0.000000000000000E+00 0.140772012355108E-03
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.301541414927221E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.879466637742888E-07 0.000000000000000E+00 0.247343440205912E-03
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.365775354659197E-06 0.000000000000000E+00 0.364769052042394E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.901859243718228E-07 0.000000000000000E+00 0.170537977473882E-03
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.884830704217591E-08 0.000000000000000E+00 0.122149939445641E-03
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.630114836959081E-01 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1440 STEPO   F00000000
 CREATE RESTART FILE ./srf00150000.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00150000.0001 MODE=w
 IO-STREAM CLOSED - ./srf00150000.0001
 MPI-TASK:   1 - ************ BYTES IN     34 RECORDS TRANSFERRED IN    5.5068 SECONDS******** Mbytes/s, TOTAL TIME=   15.3056(36.0%)
 CREATE RESTART CONTROL FILE ./rcf                                                                                                                   
 WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00141200.0001
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    298108618 BYTES IN      6 RECORDS TRANSFERRED IN    0.0008 SECONDS372635.8 Mbytes/s, TOTAL TIME=    0.0328( 2.4%)
 NSTEP =  1440 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001440 MODE=a
 IO-STREAM CLOSED - ./ICMSHh7zg+001440
 MPI-TASK:   1 -    707765250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0191 SECONDS 37055.8 Mbytes/s, TOTAL TIME=    0.6997( 2.7%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001440 MODE=a
------------------------------------------------
CNT0 not found; string returned was: 'DYNFPOS'
>>> Printing last 8 lines from file: ifs.stat

 04:47:12 0AAA00AAA STEPO     1436    51.993   51.993   52.422     82:12     83:45 0.29422553289678E-05       2GB       0MB
 04:48:06 0AAA00AAA STEPO     1437    53.841   53.841   54.365     83:06     84:39 0.29422553289678E-05       2GB       0MB
 04:49:04 0AAA00AAA STEPO     1438    53.501   53.501   57.559     83:59     85:37 0.29422553289678E-05       2GB       0MB
 04:49:57 0AAA00AAA STEPO     1439    51.591   51.591   52.665     84:51     86:29 0.29422553289678E-05       2GB       0MB
 04:50:48 F00000000 STEPO     1440    51.345   51.345   51.697     85:42     87:21 0.30878527897148E-05       2GB       0MB
 04:51:04 0AA000000 STEPO     1440     7.490    7.490   15.945     85:50     87:37 0.30878527897148E-05       2GB       0MB
 04:51:08 FULLPOS-B DYNFPOS   1440     3.435    3.435    3.446     85:53     87:41 0.30878527897148E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1440",
 CTIME   = "00150000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1428, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/3/progress_file_12209336.xml

<running_values>
  <last_cpu_time>84162.180000</last_cpu_time>
  <upload_file_number>14</upload_file_number>
  <last_iter>1440</last_iter>
  <last_upload>1210500</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1yt
(argv4) batchid: 991
(argv5) wuid: 12209336
(argv6) fclen: 15
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.11
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_41896e9817ecaa389c1209bd2d42c47d to: /var/lib/boinc-client/slots/3/oifs_43r3_bl_a1yt_2016092300_15_991_12209336.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_bl_a1yt_2016092300_15_991_12209336.zip
ic_ancil_file: ic_ancil_12209336
ifsdata_file: ifsdata_12209336
climate_data_file: clim_data_12209336
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/3/ic_ancil_12209336.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12209336.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_2c37f3143c42d52f5ba12e0acc4a128b to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12209336.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12209336.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12209336.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12209336.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12209336.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12209336.xml
-- Model is restarting --
Adjusting last_iter, 1440, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12209336.xml
last_cpu_time: 84162
upload_file_number: 14
last_iter: 1441
last_upload: 1210500
model_completed: 0
total_length_of_simulation: 1296000
result_base_name: oifs_43r3_bl_a1yt_2016092300_15_991_12209336_0_r347271434
The child process has been launched with process id: 846492
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140)
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil))
[EC_DRHOOK:AHogben-P1:1:1:846492:846492] [20230223:045221:1677156741:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:53:23 STEP 1440 H= 360:00 +CPU= 51.742
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

I      I Hor. Flux of PEII     -0.00000 I              I              I              I               I
I      I Vert.Flux of PEII      0.00000 I              I              I              I               I
I      I Work Pres.G.F  II              I      1.47563 I              I              I               I
I      I Conv. Ad.-Int. II     -1.47546 I              I              I              I               I
DDH--------------------------------------------------------------------------------------------------I
I DYN- Horizont. Flux   II      0.04332 I     -0.36881 I      0.80661 I     -0.00600 I     -0.00526  I
I    I Vertical Flux    II      0.00000 I      0.00000 I      0.00000 I      0.00000 I      0.00000  I
DDH--------------------------------------------------------------------------------------------------I
  
DDH----------------------------------------------------------------------------------------------------------------------------------
  I   - Mass balance    (10**(-6)  KG/M**2) 
  
  I INITIAL      I HORIZ. FLUX  I DYN. FLUX    I DYN. FLUX    I PHYS. FLUX   I PHYS. FLUX   I CURRENT      I I I RESIDUAL     I
  I MASS         I              I TOA          I SURFACE      I TOA          I SURFACE      I MASS         I I I              I
  ---------------------------------------------------------------------------------------------------------- I ----------------
  -              -              -              -              -              -              -              - I -              -
  - 10206523039. I           0. .           0. .           0. .           0. .           0. I 10207571266. - I -     1048226. -
  -              -              -              -              -              -              -              - I -              -
  ---------------------------------------------------------------------------------------------------------- I ----------------
 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.6486116215E-21      I       0.3807156128E-04      I
ICloud fractionI      0.0000000000E+00      I       0.4791096246E-01      I
I  Relat. Hum. I      0.1229406374E+00      I       0.2491683120E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.7865952025E+03      I
DDH------------------------------------------------------------------------
  
 04:53:23 STEP 1440 H= 360:00 +CPU= 51.742
 END CNT3
 NSTEP =  1440 CNT0   000000000  51.502  51.502  52.305   0.985               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           34/  1069125708 WRITTEN            0/           0
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1      3    0       34      1044068        0            0     5.7     5.7     0.0     7.1     0.0
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1   61885.1   61885.1    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1   52559.6   52559.6     84.93      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                    1    9642.6    9642.6     15.58      0.00
  10 SCAN2M - PHYSICS                                1   27458.3   27458.3     44.37      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  1      97.3      97.3      0.16      0.00
  28 RESTART FILES - READING                         1    8289.1    8289.1     13.39      0.00
  31 POSDDH IN STEPO                                 1      11.0      11.0      0.02      0.00
  37 CPGLAG   - SL COMPUTATIONS                      1   12327.1   12327.1     19.92      0.00
  39 SU0YOMB                                         1     673.2     673.2      1.09      0.00
  51 SCAN2M   - SL COMM. PART 1                      1      91.5      91.5      0.15      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  1       0.4       0.4      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM        4     398.0     398.0      2.57      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM         4     120.1     120.1      0.78      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      74.0      74.0      0.12      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION              4      40.8      40.8      0.26      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION              4      79.4      79.4      0.51      0.00
 400 GSTATS                                       1083       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                   963       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME       61.9 CPU TIME      59.5 VECTOR TIME       59.5


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY    24657.7
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/3/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/3/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+000192
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+000096
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12209336/ICMSHh7zg+000444
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1yt_2016092300_15_991_12209336_0_r347271434_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1296900
04:57:30 (846487): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:52:35 1515394 22309532 oifs_43r3_bl_a1yt_2016092300_15_991_12209336_0 1,296,900 84,221 0.0649


©2024 cpdn.org