climateprediction.net home page
Task 22447944

Task 22447944

Name oifs_43r3_bl_a4mo_2016092300_20_1018_12294327_0
Workunit 12294327
Created 13 Jun 2024, 4:41:56 UTC
Sent 13 Jun 2024, 21:55:19 UTC
Report deadline 12 Aug 2024, 21:55:19 UTC
Received 23 Jun 2024, 1:35:36 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1551031
Run time 19 hours 26 min
CPU time 15 hours 53 min 37 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 4.04 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.90 MB
Peak swap size 5,981.12 MB
Peak disk usage 1,370.61 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
saux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:1020379:1020379] [20240622:115053:1719078653:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:51:42 STEP 1536 H= 384:00 +CPU= 40.426
  11:52:20 STEP 1537 H= 384:15 +CPU= 35.543
  11:52:53 STEP 1538 H= 384:30 +CPU= 31.541
  11:53:27 STEP 1539 H= 384:45 +CPU= 32.161
  11:54:08 STEP 1540 H= 385:00 +CPU= 38.153
  11:54:49 STEP 1541 H= 385:15 +CPU= 37.791
  11:55:29 STEP 1542 H= 385:30 +CPU= 37.404
  11:56:03 STEP 1543 H= 385:45 +CPU= 31.507
  11:56:37 STEP 1544 H= 386:00 +CPU= 31.985
  11:57:16 STEP 1545 H= 386:15 +CPU= 36.498
  11:57:56 STEP 1546 H= 386:30 +CPU= 36.985
  11:58:36 STEP 1547 H= 386:45 +CPU= 36.946
  11:59:19 STEP 1548 H= 387:00 +CPU= 40.114
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001548
  11:59:54 STEP 1549 H= 387:15 +CPU= 32.607
  12:00:37 STEP 1550 H= 387:30 +CPU= 38.569
  12:01:19 STEP 1551 H= 387:45 +CPU= 38.283
  12:02:01 STEP 1552 H= 388:00 +CPU= 38.271
  12:02:37 STEP 1553 H= 388:15 +CPU= 32.066
  12:03:12 STEP 1554 H= 388:30 +CPU= 32.393
  12:03:55 STEP 1555 H= 388:45 +CPU= 38.651
  12:04:39 STEP 1556 H= 389:00 +CPU= 39.897
  12:05:22 STEP 1557 H= 389:15 +CPU= 38.341
  12:05:57 STEP 1558 H= 389:30 +CPU= 32.380
  12:06:33 STEP 1559 H= 389:45 +CPU= 32.452
  12:07:25 STEP 1560 H= 390:00 +CPU= 46.862
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001560
  12:08:08 STEP 1561 H= 390:15 +CPU= 38.593
  12:08:50 STEP 1562 H= 390:30 +CPU= 38.485
  12:09:27 STEP 1563 H= 390:45 +CPU= 33.113
  12:10:03 STEP 1564 H= 391:00 +CPU= 33.152
  12:10:45 STEP 1565 H= 391:15 +CPU= 37.618
  12:11:28 STEP 1566 H= 391:30 +CPU= 39.226
  12:12:13 STEP 1567 H= 391:45 +CPU= 39.177
  12:12:52 STEP 1568 H= 392:00 +CPU= 35.174
  12:13:28 STEP 1569 H= 392:15 +CPU= 32.227
  12:14:06 STEP 1570 H= 392:30 +CPU= 34.785
  12:14:48 STEP 1571 H= 392:45 +CPU= 38.273
  12:15:42 STEP 1572 H= 393:00 +CPU= 48.691
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001572
  12:16:19 STEP 1573 H= 393:15 +CPU= 32.842
  12:16:56 STEP 1574 H= 393:30 +CPU= 32.798
  12:17:41 STEP 1575 H= 393:45 +CPU= 38.633
  12:18:27 STEP 1576 H= 394:00 +CPU= 39.297
  12:19:14 STEP 1577 H= 394:15 +CPU= 39.259
  12:19:53 STEP 1578 H= 394:30 +CPU= 32.823
  12:20:33 STEP 1579 H= 394:45 +CPU= 33.775
  12:21:22 STEP 1580 H= 395:00 +CPU= 40.414
  12:22:11 STEP 1581 H= 395:15 +CPU= 39.479
  12:22:57 STEP 1582 H= 395:30 +CPU= 36.589
  12:23:38 STEP 1583 H= 395:45 +CPU= 33.315
  12:24:44 STEP 1584 H= 396:00 +CPU= 52.931
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001584
  12:25:34 STEP 1585 H= 396:15 +CPU= 39.600
  12:26:24 STEP 1586 H= 396:30 +CPU= 39.601
  12:27:07 STEP 1587 H= 396:45 +CPU= 35.028
  12:27:50 STEP 1588 H= 397:00 +CPU= 34.556
  12:28:40 STEP 1589 H= 397:15 +CPU= 39.334
  12:29:29 STEP 1590 H= 397:30 +CPU= 39.506
  12:30:19 STEP 1591 H= 397:45 +CPU= 39.600
  12:31:10 STEP 1592 H= 398:00 +CPU= 40.382
  12:31:52 STEP 1593 H= 398:15 +CPU= 34.494
  12:32:35 STEP 1594 H= 398:30 +CPU= 34.221
  12:33:25 STEP 1595 H= 398:45 +CPU= 39.642
  12:34:27 STEP 1596 H= 399:00 +CPU= 49.436
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001596
  12:35:17 STEP 1597 H= 399:15 +CPU= 39.265
  12:35:57 STEP 1598 H= 399:30 +CPU= 33.170
  12:36:42 STEP 1599 H= 399:45 +CPU= 35.686
  12:37:33 STEP 1600 H= 400:00 +CPU= 40.633
  12:38:23 STEP 1601 H= 400:15 +CPU= 39.574
  12:39:12 STEP 1602 H= 400:30 +CPU= 39.274
  12:39:53 STEP 1603 H= 400:45 +CPU= 32.842
  12:40:38 STEP 1604 H= 401:00 +CPU= 36.949
  12:41:29 STEP 1605 H= 401:15 +CPU= 39.702
  12:42:18 STEP 1606 H= 401:30 +CPU= 39.780
  12:43:08 STEP 1607 H= 401:45 +CPU= 39.619
  12:44:04 STEP 1608 H= 402:00 +CPU= 44.844
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001608
  12:44:47 STEP 1609 H= 402:15 +CPU= 34.347
  12:45:37 STEP 1610 H= 402:30 +CPU= 39.341
  12:46:26 STEP 1611 H= 402:45 +CPU= 39.520
  12:47:18 STEP 1612 H= 403:00 +CPU= 40.777
  12:48:08 STEP 1613 H= 403:15 +CPU= 39.526
  12:48:48 STEP 1614 H= 403:30 +CPU= 32.934
  12:49:34 STEP 1615 H= 403:45 +CPU= 36.247
  12:50:26 STEP 1616 H= 404:00 +CPU= 40.398
  12:51:20 STEP 1617 H= 404:15 +CPU= 39.936
  12:52:15 STEP 1618 H= 404:30 +CPU= 39.984
  12:52:59 STEP 1619 H= 404:45 +CPU= 34.110
  12:53:55 STEP 1620 H= 405:00 +CPU= 43.265
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001620
  12:54:46 STEP 1621 H= 405:15 +CPU= 39.133
  12:55:37 STEP 1622 H= 405:30 +CPU= 39.423
  12:56:30 STEP 1623 H= 405:45 +CPU= 39.442
  12:57:19 STEP 1624 H= 406:00 +CPU= 35.791
  12:58:08 STEP 1625 H= 406:15 +CPU= 35.564
  12:59:03 STEP 1626 H= 406:30 +CPU= 39.947
  12:59:59 STEP 1627 H= 406:45 +CPU= 39.985
  13:00:57 STEP 1628 H= 407:00 +CPU= 40.864
  13:01:48 STEP 1629 H= 407:15 +CPU= 36.699
  13:02:39 STEP 1630 H= 407:30 +CPU= 34.970
  13:03:38 STEP 1631 H= 407:45 +CPU= 40.215
  13:04:58 STEP 1632 H= 408:00 +CPU= 54.645
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4mo_2016092300_20_1018_12294327_0_r338955802_16.zip
Uploading the intermediate file: upload_file_16.zip
  13:06:00 STEP 1633 H= 408:15 +CPU= 40.234
  13:06:58 STEP 1634 H= 408:30 +CPU= 40.325
  13:07:58 STEP 1635 H= 408:45 +CPU= 40.338
  13:08:59 STEP 1636 H= 409:00 +CPU= 40.954
  13:09:51 STEP 1637 H= 409:15 +CPU= 36.183
  13:10:42 STEP 1638 H= 409:30 +CPU= 35.687
  13:11:41 STEP 1639 H= 409:45 +CPU= 40.187
  13:12:41 STEP 1640 H= 410:00 +CPU= 40.988
  13:13:40 STEP 1641 H= 410:15 +CPU= 40.384
  13:14:39 STEP 1642 H= 410:30 +CPU= 40.152
  13:15:37 STEP 1643 H= 410:45 +CPU= 40.133
  13:16:41 STEP 1644 H= 411:00 +CPU= 47.399
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001644
  13:17:27 STEP 1645 H= 411:15 +CPU= 34.363
  13:18:22 STEP 1646 H= 411:30 +CPU= 39.900
  13:19:17 STEP 1647 H= 411:45 +CPU= 39.928
  13:20:13 STEP 1648 H= 412:00 +CPU= 40.802
  13:21:08 STEP 1649 H= 412:15 +CPU= 40.184
  13:22:03 STEP 1650 H= 412:30 +CPU= 39.963
  13:22:58 STEP 1651 H= 412:45 +CPU= 39.959
  13:23:54 STEP 1652 H= 413:00 +CPU= 41.154
  13:24:53 STEP 1653 H= 413:15 +CPU= 40.172
  13:25:48 STEP 1654 H= 413:30 +CPU= 40.019
  13:26:43 STEP 1655 H= 413:45 +CPU= 39.758
  13:27:51 STEP 1656 H= 414:00 +CPU= 50.015
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001656
  13:28:44 STEP 1657 H= 414:15 +CPU= 39.931
  13:29:36 STEP 1658 H= 414:30 +CPU= 39.649
  13:30:31 STEP 1659 H= 414:45 +CPU= 39.961
  13:31:26 STEP 1660 H= 415:00 +CPU= 40.716
  13:32:16 STEP 1661 H= 415:15 +CPU= 39.174
  13:33:07 STEP 1662 H= 415:30 +CPU= 39.057
  13:33:55 STEP 1663 H= 415:45 +CPU= 35.972
  13:34:38 STEP 1664 H= 416:00 +CPU= 34.418
  13:35:27 STEP 1665 H= 416:15 +CPU= 39.364
  13:36:20 STEP 1666 H= 416:30 +CPU= 39.748
  13:37:13 STEP 1667 H= 416:45 +CPU= 39.829
  13:38:19 STEP 1668 H= 417:00 +CPU= 49.937
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001668
  13:39:12 STEP 1669 H= 417:15 +CPU= 40.115
  13:40:05 STEP 1670 H= 417:30 +CPU= 39.958
  13:40:58 STEP 1671 H= 417:45 +CPU= 39.861
  13:41:51 STEP 1672 H= 418:00 +CPU= 40.665
  13:42:44 STEP 1673 H= 418:15 +CPU= 39.955
  13:43:37 STEP 1674 H= 418:30 +CPU= 39.859
  13:44:31 STEP 1675 H= 418:45 +CPU= 40.028
  13:45:25 STEP 1676 H= 419:00 +CPU= 40.772
  13:46:19 STEP 1677 H= 419:15 +CPU= 39.853
  13:47:11 STEP 1678 H= 419:30 +CPU= 39.848
  13:48:05 STEP 1679 H= 419:45 +CPU= 39.869
  13:49:16 STEP 1680 H= 420:00 +CPU= 54.274
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001680
  13:50:09 STEP 1681 H= 420:15 +CPU= 39.909
  13:51:02 STEP 1682 H= 420:30 +CPU= 39.910
  13:51:56 STEP 1683 H= 420:45 +CPU= 40.129
  13:52:50 STEP 1684 H= 421:00 +CPU= 40.999
  13:53:43 STEP 1685 H= 421:15 +CPU= 40.148
  13:54:36 STEP 1686 H= 421:30 +CPU= 39.903
  13:55:29 STEP 1687 H= 421:45 +CPU= 39.943
  13:56:24 STEP 1688 H= 422:00 +CPU= 40.837
  13:57:17 STEP 1689 H= 422:15 +CPU= 40.036
  13:58:10 STEP 1690 H= 422:30 +CPU= 40.062
  13:59:04 STEP 1691 H= 422:45 +CPU= 40.012
  14:00:10 STEP 1692 H= 423:00 +CPU= 50.010
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001692
  14:01:03 STEP 1693 H= 423:15 +CPU= 39.677
  14:01:56 STEP 1694 H= 423:30 +CPU= 39.680
  14:02:48 STEP 1695 H= 423:45 +CPU= 39.658
  14:03:42 STEP 1696 H= 424:00 +CPU= 40.518
  14:04:35 STEP 1697 H= 424:15 +CPU= 39.908
  14:05:28 STEP 1698 H= 424:30 +CPU= 39.877
  14:06:22 STEP 1699 H= 424:45 +CPU= 39.937
  14:07:16 STEP 1700 H= 425:00 +CPU= 40.687
  14:08:11 STEP 1701 H= 425:15 +CPU= 39.891
  14:09:08 STEP 1702 H= 425:30 +CPU= 40.200
  14:10:05 STEP 1703 H= 425:45 +CPU= 40.126
  14:11:16 STEP 1704 H= 426:00 +CPU= 50.194
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001704
  14:12:14 STEP 1705 H= 426:15 +CPU= 40.213
  14:13:12 STEP 1706 H= 426:30 +CPU= 40.087
  14:14:09 STEP 1707 H= 426:45 +CPU= 40.079
  14:15:06 STEP 1708 H= 427:00 +CPU= 40.808
  14:16:02 STEP 1709 H= 427:15 +CPU= 39.955
  14:17:00 STEP 1710 H= 427:30 +CPU= 39.970
  14:18:00 STEP 1711 H= 427:45 +CPU= 40.142
  14:19:02 STEP 1712 H= 428:00 +CPU= 41.025
  14:20:03 STEP 1713 H= 428:15 +CPU= 40.279
  14:21:04 STEP 1714 H= 428:30 +CPU= 40.339
  14:22:05 STEP 1715 H= 428:45 +CPU= 40.280
  14:23:21 STEP 1716 H= 429:00 +CPU= 50.321
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001716
  14:24:22 STEP 1717 H= 429:15 +CPU= 39.980
  14:25:24 STEP 1718 H= 429:30 +CPU= 40.174
  14:26:24 STEP 1719 H= 429:45 +CPU= 40.093
  14:27:26 STEP 1720 H= 430:00 +CPU= 41.123
  14:28:27 STEP 1721 H= 430:15 +CPU= 40.110
  14:29:28 STEP 1722 H= 430:30 +CPU= 40.378
  14:30:29 STEP 1723 H= 430:45 +CPU= 40.306
  14:31:23 STEP 1724 H= 431:00 +CPU= 40.738
  14:32:16 STEP 1725 H= 431:15 +CPU= 39.874
  14:33:09 STEP 1726 H= 431:30 +CPU= 39.897
  14:34:02 STEP 1727 H= 431:45 +CPU= 39.885
  14:35:15 STEP 1728 H= 432:00 +CPU= 54.810
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4mo_2016092300_20_1018_12294327_0_r338955802_17.zip
Uploading the intermediate file: upload_file_17.zip
  14:36:10 STEP 1729 H= 432:15 +CPU= 40.236
  14:37:03 STEP 1730 H= 432:30 +CPU= 39.863
  14:37:56 STEP 1731 H= 432:45 +CPU= 39.902
  14:38:49 STEP 1732 H= 433:00 +CPU= 40.532
  14:39:42 STEP 1733 H= 433:15 +CPU= 39.790
  14:40:35 STEP 1734 H= 433:30 +CPU= 39.788
  14:41:27 STEP 1735 H= 433:45 +CPU= 39.704
  14:42:21 STEP 1736 H= 434:00 +CPU= 40.743
  14:43:11 STEP 1737 H= 434:15 +CPU= 39.557
  14:43:59 STEP 1738 H= 434:30 +CPU= 39.659
  14:44:48 STEP 1739 H= 434:45 +CPU= 39.779
  14:45:48 STEP 1740 H= 435:00 +CPU= 49.812
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001740
  14:46:37 STEP 1741 H= 435:15 +CPU= 39.855
  14:47:25 STEP 1742 H= 435:30 +CPU= 39.797
  14:48:14 STEP 1743 H= 435:45 +CPU= 39.866
  14:49:03 STEP 1744 H= 436:00 +CPU= 40.785
  14:49:51 STEP 1745 H= 436:15 +CPU= 39.553
  14:50:36 STEP 1746 H= 436:30 +CPU= 39.406
  14:51:21 STEP 1747 H= 436:45 +CPU= 39.335
  14:52:07 STEP 1748 H= 437:00 +CPU= 40.207
  14:52:51 STEP 1749 H= 437:15 +CPU= 39.206
  14:53:36 STEP 1750 H= 437:30 +CPU= 39.022
  14:54:21 STEP 1751 H= 437:45 +CPU= 39.410
  14:55:18 STEP 1752 H= 438:00 +CPU= 49.297
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001752
  14:56:02 STEP 1753 H= 438:15 +CPU= 39.220
  14:56:46 STEP 1754 H= 438:30 +CPU= 39.462
  14:57:29 STEP 1755 H= 438:45 +CPU= 39.030
  14:58:13 STEP 1756 H= 439:00 +CPU= 39.927
  14:58:56 STEP 1757 H= 439:15 +CPU= 39.186
  14:59:40 STEP 1758 H= 439:30 +CPU= 38.964
  15:00:23 STEP 1759 H= 439:45 +CPU= 38.897
  15:01:07 STEP 1760 H= 440:00 +CPU= 39.821
  15:01:51 STEP 1761 H= 440:15 +CPU= 39.187
  15:02:35 STEP 1762 H= 440:30 +CPU= 39.123
  15:03:18 STEP 1763 H= 440:45 +CPU= 39.192
  15:04:13 STEP 1764 H= 441:00 +CPU= 49.188
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001764
  15:04:56 STEP 1765 H= 441:15 +CPU= 39.121
  15:05:40 STEP 1766 H= 441:30 +CPU= 39.150
  15:06:23 STEP 1767 H= 441:45 +CPU= 39.139
  15:07:07 STEP 1768 H= 442:00 +CPU= 39.846
  15:07:50 STEP 1769 H= 442:15 +CPU= 39.177
  15:08:33 STEP 1770 H= 442:30 +CPU= 38.926
  15:09:17 STEP 1771 H= 442:45 +CPU= 39.207
  15:10:01 STEP 1772 H= 443:00 +CPU= 39.955
  15:10:44 STEP 1773 H= 443:15 +CPU= 39.093
  15:11:28 STEP 1774 H= 443:30 +CPU= 39.151
  15:12:11 STEP 1775 H= 443:45 +CPU= 39.143
  15:13:10 STEP 1776 H= 444:00 +CPU= 53.660
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001776
  15:13:54 STEP 1777 H= 444:15 +CPU= 39.178
  15:14:37 STEP 1778 H= 444:30 +CPU= 39.130
  15:15:20 STEP 1779 H= 444:45 +CPU= 38.944
  15:16:04 STEP 1780 H= 445:00 +CPU= 40.108
  15:16:47 STEP 1781 H= 445:15 +CPU= 39.034
  15:17:31 STEP 1782 H= 445:30 +CPU= 39.005
  15:18:14 STEP 1783 H= 445:45 +CPU= 39.257
  15:18:59 STEP 1784 H= 446:00 +CPU= 40.136
  15:19:42 STEP 1785 H= 446:15 +CPU= 38.950
  15:20:27 STEP 1786 H= 446:30 +CPU= 39.232
  15:21:12 STEP 1787 H= 446:45 +CPU= 39.321
  15:22:08 STEP 1788 H= 447:00 +CPU= 49.472
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001788
  15:22:53 STEP 1789 H= 447:15 +CPU= 39.421
  15:23:38 STEP 1790 H= 447:30 +CPU= 39.577
  15:24:22 STEP 1791 H= 447:45 +CPU= 39.370
  15:25:07 STEP 1792 H= 448:00 +CPU= 40.369
  15:25:50 STEP 1793 H= 448:15 +CPU= 39.260
  15:26:33 STEP 1794 H= 448:30 +CPU= 39.164
  15:27:16 STEP 1795 H= 448:45 +CPU= 38.916
  15:28:01 STEP 1796 H= 449:00 +CPU= 40.129
  15:28:45 STEP 1797 H= 449:15 +CPU= 39.413
  15:29:29 STEP 1798 H= 449:30 +CPU= 39.322
  15:30:13 STEP 1799 H= 449:45 +CPU= 39.235
  15:31:09 STEP 1800 H= 450:00 +CPU= 49.363
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001800
  15:31:53 STEP 1801 H= 450:15 +CPU= 39.255
  15:32:37 STEP 1802 H= 450:30 +CPU= 39.286
  15:33:21 STEP 1803 H= 450:45 +CPU= 39.625
  15:34:06 STEP 1804 H= 451:00 +CPU= 40.439
  15:34:51 STEP 1805 H= 451:15 +CPU= 39.600
  15:35:35 STEP 1806 H= 451:30 +CPU= 39.370
  15:36:19 STEP 1807 H= 451:45 +CPU= 39.367
  15:37:04 STEP 1808 H= 452:00 +CPU= 40.027
  15:37:48 STEP 1809 H= 452:15 +CPU= 39.495
  15:38:32 STEP 1810 H= 452:30 +CPU= 39.568
  15:39:18 STEP 1811 H= 452:45 +CPU= 39.556
  15:40:13 STEP 1812 H= 453:00 +CPU= 49.379
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001812
  15:40:56 STEP 1813 H= 453:15 +CPU= 39.438
  15:41:39 STEP 1814 H= 453:30 +CPU= 39.563
  15:42:23 STEP 1815 H= 453:45 +CPU= 39.428
  15:43:07 STEP 1816 H= 454:00 +CPU= 40.284
  15:43:51 STEP 1817 H= 454:15 +CPU= 39.128
  15:44:35 STEP 1818 H= 454:30 +CPU= 39.351
  15:45:18 STEP 1819 H= 454:45 +CPU= 39.332
  15:46:03 STEP 1820 H= 455:00 +CPU= 40.340
  15:46:47 STEP 1821 H= 455:15 +CPU= 39.355
  15:47:30 STEP 1822 H= 455:30 +CPU= 39.189
  15:48:12 STEP 1823 H= 455:45 +CPU= 39.086
  15:49:11 STEP 1824 H= 456:00 +CPU= 53.466
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4mo_2016092300_20_1018_12294327_0_r338955802_18.zip
Uploading the intermediate file: upload_file_18.zip
  15:49:53 STEP 1825 H= 456:15 +CPU= 38.892
  15:50:34 STEP 1826 H= 456:30 +CPU= 38.913
  15:51:16 STEP 1827 H= 456:45 +CPU= 39.039
  15:51:58 STEP 1828 H= 457:00 +CPU= 39.616
  15:52:39 STEP 1829 H= 457:15 +CPU= 38.887
  15:53:20 STEP 1830 H= 457:30 +CPU= 38.986
  15:54:02 STEP 1831 H= 457:45 +CPU= 39.026
  15:54:44 STEP 1832 H= 458:00 +CPU= 39.917
  15:55:25 STEP 1833 H= 458:15 +CPU= 38.839
  15:56:06 STEP 1834 H= 458:30 +CPU= 38.608
  15:56:47 STEP 1835 H= 458:45 +CPU= 38.679
  15:57:39 STEP 1836 H= 459:00 +CPU= 48.674
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001836
  15:58:19 STEP 1837 H= 459:15 +CPU= 38.481
  15:58:59 STEP 1838 H= 459:30 +CPU= 38.416
  15:59:40 STEP 1839 H= 459:45 +CPU= 38.482
  16:00:21 STEP 1840 H= 460:00 +CPU= 39.229
  16:01:01 STEP 1841 H= 460:15 +CPU= 38.570
  16:01:41 STEP 1842 H= 460:30 +CPU= 38.309
  16:02:21 STEP 1843 H= 460:45 +CPU= 38.445
  16:03:02 STEP 1844 H= 461:00 +CPU= 39.246
  16:03:42 STEP 1845 H= 461:15 +CPU= 38.215
  16:04:22 STEP 1846 H= 461:30 +CPU= 38.782
  16:05:02 STEP 1847 H= 461:45 +CPU= 38.121
  16:05:53 STEP 1848 H= 462:00 +CPU= 47.952
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001848
  16:06:33 STEP 1849 H= 462:15 +CPU= 38.390
  16:07:13 STEP 1850 H= 462:30 +CPU= 38.346
  16:07:53 STEP 1851 H= 462:45 +CPU= 38.466
  16:08:35 STEP 1852 H= 463:00 +CPU= 39.452
  16:09:15 STEP 1853 H= 463:15 +CPU= 38.799
  16:09:56 STEP 1854 H= 463:30 +CPU= 38.738
  16:10:36 STEP 1855 H= 463:45 +CPU= 38.611
  16:11:17 STEP 1856 H= 464:00 +CPU= 39.561
  16:11:57 STEP 1857 H= 464:15 +CPU= 38.570
  16:12:38 STEP 1858 H= 464:30 +CPU= 38.807
  16:13:19 STEP 1859 H= 464:45 +CPU= 38.841
  16:14:06 STEP 1860 H= 465:00 +CPU= 45.332
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001860
  16:14:40 STEP 1861 H= 465:15 +CPU= 32.520
  16:15:18 STEP 1862 H= 465:30 +CPU= 36.926
  16:16:00 STEP 1863 H= 465:45 +CPU= 38.908
  16:16:41 STEP 1864 H= 466:00 +CPU= 39.712
  16:17:22 STEP 1865 H= 466:15 +CPU= 38.701
  16:18:04 STEP 1866 H= 466:30 +CPU= 39.202
  16:18:45 STEP 1867 H= 466:45 +CPU= 39.219
  16:19:28 STEP 1868 H= 467:00 +CPU= 40.331
  16:20:11 STEP 1869 H= 467:15 +CPU= 39.299
  16:20:52 STEP 1870 H= 467:30 +CPU= 39.163
  16:21:34 STEP 1871 H= 467:45 +CPU= 38.973
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

 NSTEP =  1867 STEPO   0AAA00AAA
 16:18:45 STEP 1867 H= 466:45 +CPU= 39.219
 NSTEP =  1868 STEPO   0AAA00AAA
 16:19:28 STEP 1868 H= 467:00 +CPU= 40.331
 NSTEP =  1869 STEPO   0AAA00AAA
 16:20:11 STEP 1869 H= 467:15 +CPU= 39.299
ISTEP=    1870 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1870 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1870 STEPO   0AAA00AAA
 16:20:52 STEP 1870 H= 467:30 +CPU= 39.163
 NSTEP =  1871 STEPO   0AAA00AAA
 16:21:34 STEP 1871 H= 467:45 +CPU= 38.973
  DATE= 2016   10    9
ISTEP=    1872: Reset RSOLINC= 1361.4482 RIP0= 1361.4482
 SU_GHGCLIM, RCARDI,RCH4,RN2O,RCFC11,RCFC12,RNO2:   6.179466599480000E-004
  1.088035170184667E-006  4.990317592110001E-007  3.778213382954999E-009
  2.097581753513333E-009  7.940283000000001E-011
 suecaec NAERMACC =  1
 UPDT RAESHDU =  2000.0
 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT)     20160923
         810  9.999999999999998E-005
 NORMS AT NSTEP CNT4 1872
 SPECTRAL NORMS -   LOG(PREHYDS)    0.115261046673563E+02
 LEV     VORTICITY             DIVERGENCE           TEMPERATURE           KINETIC ENERGY    
 AVE 0.805406500494533E-05 0.102238634241067E-05 0.201314222864093E+03 0.321246844075367E+02
 GPNORM  HUMIDITY            AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.126331404177586E-02 0.999999999999999E-08 0.152406611320691E-01
 GPNORM  LIQUID WATER        AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.883202429575500E-06 0.000000000000000E+00 0.615694009694312E-03
 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.104897650141921E-06 0.000000000000000E+00 0.141172836464578E-03
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.120112996861934E-06 0.000000000000000E+00 0.408386688898022E-03
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.118865836282721E-07 0.000000000000000E+00 0.907776991824661E-04
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.155325505822867E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.881020517863567E-06 0.000000000000000E+00 0.595204939130035E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.104982906478952E-06 0.000000000000000E+00 0.143901411955237E-03
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.155274829349680E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.543276935535811E-07 0.000000000000000E+00 0.414891569643121E-03
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.139544780856050E-06 0.000000000000000E+00 0.441839069992872E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.949426927974136E-07 0.000000000000000E+00 0.247235217328296E-03
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.122800459015349E-07 0.000000000000000E+00 0.215373229891574E-03
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.446160518232105E-01 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1872 STEPO   F00000000
 CREATE RESTART FILE ./srf00191200.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00191200.0001 MODE=w
 IO-STREAM CLOSED - ./srf00191200.0001
 MPI-TASK:   1 - ************ BYTES IN     46 RECORDS TRANSFERRED IN    3.5985 SECONDS******** Mbytes/s, TOTAL TIME=    4.4136(81.5%)
 CREATE RESTART CONTROL FILE ./rcf                                                                                                                   
 WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00190000.0001
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1872 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001872 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001872
 MPI-TASK:   1 -    316572308 BYTES IN      6 RECORDS TRANSFERRED IN    0.0005 SECONDS633144.6 Mbytes/s, TOTAL TIME=    0.0146( 3.4%)
 NSTEP =  1872 STEPO   0AA000000
------------------------------------------------
CNT0 not found; string returned was: 'DYNFPOS'
>>> Printing last 8 lines from file: ifs.stat

 16:18:45 0AAA00AAA STEPO     1868    39.219   39.219   41.861    219:19    267:52 0.10179816393102E-05       2GB       0MB
 16:19:28 0AAA00AAA STEPO     1869    40.331   40.331   42.929    219:59    268:34 0.10179816393102E-05       2GB       0MB
 16:20:11 0AAA00AAA STEPO     1870    39.299   39.299   42.225    220:39    269:17 0.10179816393102E-05       2GB       0MB
 16:20:52 0AAA00AAA STEPO     1871    39.163   39.163   41.762    221:18    269:58 0.10179816393102E-05       2GB       0MB
 16:21:34 F00000000 STEPO     1872    39.183   39.183   41.682    221:57    270:40 0.10223863424107E-05       2GB       0MB
 16:21:39 0AA000000 STEPO     1872     4.607    4.607    4.941    222:02    270:45 0.10223863424107E-05       2GB       0MB
 16:21:42 FULLPOS-B DYNFPOS   1872     2.300    2.300    2.510    222:04    270:48 0.10223863424107E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1872",
 CTIME   = "00191200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1860, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/26/progress_file_12294327.xml

<running_values>
  <last_cpu_time>55289.670000</last_cpu_time>
  <upload_file_number>19</upload_file_number>
  <last_iter>1872</last_iter>
  <last_upload>1641600</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a4mo
(argv4) batchid: 1018
(argv5) wuid: 12294327
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/26
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/26/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/26/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5553104be8d4bcdf7e8dd6aaef954957 to: /var/lib/boinc-client/slots/26/oifs_43r3_bl_a4mo_2016092300_20_1018_12294327.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/26/oifs_43r3_bl_a4mo_2016092300_20_1018_12294327.zip
ic_ancil_file: ic_ancil_12294327
ifsdata_file: ifsdata_12294327
climate_data_file: clim_data_12294327
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/26/ic_ancil_12294327.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/26/ic_ancil_12294327.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_50acb0a6225958bb7b61f12e13d82ba1 to: /var/lib/boinc-client/slots/26/ifsdata/ifsdata_12294327.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/26/ifsdata/ifsdata_12294327.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/26/159l_2/clim_data_12294327.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/26/159l_2/clim_data_12294327.zip
Checking for progress XML file: /var/lib/boinc-client/slots/26/progress_file_12294327.xml
Opened progress file ok : /var/lib/boinc-client/slots/26/progress_file_12294327.xml
-- Model is restarting --
Adjusting last_iter, 1872, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/26/progress_file_12294327.xml
last_cpu_time: 55289
upload_file_number: 19
last_iter: 1873
last_upload: 1641600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a4mo_2016092300_20_1018_12294327_0_r338955802
The child process has been launched with process id: 1636372
Executing the command: /var/lib/boinc-client/slots/26/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.001] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.002] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:1636372:1636372] [20240622:175833:1719100713:0.002] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:59:18 STEP 1872 H= 468:00 +CPU= 39.206
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001872
  17:59:57 STEP 1873 H= 468:15 +CPU= 37.460
  18:00:35 STEP 1874 H= 468:30 +CPU= 37.794
  18:01:14 STEP 1875 H= 468:45 +CPU= 38.232
  18:01:54 STEP 1876 H= 469:00 +CPU= 38.603
  18:02:32 STEP 1877 H= 469:15 +CPU= 37.639
  18:03:11 STEP 1878 H= 469:30 +CPU= 37.448
  18:03:49 STEP 1879 H= 469:45 +CPU= 37.883
  18:04:29 STEP 1880 H= 470:00 +CPU= 38.649
  18:05:08 STEP 1881 H= 470:15 +CPU= 37.833
  18:05:47 STEP 1882 H= 470:30 +CPU= 37.677
  18:06:25 STEP 1883 H= 470:45 +CPU= 37.691
  18:07:14 STEP 1884 H= 471:00 +CPU= 47.918
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001884
  18:07:53 STEP 1885 H= 471:15 +CPU= 38.052
  18:08:32 STEP 1886 H= 471:30 +CPU= 37.680
  18:09:11 STEP 1887 H= 471:45 +CPU= 37.825
  18:09:50 STEP 1888 H= 472:00 +CPU= 38.518
  18:10:29 STEP 1889 H= 472:15 +CPU= 37.742
  18:11:08 STEP 1890 H= 472:30 +CPU= 37.947
  18:11:47 STEP 1891 H= 472:45 +CPU= 38.270
  18:12:28 STEP 1892 H= 473:00 +CPU= 39.018
  18:13:07 STEP 1893 H= 473:15 +CPU= 38.474
  18:13:46 STEP 1894 H= 473:30 +CPU= 37.851
  18:14:26 STEP 1895 H= 473:45 +CPU= 38.432
  18:15:16 STEP 1896 H= 474:00 +CPU= 48.537
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001896
  18:15:55 STEP 1897 H= 474:15 +CPU= 38.743
  18:16:35 STEP 1898 H= 474:30 +CPU= 38.461
  18:17:15 STEP 1899 H= 474:45 +CPU= 38.545
  18:17:56 STEP 1900 H= 475:00 +CPU= 39.497
  18:18:36 STEP 1901 H= 475:15 +CPU= 38.527
  18:19:16 STEP 1902 H= 475:30 +CPU= 38.714
  18:19:56 STEP 1903 H= 475:45 +CPU= 38.956
  18:20:37 STEP 1904 H= 476:00 +CPU= 39.818
  18:21:17 STEP 1905 H= 476:15 +CPU= 38.466
  18:21:57 STEP 1906 H= 476:30 +CPU= 38.495
  18:22:37 STEP 1907 H= 476:45 +CPU= 38.633
  18:23:27 STEP 1908 H= 477:00 +CPU= 48.670
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001908
  18:24:07 STEP 1909 H= 477:15 +CPU= 38.039
  18:24:46 STEP 1910 H= 477:30 +CPU= 38.158
  18:25:25 STEP 1911 H= 477:45 +CPU= 38.511
  18:26:06 STEP 1912 H= 478:00 +CPU= 39.333
  18:26:45 STEP 1913 H= 478:15 +CPU= 38.604
  18:27:25 STEP 1914 H= 478:30 +CPU= 38.650
  18:28:05 STEP 1915 H= 478:45 +CPU= 38.776
  18:28:46 STEP 1916 H= 479:00 +CPU= 39.336
  18:29:26 STEP 1917 H= 479:15 +CPU= 38.674
  18:30:06 STEP 1918 H= 479:30 +CPU= 38.544
  18:30:45 STEP 1919 H= 479:45 +CPU= 38.665
  18:31:37 STEP 1920 H= 480:00 +CPU= 50.223
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.6291122720E-21      I      -0.6332160354E-06      I
ICloud fractionI      0.0000000000E+00      I       0.3774195541E-01      I
I  Relat. Hum. I      0.2009119605E+00      I       0.3012618368E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.3338790873E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    312590996 BYTES IN      7 RECORDS TRANSFERRED IN    0.0006 SECONDS520985.0 Mbytes/s, TOTAL TIME=    0.0161( 3.7%)
 18:31:37 STEP 1920 H= 480:00 +CPU= 50.223
 END CNT3
 NSTEP =  1920 CNT0   000000000  36.883  36.883  37.810   0.975               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         1430/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     28    0       46      1044068     1430 ************     5.9     2.3     3.6     3.1     7.6
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 1984053.0 1984053.0    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 1979178.4 1979178.4     99.75      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                   53    5899.2    5899.2     15.76      0.00
   9 SPCM     - SPECTRAL COMP.                      48    1104.5    1104.5      2.67      0.00
  10 SCAN2M - PHYSICS                               49   20025.6   20025.6     49.46      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  4    1185.8    1185.8      0.24      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  3      55.6      55.6      0.01      0.00
  17 GRIDFPOS IN CNT4                                4      38.1      38.1      0.01      0.00
  27 RESTART FILES - WRITING                         1    4743.1    4743.1      0.24      0.00
  28 RESTART FILES - READING                         1    4045.1    4045.1      0.20      0.00
  29 SU4FPOS IN CNT4                                 4       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                 4    8941.1    8941.1      1.80      0.00
  31 POSDDH IN STEPO                                 1       1.3       1.3      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     49    8378.3    8378.3     20.69      0.00
  39 SU0YOMB                                         1     549.2     549.2      0.03      0.00
  51 SCAN2M   - SL COMM. PART 1                     49      44.1      44.1      0.11      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               48     186.6     186.6      0.45      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               48      71.1      71.1      0.17      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  3       0.3       0.3      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      108     663.4     663.4      3.61      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       100     379.8     379.8      1.91      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        100     137.2     137.2      0.69      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       108     217.1     217.1      1.18      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      58.2      58.2      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            108      71.5      71.5      0.39      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            100      36.7      36.7      0.18      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            108     102.0     102.0      0.56      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            100      30.3      30.3      0.15      0.00
 400 GSTATS                                       6667       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                  6149       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     1984.1 CPU TIME    1928.2 VECTOR TIME     1928.2


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      873.1
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/26/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/slots/26/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/26/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294327/ICMUAh7zg+001896
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4mo_2016092300_20_1018_12294327_0_r338955802_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
18:35:12 (1636285): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
23 Jun 2024 00:33:29 1551031 22447944 oifs_43r3_bl_a4mo_2016092300_20_1018_12294327_0 1,727,100 57,217 0.0331
20 Jun 2024 22:05:05 1551031 22447944 oifs_43r3_bl_a4mo_2016092300_20_1018_12294327_0 864,000 24,568 0.0284


©2024 climateprediction.net