climateprediction.net home page
Task 22291632

Task 22291632

Name oifs_43r3_ps_0468_2019050100_123_988_12205112_0
Workunit 12205112
Created 24 Dec 2022, 23:21:38 UTC
Sent 15 Jan 2023, 19:30:51 UTC
Report deadline 14 Feb 2023, 19:30:51 UTC
Received 18 Jan 2023, 14:51:58 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1532332
Run time 20 hours 38 min 40 sec
CPU time 20 hours 26 min 58 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 7.08 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.24 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,695.00 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
17:155017:1673970617:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:52:23 STEP  552 H= 552:00 +CPU=116.239
  15:53:30 STEP  553 H= 553:00 +CPU= 65.952
  15:54:32 STEP  554 H= 554:00 +CPU= 61.534
  15:56:05 STEP  555 H= 555:00 +CPU= 92.910
  15:57:08 STEP  556 H= 556:00 +CPU= 62.089
  15:58:07 STEP  557 H= 557:00 +CPU= 58.706
  15:59:34 STEP  558 H= 558:00 +CPU= 86.936
  16:00:51 STEP  559 H= 559:00 +CPU= 75.846
  16:01:57 STEP  560 H= 560:00 +CPU= 66.311
  16:03:38 STEP  561 H= 561:00 +CPU=100.425
  16:05:01 STEP  562 H= 562:00 +CPU= 82.483
  16:06:12 STEP  563 H= 563:00 +CPU= 70.447
  16:07:50 STEP  564 H= 564:00 +CPU= 97.546
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000564
  16:08:53 STEP  565 H= 565:00 +CPU= 62.526
  16:10:03 STEP  566 H= 566:00 +CPU= 68.951
  16:11:31 STEP  567 H= 567:00 +CPU= 87.424
  16:12:42 STEP  568 H= 568:00 +CPU= 70.467
  16:13:41 STEP  569 H= 569:00 +CPU= 58.536
  16:15:10 STEP  570 H= 570:00 +CPU= 88.605
  16:16:08 STEP  571 H= 571:00 +CPU= 57.818
  16:17:07 STEP  572 H= 572:00 +CPU= 58.360
  16:18:31 STEP  573 H= 573:00 +CPU= 83.604
  16:19:42 STEP  574 H= 574:00 +CPU= 69.797
  16:20:42 STEP  575 H= 575:00 +CPU= 59.827
  16:22:29 STEP  576 H= 576:00 +CPU=106.409
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000564
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136_23.zip
Uploading the intermediate file: upload_file_23.zip
  16:23:30 STEP  577 H= 577:00 +CPU= 60.588
  16:24:33 STEP  578 H= 578:00 +CPU= 62.015
  16:25:58 STEP  579 H= 579:00 +CPU= 84.548
  16:27:07 STEP  580 H= 580:00 +CPU= 68.431
  16:28:12 STEP  581 H= 581:00 +CPU= 63.774
  16:30:08 STEP  582 H= 582:00 +CPU=115.846
  16:31:19 STEP  583 H= 583:00 +CPU= 69.616
  16:32:28 STEP  584 H= 584:00 +CPU= 68.640
  16:34:15 STEP  585 H= 585:00 +CPU=106.860
  16:35:26 STEP  586 H= 586:00 +CPU= 69.783
  16:36:26 STEP  587 H= 587:00 +CPU= 59.376
  16:38:18 STEP  588 H= 588:00 +CPU=111.393
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000588
  16:39:20 STEP  589 H= 589:00 +CPU= 61.520
  16:40:28 STEP  590 H= 590:00 +CPU= 67.065
  16:42:03 STEP  591 H= 591:00 +CPU= 94.584
  16:43:00 STEP  592 H= 592:00 +CPU= 56.646
  16:44:14 STEP  593 H= 593:00 +CPU= 73.600
  16:46:09 STEP  594 H= 594:00 +CPU=113.335
  16:47:13 STEP  595 H= 595:00 +CPU= 63.587
  16:48:18 STEP  596 H= 596:00 +CPU= 65.349
  16:49:49 STEP  597 H= 597:00 +CPU= 90.772
  16:50:56 STEP  598 H= 598:00 +CPU= 66.244
  16:52:11 STEP  599 H= 599:00 +CPU= 74.825
  16:54:02 STEP  600 H= 600:00 +CPU=109.578
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000588
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136_24.zip
Uploading the intermediate file: upload_file_24.zip
  16:55:03 STEP  601 H= 601:00 +CPU= 60.637
  16:56:11 STEP  602 H= 602:00 +CPU= 67.178
  16:57:57 STEP  603 H= 603:00 +CPU=104.402
  16:59:04 STEP  604 H= 604:00 +CPU= 66.377
  17:00:15 STEP  605 H= 605:00 +CPU= 67.811
  17:01:47 STEP  606 H= 606:00 +CPU= 91.778
  17:02:45 STEP  607 H= 607:00 +CPU= 57.520
  17:03:54 STEP  608 H= 608:00 +CPU= 67.955
  17:05:27 STEP  609 H= 609:00 +CPU= 92.292
  17:06:24 STEP  610 H= 610:00 +CPU= 56.346
  17:07:27 STEP  611 H= 611:00 +CPU= 62.649
  17:09:06 STEP  612 H= 612:00 +CPU= 98.542
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000612
  17:10:13 STEP  613 H= 613:00 +CPU= 66.756
  17:11:22 STEP  614 H= 614:00 +CPU= 68.928
  17:13:00 STEP  615 H= 615:00 +CPU= 96.799
  17:14:11 STEP  616 H= 616:00 +CPU= 70.432
  17:15:31 STEP  617 H= 617:00 +CPU= 79.529
  17:16:57 STEP  618 H= 618:00 +CPU= 85.709
  17:18:10 STEP  619 H= 619:00 +CPU= 72.389
  17:19:15 STEP  620 H= 620:00 +CPU= 65.054
  17:20:47 STEP  621 H= 621:00 +CPU= 91.518
  17:21:51 STEP  622 H= 622:00 +CPU= 63.186
  17:23:09 STEP  623 H= 623:00 +CPU= 77.953
  17:25:08 STEP  624 H= 624:00 +CPU=118.262
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000612
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136_25.zip
Uploading the intermediate file: upload_file_25.zip
  17:26:06 STEP  625 H= 625:00 +CPU= 56.823
  17:27:38 STEP  626 H= 626:00 +CPU= 92.146
  17:29:21 STEP  627 H= 627:00 +CPU=102.350
  17:30:24 STEP  628 H= 628:00 +CPU= 62.286
  17:31:32 STEP  629 H= 629:00 +CPU= 67.181
  17:33:08 STEP  630 H= 630:00 +CPU= 95.631
  17:34:08 STEP  631 H= 631:00 +CPU= 58.732
  17:35:19 STEP  632 H= 632:00 +CPU= 70.567
  17:36:46 STEP  633 H= 633:00 +CPU= 87.093
  17:37:51 STEP  634 H= 634:00 +CPU= 64.182
  17:38:54 STEP  635 H= 635:00 +CPU= 62.879
  17:40:34 STEP  636 H= 636:00 +CPU= 99.801
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000636
  17:41:37 STEP  637 H= 637:00 +CPU= 62.913
  17:42:41 STEP  638 H= 638:00 +CPU= 63.521
  17:44:11 STEP  639 H= 639:00 +CPU= 88.668
  17:45:21 STEP  640 H= 640:00 +CPU= 70.147
  17:46:22 STEP  641 H= 641:00 +CPU= 60.126
  17:47:48 STEP  642 H= 642:00 +CPU= 86.010
  17:49:02 STEP  643 H= 643:00 +CPU= 73.766
  17:50:15 STEP  644 H= 644:00 +CPU= 72.452
  17:51:48 STEP  645 H= 645:00 +CPU= 91.979
  17:52:48 STEP  646 H= 646:00 +CPU= 59.295
  17:53:47 STEP  647 H= 647:00 +CPU= 59.289
  17:55:44 STEP  648 H= 648:00 +CPU=115.511
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136_26.zip
Uploading the intermediate file: upload_file_26.zip
  17:56:47 STEP  649 H= 649:00 +CPU= 62.639
  17:57:58 STEP  650 H= 650:00 +CPU= 70.440
  17:59:27 STEP  651 H= 651:00 +CPU= 88.350
  18:00:29 STEP  652 H= 652:00 +CPU= 61.057
  18:01:38 STEP  653 H= 653:00 +CPU= 68.675
  18:03:10 STEP  654 H= 654:00 +CPU= 90.923
  18:04:09 STEP  655 H= 655:00 +CPU= 58.809
  18:05:15 STEP  656 H= 656:00 +CPU= 65.515
  18:06:52 STEP  657 H= 657:00 +CPU= 96.485
  18:07:59 STEP  658 H= 658:00 +CPU= 65.707
  18:09:03 STEP  659 H= 659:00 +CPU= 63.717
  18:10:43 STEP  660 H= 660:00 +CPU= 97.211
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000660
  18:12:00 STEP  661 H= 661:00 +CPU= 75.575
  18:13:08 STEP  662 H= 662:00 +CPU= 67.168
  18:14:43 STEP  663 H= 663:00 +CPU= 94.233
  18:15:44 STEP  664 H= 664:00 +CPU= 60.531
  18:16:46 STEP  665 H= 665:00 +CPU= 60.835
  18:18:19 STEP  666 H= 666:00 +CPU= 91.708
  18:19:18 STEP  667 H= 667:00 +CPU= 58.725
  18:20:24 STEP  668 H= 668:00 +CPU= 64.957
  18:22:08 STEP  669 H= 669:00 +CPU=103.991
  18:23:09 STEP  670 H= 670:00 +CPU= 60.456
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0468
(argv4) batchid: 988
(argv5) wuid: 12205112
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/20
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_de683492d810da57ceaeaf2351e191f0 to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0468_2019050100_123_988_12205112.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0468_2019050100_123_988_12205112.zip
ic_ancil_file: ic_ancil_12205112
ifsdata_file: ifsdata_12205112
climate_data_file: clim_data_12205112
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f6b3c8f344361038b9791ba0e863d962 to: /var/lib/boinc-client/slots/20/ic_ancil_12205112.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/20/ic_ancil_12205112.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e61ab47a9b0e931e54e8f86bc332320d to: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12205112.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12205112.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/20/159l_2/clim_data_12205112.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/20/159l_2/clim_data_12205112.zip
Checking for progress XML file: /var/lib/boinc-client/slots/20/progress_file_12205112.xml
Opened progress file ok : /var/lib/boinc-client/slots/20/progress_file_12205112.xml
-- Model is restarting --
Adjusting last_iter, 671, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/20/progress_file_12205112.xml
last_cpu_time: 55766
upload_file_number: 27
last_iter: 647
last_upload: 2332800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136
The child process has been launched with process id: 1161991
Executing the command: /var/lib/boinc-client/slots/20/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1161991:1161991] [20230117:204724:1673988444:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:49:01 STEP  648 H= 648:00 +CPU= 88.059
  20:50:10 STEP  649 H= 649:00 +CPU= 68.446
  20:51:17 STEP  650 H= 650:00 +CPU= 67.240
  20:52:59 STEP  651 H= 651:00 +CPU=100.706
  20:53:58 STEP  652 H= 652:00 +CPU= 59.138
  20:54:59 STEP  653 H= 653:00 +CPU= 60.724
  20:56:28 STEP  654 H= 654:00 +CPU= 87.798
  20:57:27 STEP  655 H= 655:00 +CPU= 57.400
  20:58:38 STEP  656 H= 656:00 +CPU= 70.701
  21:00:14 STEP  657 H= 657:00 +CPU= 95.406
  21:01:07 STEP  658 H= 658:00 +CPU= 52.609
  21:01:59 STEP  659 H= 659:00 +CPU= 52.033
  21:03:28 STEP  660 H= 660:00 +CPU= 88.111
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000660
  21:04:35 STEP  661 H= 661:00 +CPU= 66.875
  21:05:37 STEP  662 H= 662:00 +CPU= 61.237
  21:07:43 STEP  663 H= 663:00 +CPU=125.288
  21:08:53 STEP  664 H= 664:00 +CPU= 68.955
  21:09:53 STEP  665 H= 665:00 +CPU= 60.330
  21:11:14 STEP  666 H= 666:00 +CPU= 79.370
  21:12:12 STEP  667 H= 667:00 +CPU= 57.738
  21:13:17 STEP  668 H= 668:00 +CPU= 64.540
  21:14:59 STEP  669 H= 669:00 +CPU=101.268
  21:16:15 STEP  670 H= 670:00 +CPU= 75.388
  21:17:13 STEP  671 H= 671:00 +CPU= 57.476
  21:19:01 STEP  672 H= 672:00 +CPU=107.296
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136_27.zip
Uploading the intermediate file: upload_file_27.zip
  21:19:56 STEP  673 H= 673:00 +CPU= 54.183
  21:20:58 STEP  674 H= 674:00 +CPU= 61.855
  21:22:29 STEP  675 H= 675:00 +CPU= 89.349
  21:23:36 STEP  676 H= 676:00 +CPU= 66.883
  21:24:34 STEP  677 H= 677:00 +CPU= 58.312
  21:26:11 STEP  678 H= 678:00 +CPU= 96.205
  21:27:25 STEP  679 H= 679:00 +CPU= 72.835
  21:28:31 STEP  680 H= 680:00 +CPU= 66.556
  21:30:14 STEP  681 H= 681:00 +CPU=101.989
  21:31:28 STEP  682 H= 682:00 +CPU= 73.294
  21:32:29 STEP  683 H= 683:00 +CPU= 60.582
  21:34:18 STEP  684 H= 684:00 +CPU=108.189
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000684
  21:35:23 STEP  685 H= 685:00 +CPU= 64.089
  21:36:24 STEP  686 H= 686:00 +CPU= 61.129
  21:37:53 STEP  687 H= 687:00 +CPU= 88.256
  21:38:51 STEP  688 H= 688:00 +CPU= 57.187
  21:39:57 STEP  689 H= 689:00 +CPU= 66.488
  21:41:30 STEP  690 H= 690:00 +CPU= 92.386
  21:42:53 STEP  691 H= 691:00 +CPU= 82.664
  21:43:53 STEP  692 H= 692:00 +CPU= 59.824
  21:45:33 STEP  693 H= 693:00 +CPU= 99.129
  21:46:37 STEP  694 H= 694:00 +CPU= 63.806
  21:47:49 STEP  695 H= 695:00 +CPU= 71.577
  21:49:34 STEP  696 H= 696:00 +CPU=104.214
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136_28.zip
Uploading the intermediate file: upload_file_28.zip
  21:50:34 STEP  697 H= 697:00 +CPU= 59.363
  21:51:34 STEP  698 H= 698:00 +CPU= 59.472
  21:53:09 STEP  699 H= 699:00 +CPU= 94.978
  21:54:21 STEP  700 H= 700:00 +CPU= 71.663
  21:55:26 STEP  701 H= 701:00 +CPU= 64.176
  21:57:00 STEP  702 H= 702:00 +CPU= 93.310
  21:57:56 STEP  703 H= 703:00 +CPU= 56.113
  21:58:58 STEP  704 H= 704:00 +CPU= 61.452
  22:00:35 STEP  705 H= 705:00 +CPU= 96.216
  22:01:55 STEP  706 H= 706:00 +CPU= 79.719
  22:03:00 STEP  707 H= 707:00 +CPU= 64.799
  22:04:51 STEP  708 H= 708:00 +CPU=110.789
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000708
  22:05:49 STEP  709 H= 709:00 +CPU= 57.159
  22:07:03 STEP  710 H= 710:00 +CPU= 73.929
  22:08:45 STEP  711 H= 711:00 +CPU=100.762
  22:10:02 STEP  712 H= 712:00 +CPU= 77.313
  22:10:58 STEP  713 H= 713:00 +CPU= 54.716
  22:12:32 STEP  714 H= 714:00 +CPU= 93.033
  22:13:34 STEP  715 H= 715:00 +CPU= 62.001
  22:14:35 STEP  716 H= 716:00 +CPU= 60.362
  22:15:57 STEP  717 H= 717:00 +CPU= 81.083
  22:16:58 STEP  718 H= 718:00 +CPU= 60.794
  22:17:58 STEP  719 H= 719:00 +CPU= 60.161
  22:19:42 STEP  720 H= 720:00 +CPU=102.641
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136_29.zip
Uploading the intermediate file: upload_file_29.zip
  22:20:37 STEP  721 H= 721:00 +CPU= 54.056
  22:21:43 STEP  722 H= 722:00 +CPU= 65.524
  22:23:07 STEP  723 H= 723:00 +CPU= 83.628
  22:24:23 STEP  724 H= 724:00 +CPU= 74.593
  22:25:22 STEP  725 H= 725:00 +CPU= 58.583
  22:26:56 STEP  726 H= 726:00 +CPU= 93.722
  22:27:55 STEP  727 H= 727:00 +CPU= 58.770
  22:29:02 STEP  728 H= 728:00 +CPU= 65.894
  22:30:27 STEP  729 H= 729:00 +CPU= 84.724
  22:31:30 STEP  730 H= 730:00 +CPU= 63.187
  22:32:31 STEP  731 H= 731:00 +CPU= 59.899
  22:34:04 STEP  732 H= 732:00 +CPU= 92.598
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000732
  22:35:02 STEP  733 H= 733:00 +CPU= 57.368
  22:36:07 STEP  734 H= 734:00 +CPU= 64.538
  22:37:44 STEP  735 H= 735:00 +CPU= 96.203
  22:38:45 STEP  736 H= 736:00 +CPU= 59.765
  22:39:41 STEP  737 H= 737:00 +CPU= 55.974
  22:41:10 STEP  738 H= 738:00 +CPU= 88.937
  22:42:08 STEP  739 H= 739:00 +CPU= 57.549
  22:43:09 STEP  740 H= 740:00 +CPU= 60.785
  22:44:39 STEP  741 H= 741:00 +CPU= 88.697
  22:45:39 STEP  742 H= 742:00 +CPU= 59.656
  22:46:42 STEP  743 H= 743:00 +CPU= 62.512
  22:48:33 STEP  744 H= 744:00 +CPU=110.621
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000732
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136_30.zip
Uploading the intermediate file: upload_file_30.zip
  22:49:36 STEP  745 H= 745:00 +CPU= 61.951
  22:50:40 STEP  746 H= 746:00 +CPU= 63.670
  22:52:11 STEP  747 H= 747:00 +CPU= 89.870
  22:53:14 STEP  748 H= 748:00 +CPU= 62.990
  22:54:08 STEP  749 H= 749:00 +CPU= 53.435
  22:55:47 STEP  750 H= 750:00 +CPU= 98.712
  22:57:11 STEP  751 H= 751:00 +CPU= 83.917
  22:58:16 STEP  752 H= 752:00 +CPU= 63.872
  22:59:56 STEP  753 H= 753:00 +CPU= 99.678
  23:01:06 STEP  754 H= 754:00 +CPU= 69.801
  23:02:01 STEP  755 H= 755:00 +CPU= 54.980
  23:03:41 STEP  756 H= 756:00 +CPU= 98.650
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000756
  23:04:48 STEP  757 H= 757:00 +CPU= 66.450
  23:05:51 STEP  758 H= 758:00 +CPU= 62.775
  23:07:24 STEP  759 H= 759:00 +CPU= 91.663
  23:08:35 STEP  760 H= 760:00 +CPU= 70.437
  23:09:48 STEP  761 H= 761:00 +CPU= 71.933
  23:11:34 STEP  762 H= 762:00 +CPU=105.933
  23:12:39 STEP  763 H= 763:00 +CPU= 64.211
  23:13:41 STEP  764 H= 764:00 +CPU= 61.733
  23:15:14 STEP  765 H= 765:00 +CPU= 92.133
  23:16:25 STEP  766 H= 766:00 +CPU= 70.445
  23:17:21 STEP  767 H= 767:00 +CPU= 55.107
  23:19:08 STEP  768 H= 768:00 +CPU=105.708
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136_31.zip
Uploading the intermediate file: upload_file_31.zip
  23:20:11 STEP  769 H= 769:00 +CPU= 62.260
  23:21:13 STEP  770 H= 770:00 +CPU= 61.649
  23:22:40 STEP  771 H= 771:00 +CPU= 86.654
  23:23:45 STEP  772 H= 772:00 +CPU= 64.144
  23:24:44 STEP  773 H= 773:00 +CPU= 58.163
  23:26:09 STEP  774 H= 774:00 +CPU= 84.322
  23:27:07 STEP  775 H= 775:00 +CPU= 58.030
  23:28:15 STEP  776 H= 776:00 +CPU= 68.123
  23:29:42 STEP  777 H= 777:00 +CPU= 85.814
  23:31:02 STEP  778 H= 778:00 +CPU= 79.444
  23:32:30 STEP  779 H= 779:00 +CPU= 87.304
  23:34:20 STEP  780 H= 780:00 +CPU=109.255
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000780
  23:35:27 STEP  781 H= 781:00 +CPU= 65.714
  23:36:34 STEP  782 H= 782:00 +CPU= 66.157
  23:38:05 STEP  783 H= 783:00 +CPU= 90.663
  23:39:10 STEP  784 H= 784:00 +CPU= 63.809
  23:40:12 STEP  785 H= 785:00 +CPU= 61.776
  23:41:45 STEP  786 H= 786:00 +CPU= 92.641
  23:42:45 STEP  787 H= 787:00 +CPU= 59.397
  23:43:46 STEP  788 H= 788:00 +CPU= 60.188
  23:45:16 STEP  789 H= 789:00 +CPU= 89.071
  23:46:17 STEP  790 H= 790:00 +CPU= 60.252
  23:47:31 STEP  791 H= 791:00 +CPU= 73.681
  23:49:14 STEP  792 H= 792:00 +CPU=102.563
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136_32.zip
Uploading the intermediate file: upload_file_32.zip
  23:50:20 STEP  793 H= 793:00 +CPU= 65.810
  23:51:33 STEP  794 H= 794:00 +CPU= 71.747
  23:53:02 STEP  795 H= 795:00 +CPU= 87.746
  23:54:08 STEP  796 H= 796:00 +CPU= 65.706
  23:55:10 STEP  797 H= 797:00 +CPU= 61.773
  23:56:41 STEP  798 H= 798:00 +CPU= 90.295
  23:57:40 STEP  799 H= 799:00 +CPU= 58.179
  23:58:45 STEP  800 H= 800:00 +CPU= 65.113
  00:00:17 STEP  801 H= 801:00 +CPU= 90.856
  00:01:24 STEP  802 H= 802:00 +CPU= 66.139
  00:02:42 STEP  803 H= 803:00 +CPU= 77.275
  00:04:23 STEP  804 H= 804:00 +CPU=100.828
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000804
  00:05:24 STEP  805 H= 805:00 +CPU= 60.675
  00:06:22 STEP  806 H= 806:00 +CPU= 57.328
  00:07:53 STEP  807 H= 807:00 +CPU= 90.437
  00:08:54 STEP  808 H= 808:00 +CPU= 59.976
  00:10:09 STEP  809 H= 809:00 +CPU= 74.247
  00:11:49 STEP  810 H= 810:00 +CPU= 99.809
  00:12:50 STEP  811 H= 811:00 +CPU= 59.730
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0468
(argv4) batchid: 988
(argv5) wuid: 12205112
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/20
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_de683492d810da57ceaeaf2351e191f0 to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0468_2019050100_123_988_12205112.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0468_2019050100_123_988_12205112.zip
ic_ancil_file: ic_ancil_12205112
ifsdata_file: ifsdata_12205112
climate_data_file: clim_data_12205112
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f6b3c8f344361038b9791ba0e863d962 to: /var/lib/boinc-client/slots/20/ic_ancil_12205112.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/20/ic_ancil_12205112.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e61ab47a9b0e931e54e8f86bc332320d to: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12205112.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12205112.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/20/159l_2/clim_data_12205112.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/20/159l_2/clim_data_12205112.zip
Checking for progress XML file: /var/lib/boinc-client/slots/20/progress_file_12205112.xml
Opened progress file ok : /var/lib/boinc-client/slots/20/progress_file_12205112.xml
-- Model is restarting --
Adjusting last_iter, 812, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/20/progress_file_12205112.xml
last_cpu_time: 68042
upload_file_number: 33
last_iter: 791
last_upload: 2851200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136
The child process has been launched with process id: 1165116
Executing the command: /var/lib/boinc-client/slots/20/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1165116:1165116] [20230118:011845:1674004725:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:20:18 STEP  792 H= 792:00 +CPU= 85.697
  01:21:20 STEP  793 H= 793:00 +CPU= 61.330
  01:22:16 STEP  794 H= 794:00 +CPU= 55.945
  01:23:46 STEP  795 H= 795:00 +CPU= 88.909
  01:24:44 STEP  796 H= 796:00 +CPU= 57.225
  01:25:42 STEP  797 H= 797:00 +CPU= 58.076
  01:27:01 STEP  798 H= 798:00 +CPU= 77.938
  01:28:02 STEP  799 H= 799:00 +CPU= 60.723
  01:29:13 STEP  800 H= 800:00 +CPU= 70.306
  01:30:38 STEP  801 H= 801:00 +CPU= 84.566
  01:31:40 STEP  802 H= 802:00 +CPU= 62.419
  01:32:35 STEP  803 H= 803:00 +CPU= 54.848
  01:34:09 STEP  804 H= 804:00 +CPU= 93.609
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000804
  01:35:36 STEP  805 H= 805:00 +CPU= 86.300
  01:36:54 STEP  806 H= 806:00 +CPU= 77.519
  01:38:43 STEP  807 H= 807:00 +CPU=108.414
  01:39:35 STEP  808 H= 808:00 +CPU= 51.352
  01:40:29 STEP  809 H= 809:00 +CPU= 53.783
  01:41:55 STEP  810 H= 810:00 +CPU= 86.137
  01:42:58 STEP  811 H= 811:00 +CPU= 62.208
  01:44:02 STEP  812 H= 812:00 +CPU= 64.292
  01:45:20 STEP  813 H= 813:00 +CPU= 77.684
  01:46:06 STEP  814 H= 814:00 +CPU= 44.951
  01:47:01 STEP  815 H= 815:00 +CPU= 55.068
  01:48:38 STEP  816 H= 816:00 +CPU= 96.563
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000804
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136_33.zip
Uploading the intermediate file: upload_file_33.zip
  01:49:29 STEP  817 H= 817:00 +CPU= 50.288
  01:50:10 STEP  818 H= 818:00 +CPU= 40.909
  01:51:24 STEP  819 H= 819:00 +CPU= 73.179
  01:52:25 STEP  820 H= 820:00 +CPU= 61.260
  01:53:21 STEP  821 H= 821:00 +CPU= 55.645
  01:54:36 STEP  822 H= 822:00 +CPU= 74.489
  01:55:25 STEP  823 H= 823:00 +CPU= 48.332
  01:56:16 STEP  824 H= 824:00 +CPU= 51.229
  01:57:44 STEP  825 H= 825:00 +CPU= 87.340
  01:58:34 STEP  826 H= 826:00 +CPU= 49.575
  01:59:33 STEP  827 H= 827:00 +CPU= 58.925
  02:01:02 STEP  828 H= 828:00 +CPU= 88.453
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000828
  02:02:01 STEP  829 H= 829:00 +CPU= 58.158
  02:02:55 STEP  830 H= 830:00 +CPU= 54.327
  02:04:13 STEP  831 H= 831:00 +CPU= 77.068
  02:05:10 STEP  832 H= 832:00 +CPU= 56.121
  02:05:53 STEP  833 H= 833:00 +CPU= 42.780
  02:07:03 STEP  834 H= 834:00 +CPU= 69.585
  02:07:54 STEP  835 H= 835:00 +CPU= 50.745
  02:08:37 STEP  836 H= 836:00 +CPU= 42.741
  02:09:26 STEP  837 H= 837:00 +CPU= 48.999
  02:10:11 STEP  838 H= 838:00 +CPU= 44.909
  02:10:48 STEP  839 H= 839:00 +CPU= 36.603
  02:11:52 STEP  840 H= 840:00 +CPU= 63.081
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136_34.zip
Uploading the intermediate file: upload_file_34.zip
  02:12:32 STEP  841 H= 841:00 +CPU= 39.916
  02:13:10 STEP  842 H= 842:00 +CPU= 38.581
  02:14:03 STEP  843 H= 843:00 +CPU= 52.361
  02:14:39 STEP  844 H= 844:00 +CPU= 35.626
  02:15:14 STEP  845 H= 845:00 +CPU= 35.172
  02:16:10 STEP  846 H= 846:00 +CPU= 55.349
  02:16:42 STEP  847 H= 847:00 +CPU= 32.131
  02:17:14 STEP  848 H= 848:00 +CPU= 31.600
  02:18:04 STEP  849 H= 849:00 +CPU= 50.077
  02:18:38 STEP  850 H= 850:00 +CPU= 34.035
  02:19:16 STEP  851 H= 851:00 +CPU= 38.208
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0468
(argv4) batchid: 988
(argv5) wuid: 12205112
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/20
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_de683492d810da57ceaeaf2351e191f0 to: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0468_2019050100_123_988_12205112.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/20/oifs_43r3_ps_0468_2019050100_123_988_12205112.zip
ic_ancil_file: ic_ancil_12205112
ifsdata_file: ifsdata_12205112
climate_data_file: clim_data_12205112
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f6b3c8f344361038b9791ba0e863d962 to: /var/lib/boinc-client/slots/20/ic_ancil_12205112.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/20/ic_ancil_12205112.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e61ab47a9b0e931e54e8f86bc332320d to: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12205112.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/20/ifsdata/ifsdata_12205112.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/20/159l_2/clim_data_12205112.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/20/159l_2/clim_data_12205112.zip
Checking for progress XML file: /var/lib/boinc-client/slots/20/progress_file_12205112.xml
Opened progress file ok : /var/lib/boinc-client/slots/20/progress_file_12205112.xml
-- Model is restarting --
Adjusting last_iter, 852, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/20/progress_file_12205112.xml
last_cpu_time: 71653
upload_file_number: 35
last_iter: 839
last_upload: 3024000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136
The child process has been launched with process id: 1178107
Executing the command: /var/lib/boinc-client/slots/20/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Serveur-Master3:1:1:1178107:1178107] [20230118:141710:1674051430:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:19:06 STEP  840 H= 840:00 +CPU=106.019
  14:20:14 STEP  841 H= 841:00 +CPU= 67.451
  14:21:24 STEP  842 H= 842:00 +CPU= 69.222
  14:23:02 STEP  843 H= 843:00 +CPU= 97.536
  14:24:03 STEP  844 H= 844:00 +CPU= 60.913
  14:25:00 STEP  845 H= 845:00 +CPU= 57.103
  14:26:28 STEP  846 H= 846:00 +CPU= 86.809
  14:27:25 STEP  847 H= 847:00 +CPU= 57.400
  14:28:27 STEP  848 H= 848:00 +CPU= 61.660
  14:29:50 STEP  849 H= 849:00 +CPU= 82.434
  14:30:56 STEP  850 H= 850:00 +CPU= 65.827
  14:31:59 STEP  851 H= 851:00 +CPU= 62.639
  14:33:45 STEP  852 H= 852:00 +CPU=105.683
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000852
  14:34:53 STEP  853 H= 853:00 +CPU= 67.677
  14:35:55 STEP  854 H= 854:00 +CPU= 61.440
  14:37:19 STEP  855 H= 855:00 +CPU= 83.892
  14:38:19 STEP  856 H= 856:00 +CPU= 59.035
  14:39:17 STEP  857 H= 857:00 +CPU= 58.200
  14:40:47 STEP  858 H= 858:00 +CPU= 89.429
  14:41:51 STEP  859 H= 859:00 +CPU= 63.141
  14:42:51 STEP  860 H= 860:00 +CPU= 60.369
  14:44:19 STEP  861 H= 861:00 +CPU= 87.771
  14:45:24 STEP  862 H= 862:00 +CPU= 64.116
  14:46:31 STEP  863 H= 863:00 +CPU= 66.324
  14:48:24 STEP  864 H= 864:00 +CPU=111.587
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/20/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205112/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0468_2019050100_123_988_12205112_0_r1324753136_35.zip
Uploading the intermediate file: upload_file_35.zip
  14:49:30 STEP  865 H= 865:00 +CPU= 65.770
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net