Name | oifs_43r3_ps_0895_2018050100_123_987_12204539_0 |
Workunit | 12204539 |
Created | 24 Dec 2022, 21:50:30 UTC |
Sent | 16 Jan 2023, 21:59:25 UTC |
Report deadline | 17 Mar 2023, 21:59:25 UTC |
Received | 24 Jan 2023, 20:56:34 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1538188 |
Run time | 10 hours 48 min 35 sec |
CPU time | 2 hours 45 min |
Validate state | Invalid |
Credit | 7,862.40 |
Device peak FLOPS | 9.06 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,408.38 MB |
Peak swap size | 4,978.08 MB |
Peak disk usage | 7,437.14 MB |
Stderr | <core_client_version>7.18.1</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> q0f+001704 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001704 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMGGhq0f+001680 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMSHhq0f+001680 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMUAhq0f+001680 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMGGhq0f+001692 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMSHhq0f+001692 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMUAhq0f+001692 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0895_2018050100_123_987_12204539_0_r122444160_36.zip Uploading the intermediate file: upload_file_36.zip 17:10:37 STEP 1705 H=1705:00 +CPU= 7.362 17:10:44 STEP 1706 H=1706:00 +CPU= 6.779 17:10:55 STEP 1707 H=1707:00 +CPU= 11.233 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:11:04 STEP 1708 H=1708:00 +CPU= 6.879 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:11:15 STEP 1709 H=1709:00 +CPU= 6.836 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:11:35 STEP 1710 H=1710:00 +CPU= 11.696 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:11:44 STEP 1711 H=1711:00 +CPU= 7.168 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:11:53 STEP 1712 H=1712:00 +CPU= 6.947 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:12:12 STEP 1713 H=1713:00 +CPU= 10.752 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:12:25 STEP 1714 H=1714:00 +CPU= 7.798 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:12:38 STEP 1715 H=1715:00 +CPU= 7.029 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:12:55 STEP 1716 H=1716:00 +CPU= 12.562 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001716 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001716 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001716 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:13:08 STEP 1717 H=1717:00 +CPU= 7.058 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:13:19 STEP 1718 H=1718:00 +CPU= 7.086 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:13:38 STEP 1719 H=1719:00 +CPU= 11.001 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:13:50 STEP 1720 H=1720:00 +CPU= 7.677 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:14:03 STEP 1721 H=1721:00 +CPU= 6.819 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:14:24 STEP 1722 H=1722:00 +CPU= 12.293 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:14:35 STEP 1723 H=1723:00 +CPU= 6.884 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:14:48 STEP 1724 H=1724:00 +CPU= 6.863 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:15:05 STEP 1725 H=1725:00 +CPU= 11.193 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:15:15 STEP 1726 H=1726:00 +CPU= 8.221 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:15:30 STEP 1727 H=1727:00 +CPU= 8.096 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:15:55 STEP 1728 H=1728:00 +CPU= 14.797 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001728 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001728 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001728 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMGGhq0f+001704 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMSHhq0f+001704 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMUAhq0f+001704 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMGGhq0f+001716 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMSHhq0f+001716 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMUAhq0f+001716 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0895_2018050100_123_987_12204539_0_r122444160_37.zip Uploading the intermediate file: upload_file_37.zip 17:16:03 STEP 1729 H=1729:00 +CPU= 6.607 17:16:10 STEP 1730 H=1730:00 +CPU= 6.862 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:16:24 STEP 1731 H=1731:00 +CPU= 11.136 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:16:35 STEP 1732 H=1732:00 +CPU= 7.214 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:16:48 STEP 1733 H=1733:00 +CPU= 7.010 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:17:08 STEP 1734 H=1734:00 +CPU= 11.625 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:17:20 STEP 1735 H=1735:00 +CPU= 7.313 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:17:34 STEP 1736 H=1736:00 +CPU= 8.044 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:17:49 STEP 1737 H=1737:00 +CPU= 11.329 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:17:59 STEP 1738 H=1738:00 +CPU= 7.141 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:18:09 STEP 1739 H=1739:00 +CPU= 6.664 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:18:30 STEP 1740 H=1740:00 +CPU= 12.084 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:18:41 STEP 1741 H=1741:00 +CPU= 6.648 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001740 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001740 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001740 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:18:54 STEP 1742 H=1742:00 +CPU= 7.682 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:19:08 STEP 1743 H=1743:00 +CPU= 11.470 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:19:20 STEP 1744 H=1744:00 +CPU= 7.486 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:19:31 STEP 1745 H=1745:00 +CPU= 7.021 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:19:51 STEP 1746 H=1746:00 +CPU= 11.401 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:20:04 STEP 1747 H=1747:00 +CPU= 7.336 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:20:13 STEP 1748 H=1748:00 +CPU= 6.925 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:20:31 STEP 1749 H=1749:00 +CPU= 11.146 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:20:44 STEP 1750 H=1750:00 +CPU= 6.930 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:20:57 STEP 1751 H=1751:00 +CPU= 7.380 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:21:19 STEP 1752 H=1752:00 +CPU= 13.121 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001752 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001752 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001752 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMGGhq0f+001728 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMSHhq0f+001728 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMUAhq0f+001728 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMGGhq0f+001740 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMSHhq0f+001740 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMUAhq0f+001740 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0895_2018050100_123_987_12204539_0_r122444160_38.zip Uploading the intermediate file: upload_file_38.zip 17:21:26 STEP 1753 H=1753:00 +CPU= 7.096 17:21:36 STEP 1754 H=1754:00 +CPU= 9.145 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:21:53 STEP 1755 H=1755:00 +CPU= 11.190 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:22:04 STEP 1756 H=1756:00 +CPU= 7.194 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 19:55:50 STEP 1757 H=1757:00 +CPU= 8.343 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 19:56:10 STEP 1758 H=1758:00 +CPU= 12.130 Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2018050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0895 (argv4) batchid: 987 (argv5) wuid: 12204539 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/11 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_9220dea88d4d82008ded104d9db4048c to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0895_2018050100_123_987_12204539.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0895_2018050100_123_987_12204539.zip ic_ancil_file: ic_ancil_12204539 ifsdata_file: ifsdata_12204539 climate_data_file: clim_data_12204539 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_444c5e160f6c70a39f47db9ae10e16fd to: /var/lib/boinc-client/slots/11/ic_ancil_12204539.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12204539.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_878facb89a0e1b5f1375b5182be7d639 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12204539.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12204539.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12204539.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12204539.zip Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12204539.xml Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12204539.xml -- Model is restarting -- Adjusting last_iter, 1758, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12204539.xml last_cpu_time: 9309 upload_file_number: 39 last_iter: 1751 last_upload: 6307200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0895_2018050100_123_987_12204539_0_r122444160 The child process has been launched with process id: 1130539 Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1130539:1130539] [20230124:202039:1674584439:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:21:08 STEP 1752 H=1752:00 +CPU= 16.153 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:21:20 STEP 1753 H=1753:00 +CPU= 7.929 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:21:35 STEP 1754 H=1754:00 +CPU= 8.287 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:21:52 STEP 1755 H=1755:00 +CPU= 10.885 Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:22:01 STEP 1756 H=1756:00 +CPU= 7.286 Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:22:11 STEP 1757 H=1757:00 +CPU= 7.651 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:22:30 STEP 1758 H=1758:00 +CPU= 10.747 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:22:43 STEP 1759 H=1759:00 +CPU= 6.664 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:22:54 STEP 1760 H=1760:00 +CPU= 6.817 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:23:13 STEP 1761 H=1761:00 +CPU= 10.342 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:23:24 STEP 1762 H=1762:00 +CPU= 7.490 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:23:38 STEP 1763 H=1763:00 +CPU= 7.255 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:24:00 STEP 1764 H=1764:00 +CPU= 13.729 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001764 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001764 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001764 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:24:13 STEP 1765 H=1765:00 +CPU= 7.142 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:24:25 STEP 1766 H=1766:00 +CPU= 7.398 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:24:44 STEP 1767 H=1767:00 +CPU= 11.467 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:24:58 STEP 1768 H=1768:00 +CPU= 7.217 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:25:09 STEP 1769 H=1769:00 +CPU= 6.571 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:25:28 STEP 1770 H=1770:00 +CPU= 11.192 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:26:43 STEP 1771 H=1771:00 +CPU= 7.749 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:26:56 STEP 1772 H=1772:00 +CPU= 8.024 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:27:20 STEP 1773 H=1773:00 +CPU= 13.872 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2018050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0895 (argv4) batchid: 987 (argv5) wuid: 12204539 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/11 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_9220dea88d4d82008ded104d9db4048c to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0895_2018050100_123_987_12204539.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0895_2018050100_123_987_12204539.zip ic_ancil_file: ic_ancil_12204539 ifsdata_file: ifsdata_12204539 climate_data_file: clim_data_12204539 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_444c5e160f6c70a39f47db9ae10e16fd to: /var/lib/boinc-client/slots/11/ic_ancil_12204539.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12204539.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_878facb89a0e1b5f1375b5182be7d639 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12204539.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12204539.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12204539.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12204539.zip Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12204539.xml Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12204539.xml -- Model is restarting -- Adjusting last_iter, 1774, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12204539.xml last_cpu_time: 9512 upload_file_number: 39 last_iter: 1751 last_upload: 6307200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0895_2018050100_123_987_12204539_0_r122444160 The child process has been launched with process id: 1254352 Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1254352:1254352] [20230124:212832:1674588512:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:28:54 STEP 1752 H=1752:00 +CPU= 14.115 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:29:05 STEP 1753 H=1753:00 +CPU= 7.316 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:29:20 STEP 1754 H=1754:00 +CPU= 8.247 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:29:36 STEP 1755 H=1755:00 +CPU= 11.790 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:29:49 STEP 1756 H=1756:00 +CPU= 7.099 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:30:00 STEP 1757 H=1757:00 +CPU= 6.757 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:30:21 STEP 1758 H=1758:00 +CPU= 12.045 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:30:34 STEP 1759 H=1759:00 +CPU= 6.647 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:30:48 STEP 1760 H=1760:00 +CPU= 8.083 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:31:09 STEP 1761 H=1761:00 +CPU= 12.227 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:31:21 STEP 1762 H=1762:00 +CPU= 7.553 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:31:34 STEP 1763 H=1763:00 +CPU= 7.091 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:31:58 STEP 1764 H=1764:00 +CPU= 13.795 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001764 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001764 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001764 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:32:10 STEP 1765 H=1765:00 +CPU= 7.388 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:32:24 STEP 1766 H=1766:00 +CPU= 7.676 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:32:45 STEP 1767 H=1767:00 +CPU= 12.108 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:32:58 STEP 1768 H=1768:00 +CPU= 7.482 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:33:09 STEP 1769 H=1769:00 +CPU= 6.956 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:33:53 STEP 1770 H=1770:00 +CPU= 16.002 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:06:49 STEP 1771 H=1771:00 +CPU= 9.543 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:07:19 STEP 1772 H=1772:00 +CPU= 10.754 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:08:04 STEP 1773 H=1773:00 +CPU= 17.316 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:08:20 STEP 1774 H=1774:00 +CPU= 8.451 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:08:38 STEP 1775 H=1775:00 +CPU= 8.189 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:09:10 STEP 1776 H=1776:00 +CPU= 15.841 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001776 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001776 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001776 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMGGhq0f+001752 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMSHhq0f+001752 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMUAhq0f+001752 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMGGhq0f+001764 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMSHhq0f+001764 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204539/ICMUAhq0f+001764 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0895_2018050100_123_987_12204539_0_r122444160_39.zip Uploading the intermediate file: upload_file_39.zip 22:09:22 STEP 1777 H=1777:00 +CPU= 8.616 22:09:35 STEP 1778 H=1778:00 +CPU= 8.766 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:10:00 STEP 1779 H=1779:00 +CPU= 13.109 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:10:16 STEP 1780 H=1780:00 +CPU= 8.071 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:10:31 STEP 1781 H=1781:00 +CPU= 7.578 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:10:59 STEP 1782 H=1782:00 +CPU= 13.505 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:11:14 STEP 1783 H=1783:00 +CPU= 7.346 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:11:30 STEP 1784 H=1784:00 +CPU= 8.274 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:11:56 STEP 1785 H=1785:00 +CPU= 13.610 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:12:14 STEP 1786 H=1786:00 +CPU= 8.283 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:12:30 STEP 1787 H=1787:00 +CPU= 7.884 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:13:00 STEP 1788 H=1788:00 +CPU= 14.654 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+001788 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+001788 Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+001788 Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:13:15 STEP 1789 H=1789:00 +CPU= 7.999 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:13:31 STEP 1790 H=1790:00 +CPU= 8.317 Suspend request received from the BOINC client, suspending the child process Resuming the child process Abort request received from BOINC client, ending the child process </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 16:39:29 | 1538188 | 22291041 | oifs_43r3_ps_0895_2018050100_123_987_12204539_0 | 5,616,000 | 7,561 | 0.0013 |
09 May 2023 16:39:26 | 1538188 | 22291041 | oifs_43r3_ps_0895_2018050100_123_987_12204539_0 | 4,752,000 | 5,469 | 0.0012 |
09 May 2023 16:38:15 | 1538188 | 22291041 | oifs_43r3_ps_0895_2018050100_123_987_12204539_0 | 3,888,000 | 3,183 | 0.0008 |
09 May 2023 16:24:25 | 1538188 | 22291041 | oifs_43r3_ps_0895_2018050100_123_987_12204539_0 | 2,592,000 | 12,107 | 0.0047 |
09 May 2023 16:23:18 | 1538188 | 22291041 | oifs_43r3_ps_0895_2018050100_123_987_12204539_0 | 1,728,000 | 8,001 | 0.0046 |
09 May 2023 16:23:13 | 1538188 | 22291041 | oifs_43r3_ps_0895_2018050100_123_987_12204539_0 | 864,000 | 4,136 | 0.0048 |
©2024 climateprediction.net