Name | oifs_43r3_ps_0715_1995050100_123_964_12181359_0 |
Workunit | 12181359 |
Created | 22 Dec 2022, 9:04:42 UTC |
Sent | 22 Dec 2022, 9:08:48 UTC |
Report deadline | 21 Jan 2023, 9:08:48 UTC |
Received | 1 Jan 2023, 19:52:13 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1521081 |
Run time | 1 days 8 hours 21 min 59 sec |
CPU time | 1 days 6 hours 21 min 9 sec |
Validate state | Invalid |
Credit | 13,184.64 |
Device peak FLOPS | 5.44 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,418.39 MB |
Peak swap size | 4,978.08 MB |
Peak disk usage | 7,289.56 MB |
Stderr | <core_client_version>7.18.1</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> 672578124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:56383:56383] [20230101:140204:1672578124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:56383:56383] [20230101:140204:1672578124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:56383:56383] [20230101:140204:1672578124:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:56383:56383] [20230101:140204:1672578124:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 14:03:19 STEP 2616 H=2616:00 +CPU= 67.292 14:04:05 STEP 2617 H=2617:00 +CPU= 44.686 14:04:56 STEP 2618 H=2618:00 +CPU= 49.924 14:06:15 STEP 2619 H=2619:00 +CPU= 77.610 14:07:02 STEP 2620 H=2620:00 +CPU= 45.325 14:07:45 STEP 2621 H=2621:00 +CPU= 42.326 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:14:08 STEP 2622 H=2622:00 +CPU= 57.591 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:16:03 STEP 2623 H=2623:00 +CPU= 34.340 14:16:37 STEP 2624 H=2624:00 +CPU= 33.718 14:17:35 STEP 2625 H=2625:00 +CPU= 56.934 14:18:16 STEP 2626 H=2626:00 +CPU= 40.481 14:18:52 STEP 2627 H=2627:00 +CPU= 36.036 14:19:46 STEP 2628 H=2628:00 +CPU= 53.430 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002628 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002628 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002628 14:20:20 STEP 2629 H=2629:00 +CPU= 33.379 14:21:07 STEP 2630 H=2630:00 +CPU= 45.912 14:22:28 STEP 2631 H=2631:00 +CPU= 79.144 14:23:15 STEP 2632 H=2632:00 +CPU= 46.735 14:24:03 STEP 2633 H=2633:00 +CPU= 46.750 14:25:12 STEP 2634 H=2634:00 +CPU= 68.788 14:25:59 STEP 2635 H=2635:00 +CPU= 46.403 14:26:46 STEP 2636 H=2636:00 +CPU= 46.182 14:27:56 STEP 2637 H=2637:00 +CPU= 68.249 14:28:48 STEP 2638 H=2638:00 +CPU= 51.480 14:29:39 STEP 2639 H=2639:00 +CPU= 49.090 14:31:02 STEP 2640 H=2640:00 +CPU= 81.602 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002640 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002640 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002640 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002616 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002616 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002616 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002628 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002628 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002628 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_109.zip Uploading the intermediate file: upload_file_109.zip 14:31:44 STEP 2641 H=2641:00 +CPU= 41.299 14:32:24 STEP 2642 H=2642:00 +CPU= 39.583 14:33:22 STEP 2643 H=2643:00 +CPU= 57.537 14:34:02 STEP 2644 H=2644:00 +CPU= 38.750 14:34:40 STEP 2645 H=2645:00 +CPU= 36.426 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:36:45 STEP 2646 H=2646:00 +CPU= 51.335 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:42:19 STEP 2647 H=2647:00 +CPU= 31.487 14:42:58 STEP 2648 H=2648:00 +CPU= 30.892 14:43:58 STEP 2649 H=2649:00 +CPU= 47.128 14:44:41 STEP 2650 H=2650:00 +CPU= 34.099 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:45:34 STEP 2651 H=2651:00 +CPU= 34.549 14:46:38 STEP 2652 H=2652:00 +CPU= 60.106 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002652 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002652 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002652 14:47:15 STEP 2653 H=2653:00 +CPU= 35.445 14:47:54 STEP 2654 H=2654:00 +CPU= 37.784 14:48:56 STEP 2655 H=2655:00 +CPU= 57.907 14:49:35 STEP 2656 H=2656:00 +CPU= 37.983 14:50:09 STEP 2657 H=2657:00 +CPU= 32.518 14:51:00 STEP 2658 H=2658:00 +CPU= 47.931 14:51:27 STEP 2659 H=2659:00 +CPU= 26.818 14:51:55 STEP 2660 H=2660:00 +CPU= 26.764 14:52:36 STEP 2661 H=2661:00 +CPU= 39.875 14:53:05 STEP 2662 H=2662:00 +CPU= 26.859 14:53:33 STEP 2663 H=2663:00 +CPU= 27.240 14:54:25 STEP 2664 H=2664:00 +CPU= 47.791 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002664 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002664 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002664 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002640 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002640 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002640 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002652 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002652 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002652 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_110.zip Uploading the intermediate file: upload_file_110.zip 14:54:49 STEP 2665 H=2665:00 +CPU= 19.488 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:57:24 STEP 2666 H=2666:00 +CPU= 27.989 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:58:26 STEP 2667 H=2667:00 +CPU= 44.750 14:59:02 STEP 2668 H=2668:00 +CPU= 30.639 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:12:24 STEP 2669 H=2669:00 +CPU= 28.897 15:13:20 STEP 2670 H=2670:00 +CPU= 47.995 15:13:53 STEP 2671 H=2671:00 +CPU= 28.197 15:14:25 STEP 2672 H=2672:00 +CPU= 28.255 15:15:14 STEP 2673 H=2673:00 +CPU= 42.520 15:15:46 STEP 2674 H=2674:00 +CPU= 28.052 15:16:18 STEP 2675 H=2675:00 +CPU= 27.513 15:17:12 STEP 2676 H=2676:00 +CPU= 47.116 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002676 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002676 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002676 15:17:44 STEP 2677 H=2677:00 +CPU= 27.632 15:18:17 STEP 2678 H=2678:00 +CPU= 28.631 15:19:05 STEP 2679 H=2679:00 +CPU= 42.181 15:19:40 STEP 2680 H=2680:00 +CPU= 29.758 15:20:16 STEP 2681 H=2681:00 +CPU= 30.703 15:21:12 STEP 2682 H=2682:00 +CPU= 48.249 15:21:48 STEP 2683 H=2683:00 +CPU= 30.881 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:26:08 STEP 2684 H=2684:00 +CPU= 31.085 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:30:36 STEP 2685 H=2685:00 +CPU= 45.081 15:31:10 STEP 2686 H=2686:00 +CPU= 29.863 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1995050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0715 (argv4) batchid: 964 (argv5) wuid: 12181359 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/10 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_bf8807d3ab1da18b8cda64fdcb91ea4d to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0715_1995050100_123_964_12181359.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0715_1995050100_123_964_12181359.zip ic_ancil_file: ic_ancil_12181359 ifsdata_file: ifsdata_12181359 climate_data_file: clim_data_12181359 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_e33f194d446fdf5176278169e33b28eb to: /var/lib/boinc-client/slots/10/ic_ancil_12181359.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12181359.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ec633de182e05f42f6929883a755e89f to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12181359.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12181359.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12181359.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12181359.zip Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12181359.xml Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12181359.xml -- Model is restarting -- Adjusting last_iter, 2687, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12181359.xml last_cpu_time: 99715 upload_file_number: 111 last_iter: 2663 last_upload: 9590400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501 The child process has been launched with process id: 69144 Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:69144:69144] [20230101:155956:1672585196:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 16:01:06 STEP 2664 H=2664:00 +CPU= 56.735 16:01:41 STEP 2665 H=2665:00 +CPU= 31.450 16:02:21 STEP 2666 H=2666:00 +CPU= 35.154 16:03:20 STEP 2667 H=2667:00 +CPU= 51.585 16:03:55 STEP 2668 H=2668:00 +CPU= 31.127 16:04:29 STEP 2669 H=2669:00 +CPU= 31.735 16:05:22 STEP 2670 H=2670:00 +CPU= 48.671 16:05:56 STEP 2671 H=2671:00 +CPU= 31.199 16:06:29 STEP 2672 H=2672:00 +CPU= 31.304 16:07:21 STEP 2673 H=2673:00 +CPU= 47.343 16:07:53 STEP 2674 H=2674:00 +CPU= 29.961 16:08:28 STEP 2675 H=2675:00 +CPU= 32.320 16:09:27 STEP 2676 H=2676:00 +CPU= 53.905 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002676 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002676 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002676 16:10:00 STEP 2677 H=2677:00 +CPU= 31.197 16:10:35 STEP 2678 H=2678:00 +CPU= 32.534 16:11:27 STEP 2679 H=2679:00 +CPU= 47.398 16:11:58 STEP 2680 H=2680:00 +CPU= 29.020 16:12:29 STEP 2681 H=2681:00 +CPU= 28.818 16:13:16 STEP 2682 H=2682:00 +CPU= 44.066 16:13:48 STEP 2683 H=2683:00 +CPU= 28.830 16:14:22 STEP 2684 H=2684:00 +CPU= 31.614 16:15:09 STEP 2685 H=2685:00 +CPU= 43.795 16:15:41 STEP 2686 H=2686:00 +CPU= 29.479 16:16:12 STEP 2687 H=2687:00 +CPU= 28.789 16:17:08 STEP 2688 H=2688:00 +CPU= 51.993 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002688 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002688 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002688 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002664 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002664 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002664 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002676 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002676 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002676 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_111.zip Uploading the intermediate file: upload_file_111.zip 16:17:43 STEP 2689 H=2689:00 +CPU= 33.367 16:18:20 STEP 2690 H=2690:00 +CPU= 34.126 16:19:13 STEP 2691 H=2691:00 +CPU= 49.168 16:19:49 STEP 2692 H=2692:00 +CPU= 33.782 16:20:21 STEP 2693 H=2693:00 +CPU= 29.265 16:21:13 STEP 2694 H=2694:00 +CPU= 48.942 16:21:47 STEP 2695 H=2695:00 +CPU= 31.756 16:22:24 STEP 2696 H=2696:00 +CPU= 34.180 16:23:16 STEP 2697 H=2697:00 +CPU= 48.918 16:23:52 STEP 2698 H=2698:00 +CPU= 33.914 16:24:27 STEP 2699 H=2699:00 +CPU= 31.681 16:25:19 STEP 2700 H=2700:00 +CPU= 49.504 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002700 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002700 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002700 16:25:49 STEP 2701 H=2701:00 +CPU= 27.815 16:26:22 STEP 2702 H=2702:00 +CPU= 31.194 16:27:12 STEP 2703 H=2703:00 +CPU= 46.221 16:27:45 STEP 2704 H=2704:00 +CPU= 30.630 16:28:14 STEP 2705 H=2705:00 +CPU= 28.006 16:29:01 STEP 2706 H=2706:00 +CPU= 44.265 16:29:30 STEP 2707 H=2707:00 +CPU= 27.005 16:30:01 STEP 2708 H=2708:00 +CPU= 28.781 16:30:46 STEP 2709 H=2709:00 +CPU= 42.562 16:31:17 STEP 2710 H=2710:00 +CPU= 29.188 16:31:48 STEP 2711 H=2711:00 +CPU= 28.967 16:32:41 STEP 2712 H=2712:00 +CPU= 49.978 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002712 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002712 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002712 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002688 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002688 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002688 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002700 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002700 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002700 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_112.zip Uploading the intermediate file: upload_file_112.zip 16:33:11 STEP 2713 H=2713:00 +CPU= 27.982 16:33:41 STEP 2714 H=2714:00 +CPU= 28.268 16:34:27 STEP 2715 H=2715:00 +CPU= 42.824 16:34:58 STEP 2716 H=2716:00 +CPU= 29.336 16:35:28 STEP 2717 H=2717:00 +CPU= 28.158 16:36:13 STEP 2718 H=2718:00 +CPU= 42.700 16:36:43 STEP 2719 H=2719:00 +CPU= 28.035 16:37:13 STEP 2720 H=2720:00 +CPU= 27.804 16:37:56 STEP 2721 H=2721:00 +CPU= 40.565 16:38:25 STEP 2722 H=2722:00 +CPU= 27.177 16:38:52 STEP 2723 H=2723:00 +CPU= 26.142 16:39:39 STEP 2724 H=2724:00 +CPU= 44.102 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002724 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002724 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002724 16:40:08 STEP 2725 H=2725:00 +CPU= 26.983 16:40:36 STEP 2726 H=2726:00 +CPU= 26.876 16:41:19 STEP 2727 H=2727:00 +CPU= 40.627 16:41:48 STEP 2728 H=2728:00 +CPU= 27.346 16:42:16 STEP 2729 H=2729:00 +CPU= 26.454 16:43:00 STEP 2730 H=2730:00 +CPU= 40.945 16:43:29 STEP 2731 H=2731:00 +CPU= 26.159 16:44:01 STEP 2732 H=2732:00 +CPU= 27.663 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:47:30 STEP 2733 H=2733:00 +CPU= 41.484 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:01:07 STEP 2734 H=2734:00 +CPU= 28.190 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:07:18 STEP 2735 H=2735:00 +CPU= 29.059 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1995050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0715 (argv4) batchid: 964 (argv5) wuid: 12181359 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/10 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_bf8807d3ab1da18b8cda64fdcb91ea4d to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0715_1995050100_123_964_12181359.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0715_1995050100_123_964_12181359.zip ic_ancil_file: ic_ancil_12181359 ifsdata_file: ifsdata_12181359 climate_data_file: clim_data_12181359 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_e33f194d446fdf5176278169e33b28eb to: /var/lib/boinc-client/slots/10/ic_ancil_12181359.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12181359.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ec633de182e05f42f6929883a755e89f to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12181359.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12181359.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12181359.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12181359.zip Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12181359.xml Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12181359.xml -- Model is restarting -- Adjusting last_iter, 2736, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12181359.xml last_cpu_time: 102283 upload_file_number: 113 last_iter: 2735 last_upload: 9763200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501 The child process has been launched with process id: 72744 Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:72744:72744] [20230101:172154:1672590114:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 17:22:38 STEP 2736 H=2736:00 +CPU= 40.134 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002736 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002736 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002736 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002712 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002712 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002712 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002724 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002724 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002724 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_113.zip Uploading the intermediate file: upload_file_113.zip 17:23:01 STEP 2737 H=2737:00 +CPU= 22.590 17:23:25 STEP 2738 H=2738:00 +CPU= 22.642 17:24:02 STEP 2739 H=2739:00 +CPU= 36.421 17:24:30 STEP 2740 H=2740:00 +CPU= 25.872 17:25:02 STEP 2741 H=2741:00 +CPU= 29.869 17:25:55 STEP 2742 H=2742:00 +CPU= 48.441 17:26:28 STEP 2743 H=2743:00 +CPU= 29.418 17:27:04 STEP 2744 H=2744:00 +CPU= 32.716 17:27:57 STEP 2745 H=2745:00 +CPU= 47.827 17:28:32 STEP 2746 H=2746:00 +CPU= 31.968 17:29:05 STEP 2747 H=2747:00 +CPU= 29.657 17:30:03 STEP 2748 H=2748:00 +CPU= 53.783 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002748 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002748 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002748 17:30:37 STEP 2749 H=2749:00 +CPU= 30.137 17:31:10 STEP 2750 H=2750:00 +CPU= 30.439 17:32:02 STEP 2751 H=2751:00 +CPU= 46.802 17:32:34 STEP 2752 H=2752:00 +CPU= 29.078 17:33:08 STEP 2753 H=2753:00 +CPU= 30.747 17:34:02 STEP 2754 H=2754:00 +CPU= 49.474 17:34:36 STEP 2755 H=2755:00 +CPU= 31.289 17:35:11 STEP 2756 H=2756:00 +CPU= 32.174 17:36:01 STEP 2757 H=2757:00 +CPU= 46.128 17:36:34 STEP 2758 H=2758:00 +CPU= 29.347 17:37:05 STEP 2759 H=2759:00 +CPU= 28.872 17:38:00 STEP 2760 H=2760:00 +CPU= 50.465 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002760 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002760 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002760 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002736 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002736 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002736 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002748 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002748 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002748 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_114.zip Uploading the intermediate file: upload_file_114.zip 17:38:31 STEP 2761 H=2761:00 +CPU= 27.968 17:39:03 STEP 2762 H=2762:00 +CPU= 30.252 17:39:51 STEP 2763 H=2763:00 +CPU= 43.966 17:40:22 STEP 2764 H=2764:00 +CPU= 29.564 17:40:53 STEP 2765 H=2765:00 +CPU= 27.758 17:41:41 STEP 2766 H=2766:00 +CPU= 44.158 17:42:15 STEP 2767 H=2767:00 +CPU= 31.158 17:42:50 STEP 2768 H=2768:00 +CPU= 32.325 17:43:44 STEP 2769 H=2769:00 +CPU= 49.064 17:44:21 STEP 2770 H=2770:00 +CPU= 33.477 17:44:54 STEP 2771 H=2771:00 +CPU= 30.578 17:45:51 STEP 2772 H=2772:00 +CPU= 52.494 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002772 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002772 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002772 17:46:25 STEP 2773 H=2773:00 +CPU= 30.761 17:47:01 STEP 2774 H=2774:00 +CPU= 32.710 17:47:54 STEP 2775 H=2775:00 +CPU= 47.714 17:48:29 STEP 2776 H=2776:00 +CPU= 32.349 17:49:04 STEP 2777 H=2777:00 +CPU= 31.795 17:49:54 STEP 2778 H=2778:00 +CPU= 45.598 17:50:24 STEP 2779 H=2779:00 +CPU= 27.822 17:50:57 STEP 2780 H=2780:00 +CPU= 30.354 17:51:50 STEP 2781 H=2781:00 +CPU= 48.727 17:52:24 STEP 2782 H=2782:00 +CPU= 30.782 17:52:53 STEP 2783 H=2783:00 +CPU= 27.956 17:53:49 STEP 2784 H=2784:00 +CPU= 49.914 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002784 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002784 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002784 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002760 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002760 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002760 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002772 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002772 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002772 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_115.zip Uploading the intermediate file: upload_file_115.zip 17:54:18 STEP 2785 H=2785:00 +CPU= 26.468 17:54:48 STEP 2786 H=2786:00 +CPU= 27.733 17:55:35 STEP 2787 H=2787:00 +CPU= 43.079 17:56:04 STEP 2788 H=2788:00 +CPU= 27.950 17:56:34 STEP 2789 H=2789:00 +CPU= 27.204 17:57:21 STEP 2790 H=2790:00 +CPU= 43.610 17:57:50 STEP 2791 H=2791:00 +CPU= 27.580 17:58:20 STEP 2792 H=2792:00 +CPU= 27.747 17:59:06 STEP 2793 H=2793:00 +CPU= 42.635 17:59:37 STEP 2794 H=2794:00 +CPU= 28.437 18:00:07 STEP 2795 H=2795:00 +CPU= 28.052 18:00:57 STEP 2796 H=2796:00 +CPU= 46.709 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002796 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002796 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002796 18:01:27 STEP 2797 H=2797:00 +CPU= 27.639 18:01:56 STEP 2798 H=2798:00 +CPU= 27.583 18:02:40 STEP 2799 H=2799:00 +CPU= 40.567 18:03:08 STEP 2800 H=2800:00 +CPU= 26.342 18:03:36 STEP 2801 H=2801:00 +CPU= 25.883 18:04:20 STEP 2802 H=2802:00 +CPU= 40.873 18:04:48 STEP 2803 H=2803:00 +CPU= 25.596 18:05:17 STEP 2804 H=2804:00 +CPU= 26.792 18:06:00 STEP 2805 H=2805:00 +CPU= 40.566 18:06:29 STEP 2806 H=2806:00 +CPU= 26.390 18:06:57 STEP 2807 H=2807:00 +CPU= 26.564 18:07:53 STEP 2808 H=2808:00 +CPU= 51.504 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002808 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002808 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002808 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002784 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002784 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002784 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002796 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002796 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002796 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_116.zip Uploading the intermediate file: upload_file_116.zip 18:08:24 STEP 2809 H=2809:00 +CPU= 28.281 18:08:57 STEP 2810 H=2810:00 +CPU= 30.548 18:09:45 STEP 2811 H=2811:00 +CPU= 43.619 18:10:14 STEP 2812 H=2812:00 +CPU= 26.657 18:10:41 STEP 2813 H=2813:00 +CPU= 25.933 18:11:26 STEP 2814 H=2814:00 +CPU= 41.443 18:11:53 STEP 2815 H=2815:00 +CPU= 26.124 18:12:22 STEP 2816 H=2816:00 +CPU= 26.948 18:13:05 STEP 2817 H=2817:00 +CPU= 39.711 18:13:34 STEP 2818 H=2818:00 +CPU= 27.011 18:14:02 STEP 2819 H=2819:00 +CPU= 25.964 18:14:50 STEP 2820 H=2820:00 +CPU= 44.485 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002820 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002820 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002820 18:15:20 STEP 2821 H=2821:00 +CPU= 26.963 18:15:52 STEP 2822 H=2822:00 +CPU= 29.328 18:16:40 STEP 2823 H=2823:00 +CPU= 44.299 18:17:12 STEP 2824 H=2824:00 +CPU= 29.817 18:17:42 STEP 2825 H=2825:00 +CPU= 28.364 18:18:26 STEP 2826 H=2826:00 +CPU= 41.019 18:18:54 STEP 2827 H=2827:00 +CPU= 26.001 18:19:23 STEP 2828 H=2828:00 +CPU= 26.415 18:20:06 STEP 2829 H=2829:00 +CPU= 40.236 18:20:34 STEP 2830 H=2830:00 +CPU= 26.863 18:21:03 STEP 2831 H=2831:00 +CPU= 26.913 18:21:54 STEP 2832 H=2832:00 +CPU= 47.296 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002832 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002832 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002832 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002808 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002808 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002808 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002820 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002820 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002820 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_117.zip Uploading the intermediate file: upload_file_117.zip 18:22:22 STEP 2833 H=2833:00 +CPU= 25.701 18:22:50 STEP 2834 H=2834:00 +CPU= 26.687 18:23:36 STEP 2835 H=2835:00 +CPU= 42.392 18:24:08 STEP 2836 H=2836:00 +CPU= 29.526 18:24:39 STEP 2837 H=2837:00 +CPU= 28.381 18:25:26 STEP 2838 H=2838:00 +CPU= 44.852 18:25:56 STEP 2839 H=2839:00 +CPU= 27.286 18:26:25 STEP 2840 H=2840:00 +CPU= 27.082 18:27:07 STEP 2841 H=2841:00 +CPU= 40.097 18:27:36 STEP 2842 H=2842:00 +CPU= 27.130 18:28:04 STEP 2843 H=2843:00 +CPU= 26.181 18:28:53 STEP 2844 H=2844:00 +CPU= 45.039 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002844 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002844 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002844 18:29:20 STEP 2845 H=2845:00 +CPU= 26.059 18:29:50 STEP 2846 H=2846:00 +CPU= 26.827 18:30:34 STEP 2847 H=2847:00 +CPU= 40.573 18:31:03 STEP 2848 H=2848:00 +CPU= 26.860 18:31:33 STEP 2849 H=2849:00 +CPU= 27.445 18:32:22 STEP 2850 H=2850:00 +CPU= 45.242 18:32:51 STEP 2851 H=2851:00 +CPU= 26.971 18:33:20 STEP 2852 H=2852:00 +CPU= 26.708 18:34:04 STEP 2853 H=2853:00 +CPU= 40.518 18:34:32 STEP 2854 H=2854:00 +CPU= 26.850 18:35:01 STEP 2855 H=2855:00 +CPU= 26.640 18:35:51 STEP 2856 H=2856:00 +CPU= 47.548 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002856 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002856 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002856 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002832 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002832 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002832 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002844 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002844 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002844 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_118.zip Uploading the intermediate file: upload_file_118.zip 18:36:19 STEP 2857 H=2857:00 +CPU= 25.977 18:36:48 STEP 2858 H=2858:00 +CPU= 26.717 18:37:31 STEP 2859 H=2859:00 +CPU= 40.265 18:38:00 STEP 2860 H=2860:00 +CPU= 26.697 18:38:30 STEP 2861 H=2861:00 +CPU= 28.354 18:39:19 STEP 2862 H=2862:00 +CPU= 44.716 18:39:51 STEP 2863 H=2863:00 +CPU= 28.980 18:40:24 STEP 2864 H=2864:00 +CPU= 29.980 18:41:08 STEP 2865 H=2865:00 +CPU= 40.942 18:41:37 STEP 2866 H=2866:00 +CPU= 26.487 18:42:05 STEP 2867 H=2867:00 +CPU= 26.007 18:42:53 STEP 2868 H=2868:00 +CPU= 44.906 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002868 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002868 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002868 18:43:21 STEP 2869 H=2869:00 +CPU= 26.201 18:43:51 STEP 2870 H=2870:00 +CPU= 26.939 18:44:34 STEP 2871 H=2871:00 +CPU= 40.115 18:45:03 STEP 2872 H=2872:00 +CPU= 26.657 18:45:31 STEP 2873 H=2873:00 +CPU= 26.269 18:46:17 STEP 2874 H=2874:00 +CPU= 43.201 18:46:48 STEP 2875 H=2875:00 +CPU= 28.940 18:47:20 STEP 2876 H=2876:00 +CPU= 29.735 18:48:03 STEP 2877 H=2877:00 +CPU= 40.426 18:48:32 STEP 2878 H=2878:00 +CPU= 26.492 18:49:00 STEP 2879 H=2879:00 +CPU= 26.696 18:49:52 STEP 2880 H=2880:00 +CPU= 47.087 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002880 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002880 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002880 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002856 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002856 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002856 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002868 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002868 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002868 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_119.zip Uploading the intermediate file: upload_file_119.zip 18:50:20 STEP 2881 H=2881:00 +CPU= 25.752 18:50:48 STEP 2882 H=2882:00 +CPU= 26.429 18:51:32 STEP 2883 H=2883:00 +CPU= 40.126 18:52:00 STEP 2884 H=2884:00 +CPU= 26.920 18:52:28 STEP 2885 H=2885:00 +CPU= 26.053 18:53:14 STEP 2886 H=2886:00 +CPU= 42.345 18:53:45 STEP 2887 H=2887:00 +CPU= 29.014 18:54:16 STEP 2888 H=2888:00 +CPU= 29.062 18:54:59 STEP 2889 H=2889:00 +CPU= 40.318 18:55:27 STEP 2890 H=2890:00 +CPU= 26.568 18:55:55 STEP 2891 H=2891:00 +CPU= 26.090 18:56:44 STEP 2892 H=2892:00 +CPU= 45.190 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002892 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002892 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002892 18:57:14 STEP 2893 H=2893:00 +CPU= 27.053 18:57:46 STEP 2894 H=2894:00 +CPU= 29.473 18:58:34 STEP 2895 H=2895:00 +CPU= 43.398 18:59:05 STEP 2896 H=2896:00 +CPU= 29.007 18:59:36 STEP 2897 H=2897:00 +CPU= 28.349 19:00:29 STEP 2898 H=2898:00 +CPU= 48.265 19:01:04 STEP 2899 H=2899:00 +CPU= 31.682 19:01:40 STEP 2900 H=2900:00 +CPU= 32.961 19:02:34 STEP 2901 H=2901:00 +CPU= 48.875 19:03:11 STEP 2902 H=2902:00 +CPU= 33.353 19:03:44 STEP 2903 H=2903:00 +CPU= 29.966 19:04:40 STEP 2904 H=2904:00 +CPU= 51.461 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002904 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002904 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002904 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002880 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002880 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002880 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMGGhq0f+002892 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMSHhq0f+002892 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359/ICMUAhq0f+002892 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501_120.zip Uploading the intermediate file: upload_file_120.zip 19:05:07 STEP 2905 H=2905:00 +CPU= 26.681 19:05:35 STEP 2906 H=2906:00 +CPU= 26.266 19:06:20 STEP 2907 H=2907:00 +CPU= 42.898 19:07:04 STEP 2908 H=2908:00 +CPU= 41.235 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1995050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0715 (argv4) batchid: 964 (argv5) wuid: 12181359 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/10 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12181359 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_bf8807d3ab1da18b8cda64fdcb91ea4d to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0715_1995050100_123_964_12181359.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0715_1995050100_123_964_12181359.zip ic_ancil_file: ic_ancil_12181359 ifsdata_file: ifsdata_12181359 climate_data_file: clim_data_12181359 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_e33f194d446fdf5176278169e33b28eb to: /var/lib/boinc-client/slots/10/ic_ancil_12181359.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12181359.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ec633de182e05f42f6929883a755e89f to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12181359.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12181359.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12181359.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12181359.zip Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12181359.xml Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12181359.xml -- Model is restarting -- Adjusting last_iter, 2909, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12181359.xml last_cpu_time: 108134 upload_file_number: 121 last_iter: 2903 last_upload: 10454400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0715_1995050100_123_964_12181359_0_r1776975501 The child process has been launched with process id: 73100 Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:WAW22:1:1:73100:73100] [20230101:193046:1672597846:0.003] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 19:32:18 STEP 2904 H=2904:00 +CPU= 76.504 19:33:02 STEP 2905 H=2905:00 +CPU= 39.519 19:33:48 STEP 2906 H=2906:00 +CPU= 41.755 19:35:00 STEP 2907 H=2907:00 +CPU= 63.211 19:35:53 STEP 2908 H=2908:00 +CPU= 46.490 19:36:42 STEP 2909 H=2909:00 +CPU= 43.729 19:37:58 STEP 2910 H=2910:00 +CPU= 66.632 19:38:45 STEP 2911 H=2911:00 +CPU= 41.712 19:39:24 STEP 2912 H=2912:00 +CPU= 35.545 19:40:28 STEP 2913 H=2913:00 +CPU= 57.220 19:41:13 STEP 2914 H=2914:00 +CPU= 40.724 19:41:55 STEP 2915 H=2915:00 +CPU= 37.571 19:43:01 STEP 2916 H=2916:00 +CPU= 59.100 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+002916 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+002916 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+002916 19:43:37 STEP 2917 H=2917:00 +CPU= 33.036 19:44:15 STEP 2918 H=2918:00 +CPU= 34.007 19:45:09 STEP 2919 H=2919:00 +CPU= 49.514 19:45:48 STEP 2920 H=2920:00 +CPU= 36.092 19:46:26 STEP 2921 H=2921:00 +CPU= 34.219 19:47:23 STEP 2922 H=2922:00 +CPU= 52.337 19:48:00 STEP 2923 H=2923:00 +CPU= 33.949 19:48:38 STEP 2924 H=2924:00 +CPU= 34.679 19:49:37 STEP 2925 H=2925:00 +CPU= 53.596 19:50:19 STEP 2926 H=2926:00 +CPU= 37.252 19:50:57 STEP 2927 H=2927:00 +CPU= 34.881 Abort request received from BOINC client, ending the child process ..Failed, model did not complete successfully </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 16:05:02 | 1521081 | 22266388 | oifs_43r3_ps_0715_1995050100_123_964_12181359_0 | 9,417,600 | 95,344 | 0.0101 |
09 May 2023 16:04:10 | 1521081 | 22266388 | oifs_43r3_ps_0715_1995050100_123_964_12181359_0 | 8,553,600 | 87,089 | 0.0102 |
09 May 2023 16:04:02 | 1521081 | 22266388 | oifs_43r3_ps_0715_1995050100_123_964_12181359_0 | 7,689,600 | 78,789 | 0.0102 |
09 May 2023 16:03:54 | 1521081 | 22266388 | oifs_43r3_ps_0715_1995050100_123_964_12181359_0 | 6,825,600 | 70,770 | 0.0104 |
09 May 2023 16:03:49 | 1521081 | 22266388 | oifs_43r3_ps_0715_1995050100_123_964_12181359_0 | 5,961,600 | 62,342 | 0.0105 |
09 May 2023 16:03:40 | 1521081 | 22266388 | oifs_43r3_ps_0715_1995050100_123_964_12181359_0 | 5,097,600 | 53,469 | 0.0105 |
09 May 2023 16:03:34 | 1521081 | 22266388 | oifs_43r3_ps_0715_1995050100_123_964_12181359_0 | 4,233,600 | 44,356 | 0.0105 |
09 May 2023 15:48:22 | 1521081 | 22266388 | oifs_43r3_ps_0715_1995050100_123_964_12181359_0 | 2,592,000 | 24,751 | 0.0095 |
09 May 2023 15:48:09 | 1521081 | 22266388 | oifs_43r3_ps_0715_1995050100_123_964_12181359_0 | 1,728,000 | 16,147 | 0.0093 |
09 May 2023 15:47:58 | 1521081 | 22266388 | oifs_43r3_ps_0715_1995050100_123_964_12181359_0 | 864,000 | 8,298 | 0.0096 |
©2024 cpdn.org