Name | oifs_43r3_ps_0181_2003050100_123_972_12188825_1 |
Workunit | 12188825 |
Created | 24 Jan 2023, 17:04:32 UTC |
Sent | 24 Jan 2023, 17:35:43 UTC |
Report deadline | 25 Mar 2023, 17:35:43 UTC |
Received | 25 Jan 2023, 16:02:01 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 9 (0x00000009) Unknown error code |
Computer ID | 1538127 |
Run time | 1 hours 16 min |
CPU time | 1 hours 16 min |
Validate state | Invalid |
Credit | 1,814.40 |
Device peak FLOPS | 7.48 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,408.35 MB |
Peak swap size | 4,978.07 MB |
Peak disk usage | 2,977.95 MB |
Stderr | <core_client_version>7.22.0</core_client_version> <![CDATA[ <message> process exited with code 9 (0x9, -247)</message> <stderr_txt> he child process 18:57:57 STEP 79 H= 79:00 +CPU= 7.598 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:58:13 STEP 80 H= 80:00 +CPU= 7.862 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:58:33 STEP 81 H= 81:00 +CPU= 12.119 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:58:46 STEP 82 H= 82:00 +CPU= 7.758 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:58:58 STEP 83 H= 83:00 +CPU= 7.579 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:59:21 STEP 84 H= 84:00 +CPU= 13.315 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:59:33 STEP 85 H= 85:00 +CPU= 7.554 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000084 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000084 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000084 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:59:49 STEP 86 H= 86:00 +CPU= 7.826 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3992MB (heap), 4344MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1 [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1 [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil)) [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x3e800000000], nsigs = 1 [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1 [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3992 MB (maxheap), 4344 MB (maxrss), 0 MB (maxstack) [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.143] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0 [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1 [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2 [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3 [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4 [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS_DRV [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-PHYSICS CLDPP T/S (1002) [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : EC_PHYS [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALLPAR [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : TURBULENCE_LAYER [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : VDFOUTER [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190003:1674583203:1418.144] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : VDFMAIN [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190004:1674583204:1419.144] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1 [EC_DRHOOK:stefan-Z690-UD:1:1:1122:1122] [20230124:190004:1674583204:1419.144] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1 forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown oifs_43r3_model.e 0000000000FAA5EA vdfmain_ 874 vdfmain.F90 oifs_43r3_model.e 0000000000F92699 vdfouter_ 739 vdfouter.F90 oifs_43r3_model.e 0000000000F88512 turbulence_layer_ 158 turbulence_layer.F90 oifs_43r3_model.e 0000000000F0B683 callpar_ 869 callpar.F90 oifs_43r3_model.e 0000000000E4940F ec_phys_ 670 ec_phys.F90 oifs_43r3_model.e 0000000000E3610C ec_phys_drv_ 599 ec_phys_drv.F90 oifs_43r3_model.e 0000000000BFA3D3 gp_model_ 613 gp_model.F90 oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90 oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90 oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90 oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90 oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90 oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90 oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90 oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90 oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90 oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2003050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0181 (argv4) batchid: 972 (argv5) wuid: 12188825 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /home/stefan/slots/4 Project directory is: /home/stefan/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825 ..mkdir for temp folder for results failed Copying: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/stefan/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /home/stefan/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_943de426e6fe821666cd1e1c0f309465 to: /home/stefan/slots/4/oifs_43r3_ps_0181_2003050100_123_972_12188825.zip Unzipping the namelist zip file: /home/stefan/slots/4/oifs_43r3_ps_0181_2003050100_123_972_12188825.zip ic_ancil_file: ic_ancil_12188825 ifsdata_file: ifsdata_12188825 climate_data_file: clim_data_12188825 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_170d1551d97285d1db6fa8a53abf160c to: /home/stefan/slots/4/ic_ancil_12188825.zip Unzipping the IC ancils zip file: /home/stefan/slots/4/ic_ancil_12188825.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b440f6f96d74acf6a7feb76499bffa6b to: /home/stefan/slots/4/ifsdata/ifsdata_12188825.zip Unzipping the ifsdata_zip file: /home/stefan/slots/4/ifsdata/ifsdata_12188825.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/stefan/slots/4/159l_2/clim_data_12188825.zip Unzipping the climate data zip file: /home/stefan/slots/4/159l_2/clim_data_12188825.zip Checking for progress XML file: /home/stefan/slots/4/progress_file_12188825.xml Opened progress file ok : /home/stefan/slots/4/progress_file_12188825.xml -- Model is restarting -- Adjusting last_iter, 87, to previous model restart step. Creating progress file: /home/stefan/slots/4/progress_file_12188825.xml last_cpu_time: 837 upload_file_number: 3 last_iter: 71 last_upload: 259200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030 The child process has been launched with process id: 95 Executing the command: /home/stefan/slots/4/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at 0x1) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:stefan-Z690-UD:1:1:95:95] [20230125:153846:1674657526:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 15:39:01 STEP 72 H= 72:00 +CPU= 13.496 15:39:09 STEP 73 H= 73:00 +CPU= 7.590 15:39:16 STEP 74 H= 74:00 +CPU= 7.730 15:39:28 STEP 75 H= 75:00 +CPU= 12.080 15:39:36 STEP 76 H= 76:00 +CPU= 7.782 15:39:44 STEP 77 H= 77:00 +CPU= 7.505 15:39:56 STEP 78 H= 78:00 +CPU= 12.211 15:40:04 STEP 79 H= 79:00 +CPU= 7.885 15:40:12 STEP 80 H= 80:00 +CPU= 8.095 15:40:25 STEP 81 H= 81:00 +CPU= 12.578 15:40:33 STEP 82 H= 82:00 +CPU= 8.106 15:40:41 STEP 83 H= 83:00 +CPU= 7.912 15:40:55 STEP 84 H= 84:00 +CPU= 13.883 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000084 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000084 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000084 15:41:03 STEP 85 H= 85:00 +CPU= 7.915 15:41:11 STEP 86 H= 86:00 +CPU= 8.146 15:41:24 STEP 87 H= 87:00 +CPU= 12.567 15:41:32 STEP 88 H= 88:00 +CPU= 8.161 15:41:40 STEP 89 H= 89:00 +CPU= 7.971 15:41:53 STEP 90 H= 90:00 +CPU= 12.826 15:42:01 STEP 91 H= 91:00 +CPU= 7.967 15:42:09 STEP 92 H= 92:00 +CPU= 8.161 15:42:22 STEP 93 H= 93:00 +CPU= 12.613 15:42:30 STEP 94 H= 94:00 +CPU= 8.186 15:42:39 STEP 95 H= 95:00 +CPU= 8.193 15:42:54 STEP 96 H= 96:00 +CPU= 14.988 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000096 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000096 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000096 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000072 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000072 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000072 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000084 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000084 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000084 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_3.zip Uploading the intermediate file: upload_file_3.zip 15:43:02 STEP 97 H= 97:00 +CPU= 7.925 15:43:10 STEP 98 H= 98:00 +CPU= 8.121 15:43:23 STEP 99 H= 99:00 +CPU= 12.592 15:43:31 STEP 100 H= 100:00 +CPU= 8.139 15:43:39 STEP 101 H= 101:00 +CPU= 7.928 15:43:52 STEP 102 H= 102:00 +CPU= 12.871 15:44:00 STEP 103 H= 103:00 +CPU= 7.939 15:44:08 STEP 104 H= 104:00 +CPU= 8.136 15:44:21 STEP 105 H= 105:00 +CPU= 12.595 15:44:29 STEP 106 H= 106:00 +CPU= 8.142 15:44:37 STEP 107 H= 107:00 +CPU= 7.916 15:44:51 STEP 108 H= 108:00 +CPU= 13.950 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000108 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000108 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000108 15:44:59 STEP 109 H= 109:00 +CPU= 7.990 15:45:08 STEP 110 H= 110:00 +CPU= 8.202 15:45:20 STEP 111 H= 111:00 +CPU= 12.599 15:45:29 STEP 112 H= 112:00 +CPU= 8.160 15:45:37 STEP 113 H= 113:00 +CPU= 7.956 15:45:50 STEP 114 H= 114:00 +CPU= 12.870 15:45:58 STEP 115 H= 115:00 +CPU= 7.976 15:46:06 STEP 116 H= 116:00 +CPU= 8.156 15:46:19 STEP 117 H= 117:00 +CPU= 12.585 15:46:27 STEP 118 H= 118:00 +CPU= 8.155 15:46:35 STEP 119 H= 119:00 +CPU= 8.150 15:46:50 STEP 120 H= 120:00 +CPU= 14.992 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000120 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000120 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000120 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000096 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000096 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000096 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000108 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000108 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000108 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_4.zip Uploading the intermediate file: upload_file_4.zip 15:46:58 STEP 121 H= 121:00 +CPU= 7.932 15:47:06 STEP 122 H= 122:00 +CPU= 8.146 15:47:19 STEP 123 H= 123:00 +CPU= 12.627 15:47:27 STEP 124 H= 124:00 +CPU= 8.165 15:47:35 STEP 125 H= 125:00 +CPU= 7.953 15:47:48 STEP 126 H= 126:00 +CPU= 12.845 15:47:57 STEP 127 H= 127:00 +CPU= 8.029 15:48:05 STEP 128 H= 128:00 +CPU= 8.196 15:48:18 STEP 129 H= 129:00 +CPU= 12.610 15:48:26 STEP 130 H= 130:00 +CPU= 8.153 15:48:34 STEP 131 H= 131:00 +CPU= 7.979 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2003050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0181 (argv4) batchid: 972 (argv5) wuid: 12188825 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /home/stefan/slots/4 Project directory is: /home/stefan/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825 ..mkdir for temp folder for results failed Copying: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/stefan/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /home/stefan/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_943de426e6fe821666cd1e1c0f309465 to: /home/stefan/slots/4/oifs_43r3_ps_0181_2003050100_123_972_12188825.zip Unzipping the namelist zip file: /home/stefan/slots/4/oifs_43r3_ps_0181_2003050100_123_972_12188825.zip ic_ancil_file: ic_ancil_12188825 ifsdata_file: ifsdata_12188825 climate_data_file: clim_data_12188825 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_170d1551d97285d1db6fa8a53abf160c to: /home/stefan/slots/4/ic_ancil_12188825.zip Unzipping the IC ancils zip file: /home/stefan/slots/4/ic_ancil_12188825.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b440f6f96d74acf6a7feb76499bffa6b to: /home/stefan/slots/4/ifsdata/ifsdata_12188825.zip Unzipping the ifsdata_zip file: /home/stefan/slots/4/ifsdata/ifsdata_12188825.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/stefan/slots/4/159l_2/clim_data_12188825.zip Unzipping the climate data zip file: /home/stefan/slots/4/159l_2/clim_data_12188825.zip Checking for progress XML file: /home/stefan/slots/4/progress_file_12188825.xml Opened progress file ok : /home/stefan/slots/4/progress_file_12188825.xml -- Model is restarting -- Adjusting last_iter, 132, to previous model restart step. Creating progress file: /home/stefan/slots/4/progress_file_12188825.xml last_cpu_time: 1422 upload_file_number: 5 last_iter: 119 last_upload: 432000 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030 The child process has been launched with process id: 81 Executing the command: /home/stefan/slots/4/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at 0x1) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:stefan-Z690-UD:1:1:81:81] [20230125:155016:1674658216:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 15:50:32 STEP 120 H= 120:00 +CPU= 14.257 15:50:40 STEP 121 H= 121:00 +CPU= 8.040 15:50:48 STEP 122 H= 122:00 +CPU= 8.258 15:51:01 STEP 123 H= 123:00 +CPU= 12.756 15:51:09 STEP 124 H= 124:00 +CPU= 8.207 15:51:18 STEP 125 H= 125:00 +CPU= 8.076 15:51:31 STEP 126 H= 126:00 +CPU= 12.990 15:51:39 STEP 127 H= 127:00 +CPU= 8.035 15:51:47 STEP 128 H= 128:00 +CPU= 8.227 15:52:00 STEP 129 H= 129:00 +CPU= 12.711 15:52:08 STEP 130 H= 130:00 +CPU= 8.248 15:52:16 STEP 131 H= 131:00 +CPU= 8.053 15:52:31 STEP 132 H= 132:00 +CPU= 14.046 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000132 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000132 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000132 15:52:39 STEP 133 H= 133:00 +CPU= 8.050 15:52:47 STEP 134 H= 134:00 +CPU= 8.253 15:53:00 STEP 135 H= 135:00 +CPU= 12.741 15:53:08 STEP 136 H= 136:00 +CPU= 8.248 15:53:17 STEP 137 H= 137:00 +CPU= 8.039 15:53:30 STEP 138 H= 138:00 +CPU= 13.031 15:53:38 STEP 139 H= 139:00 +CPU= 8.063 15:53:46 STEP 140 H= 140:00 +CPU= 8.221 15:53:59 STEP 141 H= 141:00 +CPU= 12.694 15:54:07 STEP 142 H= 142:00 +CPU= 8.220 15:54:16 STEP 143 H= 143:00 +CPU= 8.233 15:54:31 STEP 144 H= 144:00 +CPU= 15.146 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000144 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000144 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000144 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000120 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000120 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000120 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000132 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000132 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000132 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_5.zip Uploading the intermediate file: upload_file_5.zip 15:54:39 STEP 145 H= 145:00 +CPU= 8.033 15:54:48 STEP 146 H= 146:00 +CPU= 8.254 15:55:01 STEP 147 H= 147:00 +CPU= 12.765 15:55:09 STEP 148 H= 148:00 +CPU= 8.240 15:55:17 STEP 149 H= 149:00 +CPU= 8.021 15:55:30 STEP 150 H= 150:00 +CPU= 12.935 15:55:38 STEP 151 H= 151:00 +CPU= 8.032 15:55:47 STEP 152 H= 152:00 +CPU= 8.243 15:56:00 STEP 153 H= 153:00 +CPU= 12.793 15:56:08 STEP 154 H= 154:00 +CPU= 8.235 15:56:16 STEP 155 H= 155:00 +CPU= 8.034 15:56:30 STEP 156 H= 156:00 +CPU= 14.020 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000156 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000156 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000156 15:56:38 STEP 157 H= 157:00 +CPU= 8.039 15:56:47 STEP 158 H= 158:00 +CPU= 8.204 15:57:00 STEP 159 H= 159:00 +CPU= 12.720 15:57:08 STEP 160 H= 160:00 +CPU= 8.280 15:57:16 STEP 161 H= 161:00 +CPU= 8.055 15:57:29 STEP 162 H= 162:00 +CPU= 12.977 15:57:37 STEP 163 H= 163:00 +CPU= 8.065 15:57:46 STEP 164 H= 164:00 +CPU= 8.267 15:57:59 STEP 165 H= 165:00 +CPU= 12.781 15:58:07 STEP 166 H= 166:00 +CPU= 8.231 15:58:15 STEP 167 H= 167:00 +CPU= 8.224 15:58:31 STEP 168 H= 168:00 +CPU= 15.229 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000168 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000168 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000168 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000144 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000144 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000144 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000156 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000156 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000156 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_6.zip Uploading the intermediate file: upload_file_6.zip 15:58:39 STEP 169 H= 169:00 +CPU= 7.997 15:58:47 STEP 170 H= 170:00 +CPU= 8.198 15:59:00 STEP 171 H= 171:00 +CPU= 12.697 15:59:08 STEP 172 H= 172:00 +CPU= 8.211 15:59:16 STEP 173 H= 173:00 +CPU= 7.716 15:59:29 STEP 174 H= 174:00 +CPU= 12.443 15:59:36 STEP 175 H= 175:00 +CPU= 7.699 15:59:44 STEP 176 H= 176:00 +CPU= 7.844 15:59:56 STEP 177 H= 177:00 +CPU= 12.192 16:00:04 STEP 178 H= 178:00 +CPU= 7.855 16:00:12 STEP 179 H= 179:00 +CPU= 7.660 16:00:25 STEP 180 H= 180:00 +CPU= 13.364 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000180 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000180 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000180 16:00:33 STEP 181 H= 181:00 +CPU= 7.621 16:00:41 STEP 182 H= 182:00 +CPU= 7.791 16:00:53 STEP 183 H= 183:00 +CPU= 12.039 16:01:01 STEP 184 H= 184:00 +CPU= 7.828 16:01:08 STEP 185 H= 185:00 +CPU= 7.618 16:01:21 STEP 186 H= 186:00 +CPU= 12.408 16:01:28 STEP 187 H= 187:00 +CPU= 7.715 16:01:36 STEP 188 H= 188:00 +CPU= 7.798 16:01:48 STEP 189 H= 189:00 +CPU= 12.069 16:01:56 STEP 190 H= 190:00 +CPU= 7.766 16:02:04 STEP 191 H= 191:00 +CPU= 7.799 16:02:18 STEP 192 H= 192:00 +CPU= 14.598 16:02:26 STEP 193 H= 193:00 +CPU= 7.687 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000192 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000192 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000192 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000168 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000168 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000168 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000180 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000180 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000180 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_7.zip Uploading the intermediate file: upload_file_7.zip 16:02:34 STEP 194 H= 194:00 +CPU= 7.927 16:02:46 STEP 195 H= 195:00 +CPU= 12.413 16:02:54 STEP 196 H= 196:00 +CPU= 7.917 16:03:02 STEP 197 H= 197:00 +CPU= 7.663 16:03:14 STEP 198 H= 198:00 +CPU= 12.522 16:03:22 STEP 199 H= 199:00 +CPU= 7.585 16:03:30 STEP 200 H= 200:00 +CPU= 7.797 16:03:42 STEP 201 H= 201:00 +CPU= 12.176 16:03:50 STEP 202 H= 202:00 +CPU= 7.929 16:03:58 STEP 203 H= 203:00 +CPU= 7.781 16:04:11 STEP 204 H= 204:00 +CPU= 13.555 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000204 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000204 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000204 16:04:19 STEP 205 H= 205:00 +CPU= 7.695 16:04:27 STEP 206 H= 206:00 +CPU= 8.034 16:04:40 STEP 207 H= 207:00 +CPU= 12.762 16:04:48 STEP 208 H= 208:00 +CPU= 8.276 16:04:57 STEP 209 H= 209:00 +CPU= 8.032 16:05:10 STEP 210 H= 210:00 +CPU= 13.006 16:05:18 STEP 211 H= 211:00 +CPU= 8.056 16:05:26 STEP 212 H= 212:00 +CPU= 8.286 16:05:39 STEP 213 H= 213:00 +CPU= 12.743 16:05:47 STEP 214 H= 214:00 +CPU= 8.259 16:05:56 STEP 215 H= 215:00 +CPU= 8.196 16:06:11 STEP 216 H= 216:00 +CPU= 15.279 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000216 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000216 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000216 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000192 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000192 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000192 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000204 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000204 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000204 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_8.zip Uploading the intermediate file: upload_file_8.zip 16:06:19 STEP 217 H= 217:00 +CPU= 8.069 16:06:28 STEP 218 H= 218:00 +CPU= 8.264 16:06:41 STEP 219 H= 219:00 +CPU= 12.778 16:06:49 STEP 220 H= 220:00 +CPU= 8.244 16:06:57 STEP 221 H= 221:00 +CPU= 8.044 16:07:10 STEP 222 H= 222:00 +CPU= 13.013 16:07:19 STEP 223 H= 223:00 +CPU= 8.074 16:07:27 STEP 224 H= 224:00 +CPU= 8.255 16:07:40 STEP 225 H= 225:00 +CPU= 12.744 16:07:48 STEP 226 H= 226:00 +CPU= 8.280 16:07:56 STEP 227 H= 227:00 +CPU= 8.075 16:08:11 STEP 228 H= 228:00 +CPU= 14.073 16:08:19 STEP 229 H= 229:00 +CPU= 8.054 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000228 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000228 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000228 16:08:27 STEP 230 H= 230:00 +CPU= 8.249 16:08:40 STEP 231 H= 231:00 +CPU= 12.765 16:08:48 STEP 232 H= 232:00 +CPU= 8.281 16:08:56 STEP 233 H= 233:00 +CPU= 8.032 16:09:10 STEP 234 H= 234:00 +CPU= 12.957 16:09:18 STEP 235 H= 235:00 +CPU= 8.049 16:09:26 STEP 236 H= 236:00 +CPU= 8.211 16:09:39 STEP 237 H= 237:00 +CPU= 12.733 16:09:47 STEP 238 H= 238:00 +CPU= 8.261 16:09:55 STEP 239 H= 239:00 +CPU= 8.117 16:10:10 STEP 240 H= 240:00 +CPU= 14.846 16:10:18 STEP 241 H= 241:00 +CPU= 7.861 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000240 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000240 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000240 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000216 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000216 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000216 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000228 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000228 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000228 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_9.zip Uploading the intermediate file: upload_file_9.zip 16:10:26 STEP 242 H= 242:00 +CPU= 8.055 16:10:39 STEP 243 H= 243:00 +CPU= 12.414 16:10:47 STEP 244 H= 244:00 +CPU= 8.000 16:10:55 STEP 245 H= 245:00 +CPU= 7.783 16:11:07 STEP 246 H= 246:00 +CPU= 12.666 16:11:15 STEP 247 H= 247:00 +CPU= 7.813 16:11:23 STEP 248 H= 248:00 +CPU= 8.076 16:11:36 STEP 249 H= 249:00 +CPU= 12.407 16:11:44 STEP 250 H= 250:00 +CPU= 8.017 16:11:52 STEP 251 H= 251:00 +CPU= 7.829 16:12:05 STEP 252 H= 252:00 +CPU= 13.735 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000252 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000252 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000252 16:12:13 STEP 253 H= 253:00 +CPU= 7.904 16:12:21 STEP 254 H= 254:00 +CPU= 8.049 16:12:34 STEP 255 H= 255:00 +CPU= 12.434 16:12:42 STEP 256 H= 256:00 +CPU= 8.031 16:12:50 STEP 257 H= 257:00 +CPU= 7.846 16:13:02 STEP 258 H= 258:00 +CPU= 12.591 16:13:10 STEP 259 H= 259:00 +CPU= 7.835 16:13:18 STEP 260 H= 260:00 +CPU= 7.997 16:13:31 STEP 261 H= 261:00 +CPU= 12.382 16:13:39 STEP 262 H= 262:00 +CPU= 8.073 16:13:47 STEP 263 H= 263:00 +CPU= 8.004 16:14:02 STEP 264 H= 264:00 +CPU= 14.824 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000264 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000264 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000264 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000240 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000240 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000240 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000252 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000252 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000252 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_10.zip 16:14:10 STEP 265 H= 265:00 +CPU= 7.823 Uploading the intermediate file: upload_file_10.zip 16:14:18 STEP 266 H= 266:00 +CPU= 8.047 16:14:30 STEP 267 H= 267:00 +CPU= 12.437 16:14:38 STEP 268 H= 268:00 +CPU= 8.085 16:14:46 STEP 269 H= 269:00 +CPU= 7.874 16:14:59 STEP 270 H= 270:00 +CPU= 12.672 16:15:07 STEP 271 H= 271:00 +CPU= 7.834 16:15:15 STEP 272 H= 272:00 +CPU= 8.068 16:15:27 STEP 273 H= 273:00 +CPU= 12.478 16:15:35 STEP 274 H= 274:00 +CPU= 8.037 16:15:43 STEP 275 H= 275:00 +CPU= 7.868 16:15:57 STEP 276 H= 276:00 +CPU= 13.734 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000276 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000276 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000276 16:16:05 STEP 277 H= 277:00 +CPU= 7.853 16:16:13 STEP 278 H= 278:00 +CPU= 8.068 16:16:26 STEP 279 H= 279:00 +CPU= 12.408 16:16:34 STEP 280 H= 280:00 +CPU= 8.035 16:16:41 STEP 281 H= 281:00 +CPU= 7.826 16:16:54 STEP 282 H= 282:00 +CPU= 12.589 16:17:02 STEP 283 H= 283:00 +CPU= 7.909 16:17:10 STEP 284 H= 284:00 +CPU= 8.055 16:17:23 STEP 285 H= 285:00 +CPU= 12.365 16:17:31 STEP 286 H= 286:00 +CPU= 7.974 16:17:39 STEP 287 H= 287:00 +CPU= 7.979 16:17:53 STEP 288 H= 288:00 +CPU= 14.905 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000288 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000288 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000288 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000264 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000264 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000264 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000276 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000276 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000276 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_11.zip Uploading the intermediate file: upload_file_11.zip 16:18:01 STEP 289 H= 289:00 +CPU= 7.765 16:18:09 STEP 290 H= 290:00 +CPU= 7.975 16:18:22 STEP 291 H= 291:00 +CPU= 12.368 16:18:30 STEP 292 H= 292:00 +CPU= 7.988 16:18:37 STEP 293 H= 293:00 +CPU= 7.737 16:18:50 STEP 294 H= 294:00 +CPU= 12.621 16:18:58 STEP 295 H= 295:00 +CPU= 7.876 16:19:06 STEP 296 H= 296:00 +CPU= 7.982 16:19:19 STEP 297 H= 297:00 +CPU= 12.436 16:19:27 STEP 298 H= 298:00 +CPU= 7.970 16:19:34 STEP 299 H= 299:00 +CPU= 7.777 16:19:48 STEP 300 H= 300:00 +CPU= 13.702 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000300 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000300 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000300 16:19:56 STEP 301 H= 301:00 +CPU= 7.778 16:20:04 STEP 302 H= 302:00 +CPU= 8.086 16:20:16 STEP 303 H= 303:00 +CPU= 12.343 16:20:24 STEP 304 H= 304:00 +CPU= 8.072 16:20:32 STEP 305 H= 305:00 +CPU= 7.854 16:20:45 STEP 306 H= 306:00 +CPU= 12.632 16:20:53 STEP 307 H= 307:00 +CPU= 7.881 16:21:01 STEP 308 H= 308:00 +CPU= 8.050 16:21:13 STEP 309 H= 309:00 +CPU= 12.401 16:21:21 STEP 310 H= 310:00 +CPU= 8.060 16:21:30 STEP 311 H= 311:00 +CPU= 8.039 16:21:44 STEP 312 H= 312:00 +CPU= 14.816 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000312 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000312 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000312 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000288 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000288 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000288 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000300 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000300 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000300 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_12.zip Uploading the intermediate file: upload_file_12.zip 16:21:52 STEP 313 H= 313:00 +CPU= 7.835 16:22:00 STEP 314 H= 314:00 +CPU= 8.050 16:22:13 STEP 315 H= 315:00 +CPU= 12.420 16:22:21 STEP 316 H= 316:00 +CPU= 7.992 16:22:29 STEP 317 H= 317:00 +CPU= 7.859 16:22:41 STEP 318 H= 318:00 +CPU= 12.622 16:22:49 STEP 319 H= 319:00 +CPU= 7.773 16:22:57 STEP 320 H= 320:00 +CPU= 8.064 16:23:10 STEP 321 H= 321:00 +CPU= 12.385 16:23:18 STEP 322 H= 322:00 +CPU= 8.031 16:23:26 STEP 323 H= 323:00 +CPU= 7.770 16:23:39 STEP 324 H= 324:00 +CPU= 13.685 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000324 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000324 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000324 16:23:47 STEP 325 H= 325:00 +CPU= 7.865 16:23:55 STEP 326 H= 326:00 +CPU= 8.090 16:24:08 STEP 327 H= 327:00 +CPU= 12.427 16:24:16 STEP 328 H= 328:00 +CPU= 7.990 16:24:24 STEP 329 H= 329:00 +CPU= 7.819 16:24:36 STEP 330 H= 330:00 +CPU= 12.695 16:24:44 STEP 331 H= 331:00 +CPU= 7.857 16:24:52 STEP 332 H= 332:00 +CPU= 7.952 16:25:05 STEP 333 H= 333:00 +CPU= 12.381 16:25:13 STEP 334 H= 334:00 +CPU= 7.998 16:25:21 STEP 335 H= 335:00 +CPU= 8.077 16:25:36 STEP 336 H= 336:00 +CPU= 14.835 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000336 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000336 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000336 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000312 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000312 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000312 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000324 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000324 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000324 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_13.zip Uploading the intermediate file: upload_file_13.zip 16:25:43 STEP 337 H= 337:00 +CPU= 7.848 16:25:52 STEP 338 H= 338:00 +CPU= 8.031 16:26:04 STEP 339 H= 339:00 +CPU= 12.432 16:26:12 STEP 340 H= 340:00 +CPU= 8.003 16:26:20 STEP 341 H= 341:00 +CPU= 7.841 16:26:33 STEP 342 H= 342:00 +CPU= 12.682 16:26:40 STEP 343 H= 343:00 +CPU= 7.829 16:26:49 STEP 344 H= 344:00 +CPU= 8.048 16:27:01 STEP 345 H= 345:00 +CPU= 12.456 16:27:09 STEP 346 H= 346:00 +CPU= 8.032 16:27:17 STEP 347 H= 347:00 +CPU= 7.814 16:27:31 STEP 348 H= 348:00 +CPU= 13.776 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000348 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000348 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000348 16:27:39 STEP 349 H= 349:00 +CPU= 7.846 16:27:47 STEP 350 H= 350:00 +CPU= 7.961 16:27:59 STEP 351 H= 351:00 +CPU= 12.433 16:28:07 STEP 352 H= 352:00 +CPU= 8.000 16:28:15 STEP 353 H= 353:00 +CPU= 7.842 16:28:28 STEP 354 H= 354:00 +CPU= 12.604 16:28:35 STEP 355 H= 355:00 +CPU= 7.801 16:28:44 STEP 356 H= 356:00 +CPU= 8.010 16:28:56 STEP 357 H= 357:00 +CPU= 12.372 16:29:04 STEP 358 H= 358:00 +CPU= 8.035 16:29:12 STEP 359 H= 359:00 +CPU= 7.985 16:29:27 STEP 360 H= 360:00 +CPU= 14.682 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000360 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000360 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000360 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000336 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000336 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000336 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000348 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000348 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000348 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_14.zip Uploading the intermediate file: upload_file_14.zip 16:29:35 STEP 361 H= 361:00 +CPU= 7.786 16:29:43 STEP 362 H= 362:00 +CPU= 8.005 Uploading trickle at timestep: 1296000 16:29:55 STEP 363 H= 363:00 +CPU= 12.437 16:30:03 STEP 364 H= 364:00 +CPU= 7.988 16:30:11 STEP 365 H= 365:00 +CPU= 7.773 16:30:24 STEP 366 H= 366:00 +CPU= 12.631 16:30:31 STEP 367 H= 367:00 +CPU= 7.779 16:30:39 STEP 368 H= 368:00 +CPU= 8.051 16:30:52 STEP 369 H= 369:00 +CPU= 12.378 16:31:00 STEP 370 H= 370:00 +CPU= 7.965 16:31:08 STEP 371 H= 371:00 +CPU= 7.860 16:31:21 STEP 372 H= 372:00 +CPU= 13.670 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000372 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000372 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000372 16:31:29 STEP 373 H= 373:00 +CPU= 7.821 16:31:37 STEP 374 H= 374:00 +CPU= 7.978 16:31:50 STEP 375 H= 375:00 +CPU= 12.351 16:31:58 STEP 376 H= 376:00 +CPU= 8.005 16:32:06 STEP 377 H= 377:00 +CPU= 7.764 16:32:18 STEP 378 H= 378:00 +CPU= 12.653 16:32:26 STEP 379 H= 379:00 +CPU= 7.817 16:32:34 STEP 380 H= 380:00 +CPU= 8.029 16:32:47 STEP 381 H= 381:00 +CPU= 12.360 16:32:55 STEP 382 H= 382:00 +CPU= 8.020 16:33:03 STEP 383 H= 383:00 +CPU= 8.037 16:33:17 STEP 384 H= 384:00 +CPU= 14.706 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000384 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000384 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000384 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000360 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000360 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000360 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000372 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000372 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000372 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_15.zip Uploading the intermediate file: upload_file_15.zip 16:33:25 STEP 385 H= 385:00 +CPU= 7.809 16:33:33 STEP 386 H= 386:00 +CPU= 8.024 16:33:46 STEP 387 H= 387:00 +CPU= 12.365 16:33:54 STEP 388 H= 388:00 +CPU= 7.990 16:34:02 STEP 389 H= 389:00 +CPU= 7.820 16:34:14 STEP 390 H= 390:00 +CPU= 12.458 16:34:22 STEP 391 H= 391:00 +CPU= 7.795 16:34:30 STEP 392 H= 392:00 +CPU= 8.000 16:34:42 STEP 393 H= 393:00 +CPU= 12.380 16:34:50 STEP 394 H= 394:00 +CPU= 8.048 16:34:58 STEP 395 H= 395:00 +CPU= 7.882 16:35:12 STEP 396 H= 396:00 +CPU= 13.722 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000396 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000396 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000396 16:35:20 STEP 397 H= 397:00 +CPU= 7.830 16:35:28 STEP 398 H= 398:00 +CPU= 8.077 16:35:40 STEP 399 H= 399:00 +CPU= 12.479 16:35:49 STEP 400 H= 400:00 +CPU= 8.060 16:35:56 STEP 401 H= 401:00 +CPU= 7.836 16:36:09 STEP 402 H= 402:00 +CPU= 12.618 16:36:17 STEP 403 H= 403:00 +CPU= 7.856 16:36:25 STEP 404 H= 404:00 +CPU= 8.086 16:36:37 STEP 405 H= 405:00 +CPU= 12.291 16:36:45 STEP 406 H= 406:00 +CPU= 8.016 16:36:53 STEP 407 H= 407:00 +CPU= 8.012 16:37:08 STEP 408 H= 408:00 +CPU= 14.797 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000408 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000408 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000408 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000384 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000384 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000384 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000396 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000396 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000396 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_16.zip Uploading the intermediate file: upload_file_16.zip 16:37:16 STEP 409 H= 409:00 +CPU= 7.789 16:37:24 STEP 410 H= 410:00 +CPU= 8.027 16:37:37 STEP 411 H= 411:00 +CPU= 12.446 16:37:45 STEP 412 H= 412:00 +CPU= 8.023 16:37:53 STEP 413 H= 413:00 +CPU= 7.786 16:38:05 STEP 414 H= 414:00 +CPU= 12.699 16:38:13 STEP 415 H= 415:00 +CPU= 7.805 16:38:21 STEP 416 H= 416:00 +CPU= 8.003 16:38:34 STEP 417 H= 417:00 +CPU= 12.415 16:38:42 STEP 418 H= 418:00 +CPU= 8.031 16:38:50 STEP 419 H= 419:00 +CPU= 7.788 16:39:03 STEP 420 H= 420:00 +CPU= 13.647 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000420 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000420 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000420 16:39:11 STEP 421 H= 421:00 +CPU= 7.818 16:39:19 STEP 422 H= 422:00 +CPU= 8.003 16:39:32 STEP 423 H= 423:00 +CPU= 12.379 16:39:40 STEP 424 H= 424:00 +CPU= 7.987 16:39:47 STEP 425 H= 425:00 +CPU= 7.834 16:40:00 STEP 426 H= 426:00 +CPU= 12.622 16:40:08 STEP 427 H= 427:00 +CPU= 7.745 16:40:16 STEP 428 H= 428:00 +CPU= 7.986 16:40:28 STEP 429 H= 429:00 +CPU= 12.404 16:40:36 STEP 430 H= 430:00 +CPU= 7.992 16:40:44 STEP 431 H= 431:00 +CPU= 7.992 16:40:59 STEP 432 H= 432:00 +CPU= 14.773 Moving to projects directory: /home/stefan/slots/4/ICMGGhq0f+000432 Moving to projects directory: /home/stefan/slots/4/ICMSHhq0f+000432 Moving to projects directory: /home/stefan/slots/4/ICMUAhq0f+000432 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000408 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000408 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000408 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMGGhq0f+000420 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMSHhq0f+000420 Adding to the zip: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_12188825/ICMUAhq0f+000420 Zipping up the intermediate file: /home/stefan/projects/climateprediction.net/oifs_43r3_ps_0181_2003050100_123_972_12188825_1_r51269030_17.zip Uploading the intermediate file: upload_file_17.zip 16:41:07 STEP 433 H= 433:00 +CPU= 7.745 16:41:15 STEP 434 H= 434:00 +CPU= 7.950 16:41:27 STEP 435 H= 435:00 +CPU= 12.324 16:41:35 STEP 436 H= 436:00 +CPU= 8.039 16:41:43 STEP 437 H= 437:00 +CPU= 7.833 16:41:56 STEP 438 H= 438:00 +CPU= 12.541 16:42:04 STEP 439 H= 439:00 +CPU= 7.794 16:42:12 STEP 440 H= 440:00 +CPU= 7.985 16:42:24 STEP 441 H= 441:00 +CPU= 12.406 16:42:32 STEP 442 H= 442:00 +CPU= 8.022 16:42:40 STEP 443 H= 443:00 +CPU= 7.812 double free or corruption (out) </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 16:40:56 | 1538127 | 22303014 | oifs_43r3_ps_0181_2003050100_123_972_12188825_1 | 1,296,000 | 3,762 | 0.0029 |
©2024 cpdn.org