Name | oifs_43r3_bl_a0tp_2016092300_15_991_12207856_0 |
Workunit | 12207856 |
Created | 13 Feb 2023, 13:59:05 UTC |
Sent | 14 Feb 2023, 8:36:29 UTC |
Report deadline | 15 Apr 2023, 8:36:29 UTC |
Received | 14 Feb 2023, 9:37:15 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 1 (0x00000001) Unknown error code |
Computer ID | 1531276 |
Run time | 58 sec |
CPU time | 7 sec |
Validate state | Invalid |
Credit | 0.00 |
Device peak FLOPS | 4.15 GFLOPS |
Application version | OpenIFS 43r3 Baroclinic Lifecycle v1.11 x86_64-pc-linux-gnu |
Peak working set size | 1,645.94 MB |
Peak swap size | 2,584.93 MB |
Peak disk usage | 270.76 MB |
Stderr | <core_client_version>7.16.11</core_client_version> <![CDATA[ <message> process exited with code 1 (0x1, -255)</message> <stderr_txt> (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a0tp (argv4) batchid: 991 (argv5) wuid: 12207856 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/3 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.11 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12207856 Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/3/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/3/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_0d0da87787619fb24be610cb9ffe3cd0 to: /var/lib/boinc/slots/3/oifs_43r3_bl_a0tp_2016092300_15_991_12207856.zip Unzipping the namelist zip file: /var/lib/boinc/slots/3/oifs_43r3_bl_a0tp_2016092300_15_991_12207856.zip ic_ancil_file: ic_ancil_12207856 ifsdata_file: ifsdata_12207856 climate_data_file: clim_data_12207856 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/3/ic_ancil_12207856.zip Unzipping the IC ancils zip file: /var/lib/boinc/slots/3/ic_ancil_12207856.zip Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b9c7340450981f16b3d3ea3f5a14e594 to: /var/lib/boinc/slots/3/ifsdata/ifsdata_12207856.zip Unzipping the ifsdata_zip file: /var/lib/boinc/slots/3/ifsdata/ifsdata_12207856.zip Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/3/159l_2/clim_data_12207856.zip Unzipping the climate data zip file: /var/lib/boinc/slots/3/159l_2/clim_data_12207856.zip Checking for progress XML file: /var/lib/boinc/slots/3/progress_file_12207856.xml Creating progress file: /var/lib/boinc/slots/3/progress_file_12207856.xml last_cpu_time: 0 upload_file_number: 0 last_iter: 0 last_upload: 0 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a0tp_2016092300_15_991_12207856_0_r608925393 The child process has been launched with process id: 4863 Executing the command: /var/lib/boinc/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00 [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil)) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil)) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil)) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil)) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil)) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil)) [EC_DRHOOK:sagan.gx63.info:1:1:4863:4863] [20230214:033754:1676363874:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process forrtl: severe (41): insufficient virtual memory Image PC Routine Line Source oifs_43r3_model.e 00000000015C1B59 gp_model_heap_ 71 gp_model_heap.F90 oifs_43r3_model.e 00000000015AC40E scan2m_ 535 scan2m.F90 oifs_43r3_model.e 00000000006620B4 stepo_ 327 stepo.F90 oifs_43r3_model.e 00000000004A0E7C cnt4_ 1133 cnt4.F90 oifs_43r3_model.e 000000000049DC29 cnt3_ 267 cnt3.F90 oifs_43r3_model.e 000000000049CE72 cnt2_ 88 cnt2.F90 oifs_43r3_model.e 000000000049CD68 cnt1_ 92 cnt1.F90 oifs_43r3_model.e 00000000004078AD cnt0_ 146 cnt0.F90 Stack trace terminated abnormally. ..The child process terminated with status: 41 >>> Printing last 70 lines from file: NODE.001_01 Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc UPDTIM MACC AEROSOL CLIMATOLOGY IS LOADED SUECAEC: INLEV,ISYEAR,IEYEAR= 0 1957 2010 suecaec NAERMACC = 1 UPDT RAESHDU = 2000.0 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT) 20160923 90 9.999999999999998E-005 DATE= 2016 9 23 NORMS AT NSTEP CNT4 0 SPECTRAL NORMS - LOG(PREHYDS) 0.115129254649702E+02 LEV VORTICITY DIVERGENCE TEMPERATURE KINETIC ENERGY AVE 0.104364207820444E-04 0.555153323095659E-07 0.211481499112470E+03 0.203704065489698E+03 GPNORM HUMIDITY AVERAGE MINIMUM MAXIMUM AVE 0.943104781391838E-03 0.000000000000000E+00 0.688353493731124E-02 GPNORM LIQUID WATER AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM ICE WATER AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM SNOW AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM RAIN AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM CLOUD FRACTION AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM CLW AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM CIW AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM CC AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM RFL AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM SFL AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM CRFL AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM CSFL AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM PFRC AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 NSTEP = 0 STEPO A00000000 MAXGPFV : MAX. VALUE = 0.000000000000000E+000 MAXGPFV : MAX. VALUE = 0.000000000000000E+000 NSTEP = 0 SCAN2M_HPOS P IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH = ICMGGh7zg+000000 MODE=w IO-STREAM CLOSED - ICMGGh7zg+000000 MPI-TASK: 1 - 634316698 BYTES IN 6 RECORDS TRANSFERRED IN 0.0003 SECONDS******** Mbytes/s, TOTAL TIME= 0.0285( 1.1%) NSTEP = 0 STEPO 0AA000000 IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH = ./ICMSHh7zg+000000 MODE=a IO-STREAM CLOSED - ./ICMSHh7zg+000000 MPI-TASK: 1 - 707765250 BYTES IN 202 RECORDS TRANSFERRED IN 0.0053 SECONDS133540.6 Mbytes/s, TOTAL TIME= 0.1898( 2.8%) IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH = ./ICMUAh7zg+000000 MODE=a IO-STREAM CLOSED - ./ICMUAh7zg+000000 MPI-TASK: 1 - 5143202967 BYTES IN 125 RECORDS TRANSFERRED IN 0.0018 SECONDS******** Mbytes/s, TOTAL TIME= 0.0848( 2.1%) STEPO_FPOS WAS : B00000F00E IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH = ./ICMUAh7zg+000000 MODE=a IO-STREAM CLOSED - ./ICMUAh7zg+000000 MPI-TASK: 1 - 312876553 BYTES IN 2 RECORDS TRANSFERRED IN 0.0001 SECONDS******** Mbytes/s, TOTAL TIME= 0.0035( 2.9%) STEPO_FPOS WAS : V00000000Y IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH = ./ICMUAh7zg+000000 MODE=a IO-STREAM CLOSED - ./ICMUAh7zg+000000 MPI-TASK: 1 - 321450186 BYTES IN 4 RECORDS TRANSFERRED IN 0.0002 SECONDS******** Mbytes/s, TOTAL TIME= 0.0051( 3.9%) STEPO_FPOS WAS : T00000000M NSTEP = 0 STEPO 0AAA00AAA ------------------------------------------------ oifs_get_stat: Error. ifs.stat file is not open CNT0 not found; string returned was: '' >>> Printing last 7 lines from file: ifs.stat 03:37:54 000000000 CNT3 -999 0.292 0.292 0.292 0:00 0:00 0.00000000000000E+00 1GB 0MB 03:37:56 A00000000 STEPO 0 2.433 2.433 2.444 0:02 0:02 0.55515332309566E-07 2GB 0MB 03:37:56 0AA000000 STEPO 0 0.036 0.036 0.048 0:02 0:02 0.55515332309566E-07 2GB 0MB 03:37:57 FULLPOS-B DYNFPOS 0 0.975 0.975 1.250 0:03 0:03 0.55515332309566E-07 2GB 0MB 03:38:02 FULLPOS-V DYNFPOS 0 2.099 2.099 4.127 0:05 0:07 0.55515332309566E-07 2GB 0MB 03:38:04 FULLPOS-T DYNFPOS 0 0.359 0.359 2.361 0:05 0:10 0.55515332309566E-07 2GB 0MB 03:38:04 0AAA00AAA STEPO 0 0.356 0.356 0.357 0:06 0:10 0.55515332309566E-07 2GB 0MB ------------------------------------------------ ..Failed, model did not complete successfully </stderr_txt> ]]> |
No trickles! |
---|
©2024 climateprediction.net