Name | oifs_43r3_ps_0258_2006050100_123_975_12191902_0 |
Workunit | 12191902 |
Created | 23 Dec 2022, 12:43:08 UTC |
Sent | 16 Jan 2023, 21:59:25 UTC |
Report deadline | 17 Mar 2023, 21:59:25 UTC |
Received | 22 Jan 2023, 14:23:50 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1538188 |
Run time | 7 hours 54 min 55 sec |
CPU time | 2 hours 15 min 18 sec |
Validate state | Invalid |
Credit | 3,628.80 |
Device peak FLOPS | 9.06 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,412.36 MB |
Peak swap size | 4,978.08 MB |
Peak disk usage | 7,293.38 MB |
Stderr | <core_client_version>7.18.1</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> request received from the BOINC client, suspending the child process Resuming the child process 15:18:42 STEP 1432 H=1432:00 +CPU= 8.529 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:18:57 STEP 1433 H=1433:00 +CPU= 7.819 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:19:22 STEP 1434 H=1434:00 +CPU= 13.643 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:19:39 STEP 1435 H=1435:00 +CPU= 8.700 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:19:57 STEP 1436 H=1436:00 +CPU= 8.385 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:20:23 STEP 1437 H=1437:00 +CPU= 12.871 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:20:39 STEP 1438 H=1438:00 +CPU= 8.100 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:20:57 STEP 1439 H=1439:00 +CPU= 8.201 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Quit request received from the BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2006050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0258 (argv4) batchid: 975 (argv5) wuid: 12191902 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/10 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_1e0f6926448aadaddb979a24cec54082 to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0258_2006050100_123_975_12191902.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0258_2006050100_123_975_12191902.zip ic_ancil_file: ic_ancil_12191902 ifsdata_file: ifsdata_12191902 climate_data_file: clim_data_12191902 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_a1345cd2993e04a99f6024a229571434 to: /var/lib/boinc-client/slots/10/ic_ancil_12191902.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12191902.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a3bcbecbabe5e4376919980a811a4b3f to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12191902.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12191902.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12191902.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12191902.zip Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12191902.xml Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12191902.xml -- Model is restarting -- Adjusting last_iter, 1440, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12191902.xml last_cpu_time: 7628 upload_file_number: 29 last_iter: 1439 last_upload: 5097600 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0258_2006050100_123_975_12191902_0_r1946556803 The child process has been launched with process id: 1618491 Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1618491:1618491] [20230122:152233:1674393753:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:23:12 STEP 1440 H=1440:00 +CPU= 16.979 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+001440 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+001440 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+001440 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902/ICMGGhq0f+001416 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902/ICMSHhq0f+001416 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902/ICMUAhq0f+001416 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902/ICMGGhq0f+001428 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902/ICMSHhq0f+001428 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902/ICMUAhq0f+001428 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0258_2006050100_123_975_12191902_0_r1946556803_29.zip Uploading the intermediate file: upload_file_29.zip 15:23:27 STEP 1441 H=1441:00 +CPU= 8.403 15:23:37 STEP 1442 H=1442:00 +CPU= 8.592 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2006050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0258 (argv4) batchid: 975 (argv5) wuid: 12191902 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/10 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_1e0f6926448aadaddb979a24cec54082 to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0258_2006050100_123_975_12191902.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0258_2006050100_123_975_12191902.zip ic_ancil_file: ic_ancil_12191902 ifsdata_file: ifsdata_12191902 climate_data_file: clim_data_12191902 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_a1345cd2993e04a99f6024a229571434 to: /var/lib/boinc-client/slots/10/ic_ancil_12191902.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12191902.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a3bcbecbabe5e4376919980a811a4b3f to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12191902.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12191902.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12191902.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12191902.zip Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12191902.xml Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12191902.xml -- Model is restarting -- Adjusting last_iter, 1441, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12191902.xml last_cpu_time: 7651 upload_file_number: 30 last_iter: 1439 last_upload: 5184000 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0258_2006050100_123_975_12191902_0_r1946556803 The child process has been launched with process id: 1620451 Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1620451:1620451] [20230122:152505:1674393905:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:25:33 STEP 1440 H=1440:00 +CPU= 15.148 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:25:48 STEP 1441 H=1441:00 +CPU= 8.339 15:25:58 STEP 1442 H=1442:00 +CPU= 8.207 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2006050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0258 (argv4) batchid: 975 (argv5) wuid: 12191902 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/10 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_1e0f6926448aadaddb979a24cec54082 to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0258_2006050100_123_975_12191902.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0258_2006050100_123_975_12191902.zip ic_ancil_file: ic_ancil_12191902 ifsdata_file: ifsdata_12191902 climate_data_file: clim_data_12191902 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_a1345cd2993e04a99f6024a229571434 to: /var/lib/boinc-client/slots/10/ic_ancil_12191902.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12191902.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a3bcbecbabe5e4376919980a811a4b3f to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12191902.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12191902.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12191902.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12191902.zip Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12191902.xml Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12191902.xml -- Model is restarting -- Adjusting last_iter, 1443, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12191902.xml last_cpu_time: 7691 upload_file_number: 30 last_iter: 1439 last_upload: 5184000 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0258_2006050100_123_975_12191902_0_r1946556803 The child process has been launched with process id: 1621866 Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1621866:1621866] [20230122:152718:1674394038:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:27:57 STEP 1440 H=1440:00 +CPU= 16.773 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:28:14 STEP 1441 H=1441:00 +CPU= 8.640 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:28:33 STEP 1442 H=1442:00 +CPU= 9.156 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:28:55 STEP 1443 H=1443:00 +CPU= 12.811 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:29:13 STEP 1444 H=1444:00 +CPU= 8.511 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:29:29 STEP 1445 H=1445:00 +CPU= 7.782 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:29:57 STEP 1446 H=1446:00 +CPU= 13.440 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:30:13 STEP 1447 H=1447:00 +CPU= 8.489 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:30:30 STEP 1448 H=1448:00 +CPU= 8.197 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2006050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0258 (argv4) batchid: 975 (argv5) wuid: 12191902 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/10 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_1e0f6926448aadaddb979a24cec54082 to: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0258_2006050100_123_975_12191902.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_ps_0258_2006050100_123_975_12191902.zip ic_ancil_file: ic_ancil_12191902 ifsdata_file: ifsdata_12191902 climate_data_file: clim_data_12191902 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_a1345cd2993e04a99f6024a229571434 to: /var/lib/boinc-client/slots/10/ic_ancil_12191902.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12191902.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a3bcbecbabe5e4376919980a811a4b3f to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12191902.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12191902.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12191902.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12191902.zip Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12191902.xml Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12191902.xml -- Model is restarting -- Adjusting last_iter, 1449, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12191902.xml last_cpu_time: 7795 upload_file_number: 30 last_iter: 1439 last_upload: 5184000 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0258_2006050100_123_975_12191902_0_r1946556803 The child process has been launched with process id: 1624677 Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1624677:1624677] [20230122:153151:1674394311:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:32:22 STEP 1440 H=1440:00 +CPU= 15.413 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:32:33 STEP 1441 H=1441:00 +CPU= 8.042 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:32:50 STEP 1442 H=1442:00 +CPU= 8.594 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:33:19 STEP 1443 H=1443:00 +CPU= 13.755 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:33:34 STEP 1444 H=1444:00 +CPU= 8.711 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:33:46 STEP 1445 H=1445:00 +CPU= 8.222 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:34:08 STEP 1446 H=1446:00 +CPU= 12.182 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:34:22 STEP 1447 H=1447:00 +CPU= 8.522 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:34:35 STEP 1448 H=1448:00 +CPU= 8.638 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:35:05 STEP 1449 H=1449:00 +CPU= 14.292 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:35:19 STEP 1450 H=1450:00 +CPU= 8.252 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:35:34 STEP 1451 H=1451:00 +CPU= 7.666 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:36:03 STEP 1452 H=1452:00 +CPU= 15.163 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+001452 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+001452 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+001452 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:36:19 STEP 1453 H=1453:00 +CPU= 8.054 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:36:35 STEP 1454 H=1454:00 +CPU= 8.262 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:37:04 STEP 1455 H=1455:00 +CPU= 13.111 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:37:23 STEP 1456 H=1456:00 +CPU= 8.687 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:37:39 STEP 1457 H=1457:00 +CPU= 8.031 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:38:12 STEP 1458 H=1458:00 +CPU= 14.896 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:38:28 STEP 1459 H=1459:00 +CPU= 8.555 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:38:48 STEP 1460 H=1460:00 +CPU= 9.424 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:39:18 STEP 1461 H=1461:00 +CPU= 14.183 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:39:35 STEP 1462 H=1462:00 +CPU= 9.208 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:39:52 STEP 1463 H=1463:00 +CPU= 8.986 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:40:29 STEP 1464 H=1464:00 +CPU= 18.334 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGhq0f+001464 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHhq0f+001464 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAhq0f+001464 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902/ICMGGhq0f+001440 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902/ICMSHhq0f+001440 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902/ICMUAhq0f+001440 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902/ICMGGhq0f+001452 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902/ICMSHhq0f+001452 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191902/ICMUAhq0f+001452 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0258_2006050100_123_975_12191902_0_r1946556803_30.zip Uploading the intermediate file: upload_file_30.zip 15:40:47 STEP 1465 H=1465:00 +CPU= 9.417 15:40:58 STEP 1466 H=1466:00 +CPU= 8.800 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:41:20 STEP 1467 H=1467:00 +CPU= 14.409 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:41:39 STEP 1468 H=1468:00 +CPU= 9.598 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:41:58 STEP 1469 H=1469:00 +CPU= 8.580 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Abort request received from the BOINC client, ending the child process ..Failed, model did not complete successfully </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 16:24:25 | 1538188 | 22277609 | oifs_43r3_ps_0258_2006050100_123_975_12191902_0 | 2,592,000 | 12,132 | 0.0047 |
09 May 2023 16:23:18 | 1538188 | 22277609 | oifs_43r3_ps_0258_2006050100_123_975_12191902_0 | 1,728,000 | 8,079 | 0.0047 |
09 May 2023 16:23:13 | 1538188 | 22277609 | oifs_43r3_ps_0258_2006050100_123_975_12191902_0 | 864,000 | 4,158 | 0.0048 |
©2024 climateprediction.net