climateprediction.net home page
Task 22257385

Task 22257385

Name oifs_43r3_ps_0211_1987050100_123_956_12172855_0
Workunit 12172855
Created 21 Dec 2022, 10:40:53 UTC
Sent 13 Jan 2023, 16:02:42 UTC
Report deadline 12 Feb 2023, 16:02:42 UTC
Received 9 Feb 2023, 14:02:54 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537623
Run time 17 hours 25 min 59 sec
CPU time 17 hours 13 min 59 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 7.67 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,416.95 MB
Peak swap size 4,978.07 MB
Peak disk usage 2,828.78 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
: /home/galfridian/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855
..mkdir for temp folder for results failed
Copying: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/galfridian/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/galfridian/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e7b2a6ade3b75025ba2c9d9c687a8e0e to: /home/galfridian/slots/7/oifs_43r3_ps_0211_1987050100_123_956_12172855.zip
Unzipping the namelist zip file: /home/galfridian/slots/7/oifs_43r3_ps_0211_1987050100_123_956_12172855.zip
ic_ancil_file: ic_ancil_12172855
ifsdata_file: ifsdata_12172855
climate_data_file: clim_data_12172855
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_096ab88f55437a782a4eb2c6ed56e8ce to: /home/galfridian/slots/7/ic_ancil_12172855.zip
Unzipping the IC ancils zip file: /home/galfridian/slots/7/ic_ancil_12172855.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d63aa0bc9ceac7bbd806e50e5dc64750 to: /home/galfridian/slots/7/ifsdata/ifsdata_12172855.zip
Unzipping the ifsdata_zip file: /home/galfridian/slots/7/ifsdata/ifsdata_12172855.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/galfridian/slots/7/159l_2/clim_data_12172855.zip
Unzipping the climate data zip file: /home/galfridian/slots/7/159l_2/clim_data_12172855.zip
Checking for progress XML file: /home/galfridian/slots/7/progress_file_12172855.xml
Opened progress file ok : /home/galfridian/slots/7/progress_file_12172855.xml
-- Model is restarting --
Adjusting last_iter, 2499, to previous model restart step.
Creating progress file: /home/galfridian/slots/7/progress_file_12172855.xml
last_cpu_time: 52390
upload_file_number: 104
last_iter: 2495
last_upload: 8985600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020
The child process has been launched with process id: 2898
Executing the command: /home/galfridian/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:2898:2898] [20230209:060904:1675919344:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:09:42 STEP 2496 H=2496:00 +CPU= 30.180
  06:10:00 STEP 2497 H=2497:00 +CPU= 17.070
  06:10:17 STEP 2498 H=2498:00 +CPU= 17.699
  06:10:44 STEP 2499 H=2499:00 +CPU= 26.560
  06:11:02 STEP 2500 H=2500:00 +CPU= 17.721
  06:11:20 STEP 2501 H=2501:00 +CPU= 17.272
  06:11:47 STEP 2502 H=2502:00 +CPU= 27.342
  06:12:04 STEP 2503 H=2503:00 +CPU= 16.564
  06:12:21 STEP 2504 H=2504:00 +CPU= 17.515
  06:12:48 STEP 2505 H=2505:00 +CPU= 26.005
  06:13:05 STEP 2506 H=2506:00 +CPU= 17.543
  06:13:22 STEP 2507 H=2507:00 +CPU= 17.002
  06:13:52 STEP 2508 H=2508:00 +CPU= 29.692
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002508
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002508
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002508
  06:14:09 STEP 2509 H=2509:00 +CPU= 17.056
  06:14:27 STEP 2510 H=2510:00 +CPU= 17.584
  06:14:54 STEP 2511 H=2511:00 +CPU= 26.741
  06:15:12 STEP 2512 H=2512:00 +CPU= 17.575
  06:15:29 STEP 2513 H=2513:00 +CPU= 17.131
  06:15:56 STEP 2514 H=2514:00 +CPU= 27.052
  06:16:13 STEP 2515 H=2515:00 +CPU= 17.370
  06:16:31 STEP 2516 H=2516:00 +CPU= 17.788
  06:16:58 STEP 2517 H=2517:00 +CPU= 26.734
  06:17:16 STEP 2518 H=2518:00 +CPU= 17.777
  06:17:34 STEP 2519 H=2519:00 +CPU= 17.651
  06:18:06 STEP 2520 H=2520:00 +CPU= 31.900
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002520
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002520
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002520
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002496
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002496
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002496
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002508
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002508
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002508
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_104.zip
Uploading the intermediate file: upload_file_104.zip
  06:18:25 STEP 2521 H=2521:00 +CPU= 17.428
  06:18:44 STEP 2522 H=2522:00 +CPU= 17.713
  06:19:11 STEP 2523 H=2523:00 +CPU= 26.621
  06:19:29 STEP 2524 H=2524:00 +CPU= 17.710
  06:19:46 STEP 2525 H=2525:00 +CPU= 17.206
  06:20:14 STEP 2526 H=2526:00 +CPU= 27.278
  06:20:31 STEP 2527 H=2527:00 +CPU= 17.147
  06:20:49 STEP 2528 H=2528:00 +CPU= 18.014
  06:21:15 STEP 2529 H=2529:00 +CPU= 26.494
  06:21:33 STEP 2530 H=2530:00 +CPU= 17.668
  06:21:51 STEP 2531 H=2531:00 +CPU= 17.391
  06:22:21 STEP 2532 H=2532:00 +CPU= 29.829
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002532
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002532
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002532
  06:22:38 STEP 2533 H=2533:00 +CPU= 17.214
  06:22:56 STEP 2534 H=2534:00 +CPU= 17.906
  06:23:23 STEP 2535 H=2535:00 +CPU= 26.636
  06:23:41 STEP 2536 H=2536:00 +CPU= 17.794
  06:23:58 STEP 2537 H=2537:00 +CPU= 17.219
  06:24:26 STEP 2538 H=2538:00 +CPU= 27.397
  06:24:43 STEP 2539 H=2539:00 +CPU= 17.394
  06:25:00 STEP 2540 H=2540:00 +CPU= 16.879
  06:25:26 STEP 2541 H=2541:00 +CPU= 26.186
  06:25:44 STEP 2542 H=2542:00 +CPU= 17.607
  06:26:02 STEP 2543 H=2543:00 +CPU= 17.628
  06:26:34 STEP 2544 H=2544:00 +CPU= 31.537
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002544
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002544
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002544
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002520
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002520
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002520
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002532
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002532
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002532
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_105.zip
Uploading the intermediate file: upload_file_105.zip
  06:26:51 STEP 2545 H=2545:00 +CPU= 17.143
  06:27:09 STEP 2546 H=2546:00 +CPU= 17.651
  06:27:36 STEP 2547 H=2547:00 +CPU= 26.961
  06:27:54 STEP 2548 H=2548:00 +CPU= 17.822
  06:28:11 STEP 2549 H=2549:00 +CPU= 17.202
  06:28:39 STEP 2550 H=2550:00 +CPU= 27.394
  06:28:56 STEP 2551 H=2551:00 +CPU= 17.241
  06:29:14 STEP 2552 H=2552:00 +CPU= 17.741
  06:29:40 STEP 2553 H=2553:00 +CPU= 26.737
  06:29:58 STEP 2554 H=2554:00 +CPU= 17.764
  06:30:16 STEP 2555 H=2555:00 +CPU= 17.186
  06:30:46 STEP 2556 H=2556:00 +CPU= 29.883
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002556
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002556
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002556
  06:31:03 STEP 2557 H=2557:00 +CPU= 17.383
  06:31:21 STEP 2558 H=2558:00 +CPU= 17.773
  06:31:48 STEP 2559 H=2559:00 +CPU= 26.527
  06:32:05 STEP 2560 H=2560:00 +CPU= 17.469
  06:32:23 STEP 2561 H=2561:00 +CPU= 17.278
  06:32:50 STEP 2562 H=2562:00 +CPU= 26.969
  06:33:07 STEP 2563 H=2563:00 +CPU= 16.955
  06:33:24 STEP 2564 H=2564:00 +CPU= 17.610
  06:33:51 STEP 2565 H=2565:00 +CPU= 26.147
  06:34:09 STEP 2566 H=2566:00 +CPU= 17.815
  06:34:26 STEP 2567 H=2567:00 +CPU= 17.787
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0211
(argv4) batchid: 956
(argv5) wuid: 12172855
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /home/galfridian/slots/7
Project directory is: /home/galfridian/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855
..mkdir for temp folder for results failed
Copying: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/galfridian/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/galfridian/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e7b2a6ade3b75025ba2c9d9c687a8e0e to: /home/galfridian/slots/7/oifs_43r3_ps_0211_1987050100_123_956_12172855.zip
Unzipping the namelist zip file: /home/galfridian/slots/7/oifs_43r3_ps_0211_1987050100_123_956_12172855.zip
ic_ancil_file: ic_ancil_12172855
ifsdata_file: ifsdata_12172855
climate_data_file: clim_data_12172855
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_096ab88f55437a782a4eb2c6ed56e8ce to: /home/galfridian/slots/7/ic_ancil_12172855.zip
Unzipping the IC ancils zip file: /home/galfridian/slots/7/ic_ancil_12172855.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d63aa0bc9ceac7bbd806e50e5dc64750 to: /home/galfridian/slots/7/ifsdata/ifsdata_12172855.zip
Unzipping the ifsdata_zip file: /home/galfridian/slots/7/ifsdata/ifsdata_12172855.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/galfridian/slots/7/159l_2/clim_data_12172855.zip
Unzipping the climate data zip file: /home/galfridian/slots/7/159l_2/clim_data_12172855.zip
Checking for progress XML file: /home/galfridian/slots/7/progress_file_12172855.xml
Opened progress file ok : /home/galfridian/slots/7/progress_file_12172855.xml
-- Model is restarting --
Adjusting last_iter, 2568, to previous model restart step.
Creating progress file: /home/galfridian/slots/7/progress_file_12172855.xml
last_cpu_time: 53917
upload_file_number: 106
last_iter: 2567
last_upload: 9158400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020
The child process has been launched with process id: 1977
Executing the command: /home/galfridian/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:galfridian-systemproductname:1:1:1977:1977] [20230209:120034:1675940434:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:01:11 STEP 2568 H=2568:00 +CPU= 29.453
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002568
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002568
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002568
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002544
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002544
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002544
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002556
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002556
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002556
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_106.zip
Uploading the intermediate file: upload_file_106.zip
  12:01:30 STEP 2569 H=2569:00 +CPU= 17.419
  12:01:49 STEP 2570 H=2570:00 +CPU= 17.910
  12:02:16 STEP 2571 H=2571:00 +CPU= 26.691
  12:02:35 STEP 2572 H=2572:00 +CPU= 17.886
  12:02:53 STEP 2573 H=2573:00 +CPU= 17.370
  12:03:21 STEP 2574 H=2574:00 +CPU= 27.352
  12:03:39 STEP 2575 H=2575:00 +CPU= 17.213
  12:03:57 STEP 2576 H=2576:00 +CPU= 17.901
  12:04:24 STEP 2577 H=2577:00 +CPU= 27.069
  12:04:43 STEP 2578 H=2578:00 +CPU= 18.118
  12:05:01 STEP 2579 H=2579:00 +CPU= 17.201
  12:05:31 STEP 2580 H=2580:00 +CPU= 29.954
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002580
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002580
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002580
  12:05:49 STEP 2581 H=2581:00 +CPU= 17.727
  12:06:08 STEP 2582 H=2582:00 +CPU= 17.823
  12:06:35 STEP 2583 H=2583:00 +CPU= 26.759
  12:06:53 STEP 2584 H=2584:00 +CPU= 17.887
  12:07:12 STEP 2585 H=2585:00 +CPU= 17.668
  12:07:39 STEP 2586 H=2586:00 +CPU= 27.213
  12:07:56 STEP 2587 H=2587:00 +CPU= 16.955
  12:08:14 STEP 2588 H=2588:00 +CPU= 17.578
  12:08:42 STEP 2589 H=2589:00 +CPU= 26.995
  12:09:00 STEP 2590 H=2590:00 +CPU= 18.108
  12:09:18 STEP 2591 H=2591:00 +CPU= 17.847
  12:09:51 STEP 2592 H=2592:00 +CPU= 32.013
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002592
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002592
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002592
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002568
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002568
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002568
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002580
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002580
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002580
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_107.zip
Uploading the intermediate file: upload_file_107.zip
  12:10:09 STEP 2593 H=2593:00 +CPU= 17.336
  12:10:27 STEP 2594 H=2594:00 +CPU= 17.896
  12:10:54 STEP 2595 H=2595:00 +CPU= 26.482
  12:11:12 STEP 2596 H=2596:00 +CPU= 17.552
  12:11:30 STEP 2597 H=2597:00 +CPU= 17.524
  12:11:58 STEP 2598 H=2598:00 +CPU= 27.126
  12:12:16 STEP 2599 H=2599:00 +CPU= 17.264
  12:12:34 STEP 2600 H=2600:00 +CPU= 18.020
  12:13:01 STEP 2601 H=2601:00 +CPU= 26.726
  12:13:20 STEP 2602 H=2602:00 +CPU= 18.144
  12:13:38 STEP 2603 H=2603:00 +CPU= 17.506
  12:14:08 STEP 2604 H=2604:00 +CPU= 29.980
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002604
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002604
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002604
  12:14:27 STEP 2605 H=2605:00 +CPU= 17.758
  12:14:46 STEP 2606 H=2606:00 +CPU= 18.089
  12:15:13 STEP 2607 H=2607:00 +CPU= 26.803
  12:15:32 STEP 2608 H=2608:00 +CPU= 18.313
  12:15:50 STEP 2609 H=2609:00 +CPU= 17.335
  12:16:18 STEP 2610 H=2610:00 +CPU= 27.406
  12:16:36 STEP 2611 H=2611:00 +CPU= 17.623
  12:16:54 STEP 2612 H=2612:00 +CPU= 17.930
  12:17:21 STEP 2613 H=2613:00 +CPU= 26.878
  12:17:40 STEP 2614 H=2614:00 +CPU= 18.147
  12:17:58 STEP 2615 H=2615:00 +CPU= 18.112
  12:18:30 STEP 2616 H=2616:00 +CPU= 31.673
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002616
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002616
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002616
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002592
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002592
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002592
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002604
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002604
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002604
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_108.zip
Uploading the intermediate file: upload_file_108.zip
  12:18:48 STEP 2617 H=2617:00 +CPU= 17.287
  12:19:05 STEP 2618 H=2618:00 +CPU= 17.317
  12:19:32 STEP 2619 H=2619:00 +CPU= 26.665
  12:19:50 STEP 2620 H=2620:00 +CPU= 18.045
  12:20:08 STEP 2621 H=2621:00 +CPU= 17.397
  12:20:35 STEP 2622 H=2622:00 +CPU= 27.119
  12:20:52 STEP 2623 H=2623:00 +CPU= 17.219
  12:21:10 STEP 2624 H=2624:00 +CPU= 17.800
  12:21:37 STEP 2625 H=2625:00 +CPU= 26.577
  12:21:54 STEP 2626 H=2626:00 +CPU= 17.698
  12:22:12 STEP 2627 H=2627:00 +CPU= 17.384
  12:22:41 STEP 2628 H=2628:00 +CPU= 28.844
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002628
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002628
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002628
  12:22:58 STEP 2629 H=2629:00 +CPU= 17.000
  12:23:15 STEP 2630 H=2630:00 +CPU= 17.379
  12:23:42 STEP 2631 H=2631:00 +CPU= 26.152
  12:24:00 STEP 2632 H=2632:00 +CPU= 17.725
  12:24:17 STEP 2633 H=2633:00 +CPU= 17.149
  12:24:44 STEP 2634 H=2634:00 +CPU= 27.258
  12:25:02 STEP 2635 H=2635:00 +CPU= 17.178
  12:25:20 STEP 2636 H=2636:00 +CPU= 17.837
  12:25:46 STEP 2637 H=2637:00 +CPU= 26.454
  12:26:04 STEP 2638 H=2638:00 +CPU= 17.236
  12:26:20 STEP 2639 H=2639:00 +CPU= 16.695
  12:26:51 STEP 2640 H=2640:00 +CPU= 30.509
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002640
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002640
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002640
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002616
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002616
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002616
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002628
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002628
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002628
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_109.zip
Uploading the intermediate file: upload_file_109.zip
  12:27:08 STEP 2641 H=2641:00 +CPU= 16.753
  12:27:26 STEP 2642 H=2642:00 +CPU= 17.765
  12:27:53 STEP 2643 H=2643:00 +CPU= 26.893
  12:28:10 STEP 2644 H=2644:00 +CPU= 17.226
  12:28:27 STEP 2645 H=2645:00 +CPU= 17.106
  12:28:54 STEP 2646 H=2646:00 +CPU= 27.080
  12:29:12 STEP 2647 H=2647:00 +CPU= 17.376
  12:29:30 STEP 2648 H=2648:00 +CPU= 17.851
  12:29:56 STEP 2649 H=2649:00 +CPU= 26.617
  12:30:14 STEP 2650 H=2650:00 +CPU= 17.677
  12:30:31 STEP 2651 H=2651:00 +CPU= 17.055
  12:31:01 STEP 2652 H=2652:00 +CPU= 29.848
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002652
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002652
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002652
  12:31:19 STEP 2653 H=2653:00 +CPU= 17.220
  12:31:37 STEP 2654 H=2654:00 +CPU= 17.839
  12:32:04 STEP 2655 H=2655:00 +CPU= 26.877
  12:32:22 STEP 2656 H=2656:00 +CPU= 17.879
  12:32:39 STEP 2657 H=2657:00 +CPU= 17.283
  12:33:06 STEP 2658 H=2658:00 +CPU= 26.440
  12:33:23 STEP 2659 H=2659:00 +CPU= 17.020
  12:33:40 STEP 2660 H=2660:00 +CPU= 17.668
  12:34:07 STEP 2661 H=2661:00 +CPU= 26.533
  12:34:25 STEP 2662 H=2662:00 +CPU= 17.585
  12:34:42 STEP 2663 H=2663:00 +CPU= 17.543
  12:35:14 STEP 2664 H=2664:00 +CPU= 31.571
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002664
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002664
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002664
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002640
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002640
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002640
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002652
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002652
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002652
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_110.zip
Uploading the intermediate file: upload_file_110.zip
  12:35:31 STEP 2665 H=2665:00 +CPU= 16.739
  12:35:49 STEP 2666 H=2666:00 +CPU= 17.903
  12:36:16 STEP 2667 H=2667:00 +CPU= 26.578
  12:36:34 STEP 2668 H=2668:00 +CPU= 18.032
  12:36:52 STEP 2669 H=2669:00 +CPU= 17.322
  12:37:19 STEP 2670 H=2670:00 +CPU= 27.260
  12:37:36 STEP 2671 H=2671:00 +CPU= 16.991
  12:37:54 STEP 2672 H=2672:00 +CPU= 17.524
  12:38:21 STEP 2673 H=2673:00 +CPU= 26.371
  12:38:38 STEP 2674 H=2674:00 +CPU= 17.803
  12:38:56 STEP 2675 H=2675:00 +CPU= 17.304
  12:39:25 STEP 2676 H=2676:00 +CPU= 28.887
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002676
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002676
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002676
  12:39:42 STEP 2677 H=2677:00 +CPU= 17.210
  12:40:00 STEP 2678 H=2678:00 +CPU= 17.556
  12:40:25 STEP 2679 H=2679:00 +CPU= 25.254
  12:40:43 STEP 2680 H=2680:00 +CPU= 17.647
  12:41:00 STEP 2681 H=2681:00 +CPU= 17.131
  12:41:28 STEP 2682 H=2682:00 +CPU= 27.644
  12:41:45 STEP 2683 H=2683:00 +CPU= 17.275
  12:42:03 STEP 2684 H=2684:00 +CPU= 17.393
  12:42:30 STEP 2685 H=2685:00 +CPU= 26.786
  12:42:47 STEP 2686 H=2686:00 +CPU= 17.718
  12:43:05 STEP 2687 H=2687:00 +CPU= 17.334
  12:43:36 STEP 2688 H=2688:00 +CPU= 31.257
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002688
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002688
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002688
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002664
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002664
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002664
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002676
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002676
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002676
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_111.zip
Uploading the intermediate file: upload_file_111.zip
  12:43:53 STEP 2689 H=2689:00 +CPU= 16.959
  12:44:11 STEP 2690 H=2690:00 +CPU= 17.391
  12:44:37 STEP 2691 H=2691:00 +CPU= 26.581
  12:44:55 STEP 2692 H=2692:00 +CPU= 17.666
  12:45:12 STEP 2693 H=2693:00 +CPU= 17.062
  12:45:39 STEP 2694 H=2694:00 +CPU= 26.896
  12:45:56 STEP 2695 H=2695:00 +CPU= 16.914
  12:46:14 STEP 2696 H=2696:00 +CPU= 17.657
  12:46:41 STEP 2697 H=2697:00 +CPU= 26.748
  12:46:59 STEP 2698 H=2698:00 +CPU= 17.336
  12:47:16 STEP 2699 H=2699:00 +CPU= 17.054
  12:47:45 STEP 2700 H=2700:00 +CPU= 28.907
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002700
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002700
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002700
  12:48:02 STEP 2701 H=2701:00 +CPU= 16.466
  12:48:19 STEP 2702 H=2702:00 +CPU= 17.581
  12:48:46 STEP 2703 H=2703:00 +CPU= 26.524
  12:49:03 STEP 2704 H=2704:00 +CPU= 17.393
  12:49:21 STEP 2705 H=2705:00 +CPU= 17.266
  12:49:49 STEP 2706 H=2706:00 +CPU= 27.790
  12:50:06 STEP 2707 H=2707:00 +CPU= 17.393
  12:50:24 STEP 2708 H=2708:00 +CPU= 17.888
  12:50:51 STEP 2709 H=2709:00 +CPU= 26.868
  12:51:09 STEP 2710 H=2710:00 +CPU= 17.821
  12:51:28 STEP 2711 H=2711:00 +CPU= 17.985
  12:52:00 STEP 2712 H=2712:00 +CPU= 31.720
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002712
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002712
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002712
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002688
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002688
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002688
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002700
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002700
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002700
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_112.zip
Uploading the intermediate file: upload_file_112.zip
  12:52:17 STEP 2713 H=2713:00 +CPU= 17.171
  12:52:35 STEP 2714 H=2714:00 +CPU= 17.688
  12:53:02 STEP 2715 H=2715:00 +CPU= 26.667
  12:53:19 STEP 2716 H=2716:00 +CPU= 16.931
  12:53:35 STEP 2717 H=2717:00 +CPU= 16.679
  12:54:03 STEP 2718 H=2718:00 +CPU= 27.535
  12:54:20 STEP 2719 H=2719:00 +CPU= 17.405
  12:54:38 STEP 2720 H=2720:00 +CPU= 17.589
  12:55:05 STEP 2721 H=2721:00 +CPU= 26.806
  12:55:23 STEP 2722 H=2722:00 +CPU= 17.826
  12:55:41 STEP 2723 H=2723:00 +CPU= 17.581
  12:56:11 STEP 2724 H=2724:00 +CPU= 29.829
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002724
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002724
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002724
  12:56:28 STEP 2725 H=2725:00 +CPU= 17.109
  12:56:47 STEP 2726 H=2726:00 +CPU= 18.132
  12:57:15 STEP 2727 H=2727:00 +CPU= 27.215
  12:57:34 STEP 2728 H=2728:00 +CPU= 17.857
  12:57:52 STEP 2729 H=2729:00 +CPU= 17.516
  12:58:20 STEP 2730 H=2730:00 +CPU= 27.900
  12:58:38 STEP 2731 H=2731:00 +CPU= 17.216
  12:58:55 STEP 2732 H=2732:00 +CPU= 17.527
  12:59:23 STEP 2733 H=2733:00 +CPU= 26.665
  12:59:41 STEP 2734 H=2734:00 +CPU= 17.610
  12:59:59 STEP 2735 H=2735:00 +CPU= 17.745
  13:00:32 STEP 2736 H=2736:00 +CPU= 31.986
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002736
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002736
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002736
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002712
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002712
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002712
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002724
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002724
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002724
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_113.zip
Uploading the intermediate file: upload_file_113.zip
  13:00:50 STEP 2737 H=2737:00 +CPU= 17.191
  13:01:08 STEP 2738 H=2738:00 +CPU= 17.875
  13:01:36 STEP 2739 H=2739:00 +CPU= 27.172
  13:01:55 STEP 2740 H=2740:00 +CPU= 17.834
  13:02:13 STEP 2741 H=2741:00 +CPU= 17.215
  13:02:41 STEP 2742 H=2742:00 +CPU= 27.610
  13:02:58 STEP 2743 H=2743:00 +CPU= 17.438
  13:03:17 STEP 2744 H=2744:00 +CPU= 18.073
  13:03:45 STEP 2745 H=2745:00 +CPU= 26.890
  13:04:04 STEP 2746 H=2746:00 +CPU= 17.912
  13:04:22 STEP 2747 H=2747:00 +CPU= 17.135
  13:04:52 STEP 2748 H=2748:00 +CPU= 29.454
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002748
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002748
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002748
  13:05:10 STEP 2749 H=2749:00 +CPU= 17.153
  13:05:28 STEP 2750 H=2750:00 +CPU= 17.784
  13:05:55 STEP 2751 H=2751:00 +CPU= 26.767
  13:06:13 STEP 2752 H=2752:00 +CPU= 17.767
  13:06:31 STEP 2753 H=2753:00 +CPU= 17.217
  13:06:59 STEP 2754 H=2754:00 +CPU= 27.543
  13:07:16 STEP 2755 H=2755:00 +CPU= 17.314
  13:07:35 STEP 2756 H=2756:00 +CPU= 17.830
  13:08:02 STEP 2757 H=2757:00 +CPU= 27.040
  13:08:20 STEP 2758 H=2758:00 +CPU= 17.896
  13:08:38 STEP 2759 H=2759:00 +CPU= 17.665
  13:09:10 STEP 2760 H=2760:00 +CPU= 31.810
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002760
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002760
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002760
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002736
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002736
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002736
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002748
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002748
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002748
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_114.zip
Uploading the intermediate file: upload_file_114.zip
  13:09:28 STEP 2761 H=2761:00 +CPU= 17.428
  13:09:46 STEP 2762 H=2762:00 +CPU= 17.793
  13:10:14 STEP 2763 H=2763:00 +CPU= 27.085
  13:10:34 STEP 2764 H=2764:00 +CPU= 18.272
  13:10:51 STEP 2765 H=2765:00 +CPU= 17.671
  13:11:20 STEP 2766 H=2766:00 +CPU= 27.396
  13:11:38 STEP 2767 H=2767:00 +CPU= 17.804
  13:11:57 STEP 2768 H=2768:00 +CPU= 18.182
  13:12:25 STEP 2769 H=2769:00 +CPU= 27.531
  13:12:43 STEP 2770 H=2770:00 +CPU= 18.019
  13:13:02 STEP 2771 H=2771:00 +CPU= 17.714
  13:13:34 STEP 2772 H=2772:00 +CPU= 30.294
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002772
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002772
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002772
  13:13:52 STEP 2773 H=2773:00 +CPU= 17.645
  13:14:11 STEP 2774 H=2774:00 +CPU= 17.883
  13:14:38 STEP 2775 H=2775:00 +CPU= 27.245
  13:14:57 STEP 2776 H=2776:00 +CPU= 17.899
  13:15:15 STEP 2777 H=2777:00 +CPU= 17.464
  13:15:44 STEP 2778 H=2778:00 +CPU= 27.832
  13:16:01 STEP 2779 H=2779:00 +CPU= 17.245
  13:16:20 STEP 2780 H=2780:00 +CPU= 17.944
  13:16:48 STEP 2781 H=2781:00 +CPU= 27.647
  13:17:06 STEP 2782 H=2782:00 +CPU= 17.916
  13:17:24 STEP 2783 H=2783:00 +CPU= 17.804
  13:17:57 STEP 2784 H=2784:00 +CPU= 32.436
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002784
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002784
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002784
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002760
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002760
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002760
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002772
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002772
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002772
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_115.zip
Uploading the intermediate file: upload_file_115.zip
  13:18:15 STEP 2785 H=2785:00 +CPU= 17.410
Uploading trickle at timestep: 10022400
  13:18:33 STEP 2786 H=2786:00 +CPU= 17.898
  13:19:00 STEP 2787 H=2787:00 +CPU= 27.134
  13:19:18 STEP 2788 H=2788:00 +CPU= 17.697
  13:19:36 STEP 2789 H=2789:00 +CPU= 17.268
  13:20:05 STEP 2790 H=2790:00 +CPU= 27.911
  13:20:23 STEP 2791 H=2791:00 +CPU= 17.540
  13:20:43 STEP 2792 H=2792:00 +CPU= 18.345
  13:21:11 STEP 2793 H=2793:00 +CPU= 27.436
  13:21:30 STEP 2794 H=2794:00 +CPU= 18.186
  13:21:48 STEP 2795 H=2795:00 +CPU= 17.402
  13:22:18 STEP 2796 H=2796:00 +CPU= 30.084
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002796
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002796
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002796
  13:22:35 STEP 2797 H=2797:00 +CPU= 17.091
  13:22:54 STEP 2798 H=2798:00 +CPU= 17.929
  13:23:21 STEP 2799 H=2799:00 +CPU= 26.924
  13:23:39 STEP 2800 H=2800:00 +CPU= 17.278
  13:23:54 STEP 2801 H=2801:00 +CPU= 15.757
  13:24:22 STEP 2802 H=2802:00 +CPU= 27.054
  13:24:40 STEP 2803 H=2803:00 +CPU= 17.494
  13:24:59 STEP 2804 H=2804:00 +CPU= 18.204
  13:25:27 STEP 2805 H=2805:00 +CPU= 27.020
  13:25:46 STEP 2806 H=2806:00 +CPU= 17.935
  13:26:04 STEP 2807 H=2807:00 +CPU= 18.012
  13:26:37 STEP 2808 H=2808:00 +CPU= 32.300
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002808
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002808
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002808
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002784
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002784
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002784
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002796
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002796
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002796
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_116.zip
Uploading the intermediate file: upload_file_116.zip
  13:26:54 STEP 2809 H=2809:00 +CPU= 16.666
  13:27:12 STEP 2810 H=2810:00 +CPU= 17.654
  13:27:40 STEP 2811 H=2811:00 +CPU= 26.932
  13:27:58 STEP 2812 H=2812:00 +CPU= 18.417
  13:28:17 STEP 2813 H=2813:00 +CPU= 17.522
  13:28:45 STEP 2814 H=2814:00 +CPU= 27.632
  13:29:02 STEP 2815 H=2815:00 +CPU= 17.517
  13:29:21 STEP 2816 H=2816:00 +CPU= 18.024
  13:29:48 STEP 2817 H=2817:00 +CPU= 26.763
  13:30:07 STEP 2818 H=2818:00 +CPU= 18.473
  13:30:26 STEP 2819 H=2819:00 +CPU= 18.027
  13:30:57 STEP 2820 H=2820:00 +CPU= 30.421
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002820
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002820
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002820
  13:31:16 STEP 2821 H=2821:00 +CPU= 18.062
  13:31:35 STEP 2822 H=2822:00 +CPU= 18.587
  13:32:03 STEP 2823 H=2823:00 +CPU= 26.860
  13:32:22 STEP 2824 H=2824:00 +CPU= 18.243
  13:32:40 STEP 2825 H=2825:00 +CPU= 17.611
  13:33:07 STEP 2826 H=2826:00 +CPU= 27.196
  13:33:25 STEP 2827 H=2827:00 +CPU= 17.650
  13:33:43 STEP 2828 H=2828:00 +CPU= 18.084
  13:34:10 STEP 2829 H=2829:00 +CPU= 26.904
  13:34:28 STEP 2830 H=2830:00 +CPU= 17.736
  13:34:47 STEP 2831 H=2831:00 +CPU= 18.037
  13:35:20 STEP 2832 H=2832:00 +CPU= 32.489
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002832
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002832
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002832
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002808
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002808
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002808
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002820
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002820
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002820
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_117.zip
Uploading the intermediate file: upload_file_117.zip
  13:35:38 STEP 2833 H=2833:00 +CPU= 17.303
  13:35:56 STEP 2834 H=2834:00 +CPU= 17.672
  13:36:23 STEP 2835 H=2835:00 +CPU= 27.041
  13:36:41 STEP 2836 H=2836:00 +CPU= 17.980
  13:36:59 STEP 2837 H=2837:00 +CPU= 17.569
  13:37:27 STEP 2838 H=2838:00 +CPU= 27.451
  13:37:45 STEP 2839 H=2839:00 +CPU= 17.818
  13:38:03 STEP 2840 H=2840:00 +CPU= 17.770
  13:38:30 STEP 2841 H=2841:00 +CPU= 26.684
  13:38:48 STEP 2842 H=2842:00 +CPU= 17.492
  13:39:05 STEP 2843 H=2843:00 +CPU= 17.104
  13:39:35 STEP 2844 H=2844:00 +CPU= 30.304
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002844
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002844
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002844
  13:39:53 STEP 2845 H=2845:00 +CPU= 17.327
  13:40:11 STEP 2846 H=2846:00 +CPU= 17.988
  13:40:39 STEP 2847 H=2847:00 +CPU= 26.857
  13:40:57 STEP 2848 H=2848:00 +CPU= 18.371
  13:41:15 STEP 2849 H=2849:00 +CPU= 17.477
  13:41:42 STEP 2850 H=2850:00 +CPU= 27.425
  13:42:00 STEP 2851 H=2851:00 +CPU= 17.708
  13:42:18 STEP 2852 H=2852:00 +CPU= 18.054
  13:42:46 STEP 2853 H=2853:00 +CPU= 27.201
  13:43:04 STEP 2854 H=2854:00 +CPU= 18.099
  13:43:22 STEP 2855 H=2855:00 +CPU= 18.051
  13:43:55 STEP 2856 H=2856:00 +CPU= 32.044
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002856
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002856
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002856
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002832
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002832
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002832
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002844
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002844
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002844
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_118.zip
Uploading the intermediate file: upload_file_118.zip
  13:44:13 STEP 2857 H=2857:00 +CPU= 17.573
  13:44:30 STEP 2858 H=2858:00 +CPU= 17.692
  13:44:58 STEP 2859 H=2859:00 +CPU= 27.192
  13:45:16 STEP 2860 H=2860:00 +CPU= 17.976
  13:45:33 STEP 2861 H=2861:00 +CPU= 17.330
  13:46:02 STEP 2862 H=2862:00 +CPU= 27.589
  13:46:19 STEP 2863 H=2863:00 +CPU= 17.523
  13:46:38 STEP 2864 H=2864:00 +CPU= 18.076
  13:47:05 STEP 2865 H=2865:00 +CPU= 26.732
  13:47:23 STEP 2866 H=2866:00 +CPU= 17.958
  13:47:40 STEP 2867 H=2867:00 +CPU= 17.537
  13:48:11 STEP 2868 H=2868:00 +CPU= 30.342
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002868
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002868
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002868
  13:48:29 STEP 2869 H=2869:00 +CPU= 17.558
  13:48:47 STEP 2870 H=2870:00 +CPU= 18.393
  13:49:14 STEP 2871 H=2871:00 +CPU= 26.877
  13:49:33 STEP 2872 H=2872:00 +CPU= 18.380
  13:49:51 STEP 2873 H=2873:00 +CPU= 17.960
  13:50:19 STEP 2874 H=2874:00 +CPU= 27.697
  13:50:37 STEP 2875 H=2875:00 +CPU= 17.604
  13:50:55 STEP 2876 H=2876:00 +CPU= 17.852
  13:51:23 STEP 2877 H=2877:00 +CPU= 27.188
  13:51:41 STEP 2878 H=2878:00 +CPU= 18.058
  13:51:59 STEP 2879 H=2879:00 +CPU= 17.693
  13:52:31 STEP 2880 H=2880:00 +CPU= 31.635
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002880
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002880
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002880
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002856
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002856
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002856
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002868
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002868
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002868
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_119.zip
Uploading the intermediate file: upload_file_119.zip
  13:52:48 STEP 2881 H=2881:00 +CPU= 17.326
  13:53:06 STEP 2882 H=2882:00 +CPU= 17.962
  13:53:34 STEP 2883 H=2883:00 +CPU= 27.190
  13:53:52 STEP 2884 H=2884:00 +CPU= 17.600
  13:54:09 STEP 2885 H=2885:00 +CPU= 17.117
  13:54:36 STEP 2886 H=2886:00 +CPU= 26.972
  13:54:54 STEP 2887 H=2887:00 +CPU= 17.327
  13:55:12 STEP 2888 H=2888:00 +CPU= 17.961
  13:55:39 STEP 2889 H=2889:00 +CPU= 27.325
  13:55:58 STEP 2890 H=2890:00 +CPU= 17.995
  13:56:15 STEP 2891 H=2891:00 +CPU= 17.253
  13:56:45 STEP 2892 H=2892:00 +CPU= 29.831
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002892
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002892
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002892
  13:57:02 STEP 2893 H=2893:00 +CPU= 16.968
  13:57:20 STEP 2894 H=2894:00 +CPU= 17.666
  13:57:47 STEP 2895 H=2895:00 +CPU= 26.693
  13:58:05 STEP 2896 H=2896:00 +CPU= 17.580
  13:58:23 STEP 2897 H=2897:00 +CPU= 17.350
  13:58:51 STEP 2898 H=2898:00 +CPU= 27.899
  13:59:08 STEP 2899 H=2899:00 +CPU= 17.270
  13:59:26 STEP 2900 H=2900:00 +CPU= 17.436
  13:59:53 STEP 2901 H=2901:00 +CPU= 27.067
  14:00:11 STEP 2902 H=2902:00 +CPU= 17.425
  14:00:29 STEP 2903 H=2903:00 +CPU= 17.736
  14:01:01 STEP 2904 H=2904:00 +CPU= 32.171
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002904
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002904
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002904
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002880
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002880
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002880
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002892
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002892
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002892
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_120.zip
Uploading the intermediate file: upload_file_120.zip
  14:01:18 STEP 2905 H=2905:00 +CPU= 17.191
  14:01:36 STEP 2906 H=2906:00 +CPU= 17.287
  14:02:03 STEP 2907 H=2907:00 +CPU= 26.848
  14:02:21 STEP 2908 H=2908:00 +CPU= 17.803
  14:02:39 STEP 2909 H=2909:00 +CPU= 17.561
  14:03:06 STEP 2910 H=2910:00 +CPU= 27.247
  14:03:24 STEP 2911 H=2911:00 +CPU= 17.218
  14:03:41 STEP 2912 H=2912:00 +CPU= 17.145
  14:04:08 STEP 2913 H=2913:00 +CPU= 26.568
  14:04:26 STEP 2914 H=2914:00 +CPU= 17.761
  14:04:43 STEP 2915 H=2915:00 +CPU= 17.152
  14:05:14 STEP 2916 H=2916:00 +CPU= 30.298
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002916
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002916
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002916
  14:05:31 STEP 2917 H=2917:00 +CPU= 17.433
  14:05:50 STEP 2918 H=2918:00 +CPU= 18.192
  14:06:17 STEP 2919 H=2919:00 +CPU= 26.843
  14:06:35 STEP 2920 H=2920:00 +CPU= 18.050
  14:06:53 STEP 2921 H=2921:00 +CPU= 17.322
  14:07:21 STEP 2922 H=2922:00 +CPU= 27.729
  14:07:38 STEP 2923 H=2923:00 +CPU= 17.598
  14:07:56 STEP 2924 H=2924:00 +CPU= 17.940
  14:08:24 STEP 2925 H=2925:00 +CPU= 27.387
  14:08:42 STEP 2926 H=2926:00 +CPU= 18.066
  14:09:00 STEP 2927 H=2927:00 +CPU= 17.926
  14:09:33 STEP 2928 H=2928:00 +CPU= 32.498
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002928
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002928
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002928
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002904
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002904
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002904
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002916
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002916
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002916
Zipping up the intermediate file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_121.zip
Uploading the intermediate file: upload_file_121.zip
  14:09:51 STEP 2929 H=2929:00 +CPU= 17.509
  14:10:09 STEP 2930 H=2930:00 +CPU= 18.047
  14:10:37 STEP 2931 H=2931:00 +CPU= 27.119
  14:10:55 STEP 2932 H=2932:00 +CPU= 18.296
  14:11:12 STEP 2933 H=2933:00 +CPU= 17.215
  14:11:40 STEP 2934 H=2934:00 +CPU= 27.632
  14:11:58 STEP 2935 H=2935:00 +CPU= 17.510
  14:12:16 STEP 2936 H=2936:00 +CPU= 18.082
  14:12:43 STEP 2937 H=2937:00 +CPU= 26.953
  14:13:01 STEP 2938 H=2938:00 +CPU= 17.590
  14:13:19 STEP 2939 H=2939:00 +CPU= 17.587
  14:13:49 STEP 2940 H=2940:00 +CPU= 30.210
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002940
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002940
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002940
  14:14:07 STEP 2941 H=2941:00 +CPU= 18.021
  14:14:26 STEP 2942 H=2942:00 +CPU= 18.132
  14:14:53 STEP 2943 H=2943:00 +CPU= 26.986
  14:15:11 STEP 2944 H=2944:00 +CPU= 17.903
  14:15:29 STEP 2945 H=2945:00 +CPU= 17.475
  14:15:57 STEP 2946 H=2946:00 +CPU= 27.427
  14:16:14 STEP 2947 H=2947:00 +CPU= 17.264
  14:16:32 STEP 2948 H=2948:00 +CPU= 17.985
  14:16:59 STEP 2949 H=2949:00 +CPU= 26.799
  14:17:16 STEP 2950 H=2950:00 +CPU= 16.863
  14:17:35 STEP 2951 H=2951:00 +CPU= 18.132
  14:18:05 STEP 2952 H=2952:00 +CPU= 30.077
The child process terminated with status: 0
Moving to projects directory: /home/galfridian/slots/7/ICMGGhq0f+002952
Moving to projects directory: /home/galfridian/slots/7/ICMSHhq0f+002952
Moving to projects directory: /home/galfridian/slots/7/ICMUAhq0f+002952
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002952
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002928
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002928
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002940
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002928
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMUAhq0f+002940
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMGGhq0f+002952
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002952
Adding to the zip: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_12172855/ICMSHhq0f+002940
Zipping up the final file: /home/galfridian/projects/climateprediction.net/oifs_43r3_ps_0211_1987050100_123_956_12172855_0_r2100111020_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
14:21:28 (1858): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:50:10 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 10,623,600 62,039 0.0058
09 May 2023 16:50:09 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 10,022,400 58,490 0.0058
09 May 2023 16:49:57 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 8,640,000 50,306 0.0058
09 May 2023 16:49:56 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 7,776,000 45,266 0.0058
09 May 2023 16:49:56 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 7,776,000 45,266 0.0058
09 May 2023 16:49:54 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 6,912,000 40,234 0.0058
09 May 2023 16:49:53 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 6,048,000 35,216 0.0058
09 May 2023 16:49:51 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 5,184,000 30,202 0.0058
09 May 2023 16:49:50 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 4,320,000 25,181 0.0058
09 May 2023 16:49:49 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 3,456,000 20,143 0.0058
09 May 2023 16:49:49 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 3,456,000 20,143 0.0058
09 May 2023 16:49:47 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 2,592,000 15,111 0.0058
09 May 2023 16:49:45 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 1,728,000 10,077 0.0058
09 May 2023 16:49:45 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 1,728,000 10,077 0.0058
09 May 2023 16:49:43 1537623 22257385 oifs_43r3_ps_0211_1987050100_123_956_12172855_0 864,000 5,054 0.0058


©2024 climateprediction.net