climateprediction.net home page
Task 22259943

Task 22259943

Name oifs_43r3_ps_0610_1989050100_123_958_12175254_0
Workunit 12175254
Created 21 Dec 2022, 17:00:50 UTC
Sent 21 Dec 2022, 17:10:47 UTC
Report deadline 20 Jan 2023, 17:10:47 UTC
Received 24 Dec 2022, 1:28:53 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1531398
Run time 23 hours 43 min 59 sec
CPU time 18 hours 9 min 57 sec
Validate state Invalid
Credit 14,152.32
Device peak FLOPS 7.99 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,791.00 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,220.13 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
TEP 2490 H=2490:00 +CPU= 36.091
  21:43:20 STEP 2491 H=2491:00 +CPU= 23.133
  21:43:45 STEP 2492 H=2492:00 +CPU= 24.499
  21:44:22 STEP 2493 H=2493:00 +CPU= 34.518
  21:44:47 STEP 2494 H=2494:00 +CPU= 24.025
  21:45:12 STEP 2495 H=2495:00 +CPU= 24.151
  21:45:54 STEP 2496 H=2496:00 +CPU= 40.308
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_79.zip
Uploading the intermediate file: upload_file_79.zip
  21:46:18 STEP 2497 H=2497:00 +CPU= 22.760
  21:46:43 STEP 2498 H=2498:00 +CPU= 23.465
  21:47:19 STEP 2499 H=2499:00 +CPU= 34.764
  21:47:44 STEP 2500 H=2500:00 +CPU= 23.527
  21:48:08 STEP 2501 H=2501:00 +CPU= 22.841
  21:48:43 STEP 2502 H=2502:00 +CPU= 34.232
  21:49:07 STEP 2503 H=2503:00 +CPU= 22.379
  21:49:31 STEP 2504 H=2504:00 +CPU= 23.355
  21:50:08 STEP 2505 H=2505:00 +CPU= 35.562
  21:50:32 STEP 2506 H=2506:00 +CPU= 23.127
  21:50:58 STEP 2507 H=2507:00 +CPU= 24.824
  21:51:38 STEP 2508 H=2508:00 +CPU= 38.322
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002508
  21:52:02 STEP 2509 H=2509:00 +CPU= 23.374
  21:52:28 STEP 2510 H=2510:00 +CPU= 25.032
  21:53:04 STEP 2511 H=2511:00 +CPU= 34.395
  21:53:29 STEP 2512 H=2512:00 +CPU= 23.859
  21:53:54 STEP 2513 H=2513:00 +CPU= 23.932
  21:54:31 STEP 2514 H=2514:00 +CPU= 35.736
  21:54:55 STEP 2515 H=2515:00 +CPU= 22.703
  21:55:19 STEP 2516 H=2516:00 +CPU= 23.194
  21:55:56 STEP 2517 H=2517:00 +CPU= 35.824
  21:56:21 STEP 2518 H=2518:00 +CPU= 23.560
  21:56:46 STEP 2519 H=2519:00 +CPU= 24.015
  21:57:28 STEP 2520 H=2520:00 +CPU= 39.485
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_80.zip
Uploading the intermediate file: upload_file_80.zip
  21:57:52 STEP 2521 H=2521:00 +CPU= 22.525
  21:58:18 STEP 2522 H=2522:00 +CPU= 24.336
  21:58:55 STEP 2523 H=2523:00 +CPU= 35.299
  21:59:20 STEP 2524 H=2524:00 +CPU= 23.683
  21:59:43 STEP 2525 H=2525:00 +CPU= 22.637
  22:00:19 STEP 2526 H=2526:00 +CPU= 34.678
  22:00:44 STEP 2527 H=2527:00 +CPU= 23.182
  22:01:07 STEP 2528 H=2528:00 +CPU= 22.926
  22:01:43 STEP 2529 H=2529:00 +CPU= 33.685
  22:02:08 STEP 2530 H=2530:00 +CPU= 23.841
  22:02:32 STEP 2531 H=2531:00 +CPU= 23.444
  22:03:12 STEP 2532 H=2532:00 +CPU= 38.072
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002532
  22:03:37 STEP 2533 H=2533:00 +CPU= 23.785
  22:04:01 STEP 2534 H=2534:00 +CPU= 23.182
  22:04:37 STEP 2535 H=2535:00 +CPU= 34.589
  22:05:02 STEP 2536 H=2536:00 +CPU= 23.505
  22:05:25 STEP 2537 H=2537:00 +CPU= 22.725
  22:06:01 STEP 2538 H=2538:00 +CPU= 34.777
  22:06:27 STEP 2539 H=2539:00 +CPU= 24.319
  22:06:51 STEP 2540 H=2540:00 +CPU= 23.573
  22:07:27 STEP 2541 H=2541:00 +CPU= 34.255
  22:07:52 STEP 2542 H=2542:00 +CPU= 24.016
  22:08:18 STEP 2543 H=2543:00 +CPU= 24.578
  22:09:01 STEP 2544 H=2544:00 +CPU= 40.839
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_81.zip
Uploading the intermediate file: upload_file_81.zip
  22:09:25 STEP 2545 H=2545:00 +CPU= 22.422
  22:09:48 STEP 2546 H=2546:00 +CPU= 22.762
  22:10:25 STEP 2547 H=2547:00 +CPU= 34.588
  22:10:50 STEP 2548 H=2548:00 +CPU= 23.663
  22:11:14 STEP 2549 H=2549:00 +CPU= 23.293
  22:11:52 STEP 2550 H=2550:00 +CPU= 35.936
  22:12:16 STEP 2551 H=2551:00 +CPU= 23.623
  22:12:42 STEP 2552 H=2552:00 +CPU= 24.385
  22:13:17 STEP 2553 H=2553:00 +CPU= 34.134
  22:13:42 STEP 2554 H=2554:00 +CPU= 23.371
  22:14:06 STEP 2555 H=2555:00 +CPU= 23.815
  22:14:47 STEP 2556 H=2556:00 +CPU= 38.706
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002556
  22:15:12 STEP 2557 H=2557:00 +CPU= 24.067
  22:15:38 STEP 2558 H=2558:00 +CPU= 24.536
  22:16:13 STEP 2559 H=2559:00 +CPU= 33.865
  22:16:38 STEP 2560 H=2560:00 +CPU= 24.426
  22:17:04 STEP 2561 H=2561:00 +CPU= 24.368
  22:17:40 STEP 2562 H=2562:00 +CPU= 34.943
  22:18:04 STEP 2563 H=2563:00 +CPU= 22.855
  22:18:30 STEP 2564 H=2564:00 +CPU= 24.580
  22:19:06 STEP 2565 H=2565:00 +CPU= 34.227
  22:19:30 STEP 2566 H=2566:00 +CPU= 23.437
  22:19:56 STEP 2567 H=2567:00 +CPU= 24.446
  22:20:38 STEP 2568 H=2568:00 +CPU= 39.315
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_82.zip
Uploading the intermediate file: upload_file_82.zip
  22:21:03 STEP 2569 H=2569:00 +CPU= 21.399
  22:21:30 STEP 2570 H=2570:00 +CPU= 23.515
  22:22:08 STEP 2571 H=2571:00 +CPU= 32.526
  22:22:34 STEP 2572 H=2572:00 +CPU= 22.208
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0610
(argv4) batchid: 958
(argv5) wuid: 12175254
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/18
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/18/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/18/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_af179f5e57efa2a19ff4c1024b893661 to: /var/lib/boinc/slots/18/oifs_43r3_ps_0610_1989050100_123_958_12175254.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/18/oifs_43r3_ps_0610_1989050100_123_958_12175254.zip
ic_ancil_file: ic_ancil_12175254
ifsdata_file: ifsdata_12175254
climate_data_file: clim_data_12175254
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d87a761f09f2bb60f129754c0c0b318a to: /var/lib/boinc/slots/18/ic_ancil_12175254.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/18/ic_ancil_12175254.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0398ed82e0806656d6fdc93c14ea537f to: /var/lib/boinc/slots/18/ifsdata/ifsdata_12175254.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/18/ifsdata/ifsdata_12175254.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/18/159l_2/clim_data_12175254.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/18/159l_2/clim_data_12175254.zip
Checking for progress XML file: /var/lib/boinc/slots/18/progress_file_12175254.xml
Opened progress file ok : /var/lib/boinc/slots/18/progress_file_12175254.xml
-- Model is restarting --
Adjusting last_iter, 2572, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/18/progress_file_12175254.xml
last_cpu_time: 57255
upload_file_number: 83
last_iter: 2567
last_upload: 9244800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929
The child process has been launched with process id: 6334
Executing the command: /var/lib/boinc/slots/18/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6334:6334] [20221222:232659:1671751619:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0610
(argv4) batchid: 958
(argv5) wuid: 12175254
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/18
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/18/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/18/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_af179f5e57efa2a19ff4c1024b893661 to: /var/lib/boinc/slots/18/oifs_43r3_ps_0610_1989050100_123_958_12175254.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/18/oifs_43r3_ps_0610_1989050100_123_958_12175254.zip
ic_ancil_file: ic_ancil_12175254
ifsdata_file: ifsdata_12175254
climate_data_file: clim_data_12175254
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d87a761f09f2bb60f129754c0c0b318a to: /var/lib/boinc/slots/18/ic_ancil_12175254.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/18/ic_ancil_12175254.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0398ed82e0806656d6fdc93c14ea537f to: /var/lib/boinc/slots/18/ifsdata/ifsdata_12175254.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/18/ifsdata/ifsdata_12175254.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/18/159l_2/clim_data_12175254.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/18/159l_2/clim_data_12175254.zip
Checking for progress XML file: /var/lib/boinc/slots/18/progress_file_12175254.xml
Opened progress file ok : /var/lib/boinc/slots/18/progress_file_12175254.xml
-- Model is restarting --
Adjusting last_iter, 2568, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/18/progress_file_12175254.xml
last_cpu_time: 57263
upload_file_number: 83
last_iter: 2567
last_upload: 9244800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929
The child process has been launched with process id: 7019
Executing the command: /var/lib/boinc/slots/18/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7019:7019] [20221223:122633:1671798393:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:27:18 STEP 2568 H=2568:00 +CPU= 39.307
  12:27:40 STEP 2569 H=2569:00 +CPU= 22.397
  12:28:02 STEP 2570 H=2570:00 +CPU= 21.357
  12:28:30 STEP 2571 H=2571:00 +CPU= 27.426
  12:28:52 STEP 2572 H=2572:00 +CPU= 20.339
  12:29:14 STEP 2573 H=2573:00 +CPU= 20.054
  12:29:46 STEP 2574 H=2574:00 +CPU= 31.636
  12:30:07 STEP 2575 H=2575:00 +CPU= 20.008
  12:30:28 STEP 2576 H=2576:00 +CPU= 21.116
  12:31:00 STEP 2577 H=2577:00 +CPU= 30.658
  12:31:21 STEP 2578 H=2578:00 +CPU= 20.947
  12:31:44 STEP 2579 H=2579:00 +CPU= 21.772
  12:32:32 STEP 2580 H=2580:00 +CPU= 40.362
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002580
  12:32:59 STEP 2581 H=2581:00 +CPU= 23.713
  12:33:31 STEP 2582 H=2582:00 +CPU= 25.199
  12:34:13 STEP 2583 H=2583:00 +CPU= 35.760
  12:34:43 STEP 2584 H=2584:00 +CPU= 25.682
  12:35:11 STEP 2585 H=2585:00 +CPU= 23.670
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:36:10 STEP 2586 H=2586:00 +CPU= 35.729
  12:36:42 STEP 2587 H=2587:00 +CPU= 23.873
  12:37:15 STEP 2588 H=2588:00 +CPU= 24.990
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:38:36 STEP 2589 H=2589:00 +CPU= 34.261
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:39:32 STEP 2590 H=2590:00 +CPU= 24.643
  12:40:05 STEP 2591 H=2591:00 +CPU= 26.008
  12:40:51 STEP 2592 H=2592:00 +CPU= 39.980
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_83.zip
Uploading the intermediate file: upload_file_83.zip
  12:41:17 STEP 2593 H=2593:00 +CPU= 22.354
  12:41:46 STEP 2594 H=2594:00 +CPU= 23.453
  12:42:25 STEP 2595 H=2595:00 +CPU= 33.056
  12:42:55 STEP 2596 H=2596:00 +CPU= 23.827
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:43:34 STEP 2597 H=2597:00 +CPU= 22.015
  12:44:17 STEP 2598 H=2598:00 +CPU= 34.952
  12:44:46 STEP 2599 H=2599:00 +CPU= 23.624
  12:45:16 STEP 2600 H=2600:00 +CPU= 23.952
  12:45:57 STEP 2601 H=2601:00 +CPU= 34.312
  12:46:27 STEP 2602 H=2602:00 +CPU= 24.059
  12:46:54 STEP 2603 H=2603:00 +CPU= 22.426
  12:47:31 STEP 2604 H=2604:00 +CPU= 32.929
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002604
  12:47:55 STEP 2605 H=2605:00 +CPU= 19.639
  12:48:23 STEP 2606 H=2606:00 +CPU= 21.371
  12:48:57 STEP 2607 H=2607:00 +CPU= 29.026
  12:49:21 STEP 2608 H=2608:00 +CPU= 19.948
  12:49:44 STEP 2609 H=2609:00 +CPU= 19.338
  12:50:25 STEP 2610 H=2610:00 +CPU= 31.135
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:51:17 STEP 2611 H=2611:00 +CPU= 22.478
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:51:56 STEP 2612 H=2612:00 +CPU= 23.284
  12:52:38 STEP 2613 H=2613:00 +CPU= 34.553
  12:53:07 STEP 2614 H=2614:00 +CPU= 24.069
  12:53:38 STEP 2615 H=2615:00 +CPU= 24.571
  12:54:28 STEP 2616 H=2616:00 +CPU= 40.854
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_84.zip
Uploading the intermediate file: upload_file_84.zip
  12:54:58 STEP 2617 H=2617:00 +CPU= 23.826
  12:55:27 STEP 2618 H=2618:00 +CPU= 23.845
  12:56:08 STEP 2619 H=2619:00 +CPU= 34.133
  12:56:36 STEP 2620 H=2620:00 +CPU= 23.670
  12:57:05 STEP 2621 H=2621:00 +CPU= 23.509
  12:57:48 STEP 2622 H=2622:00 +CPU= 35.248
  12:58:17 STEP 2623 H=2623:00 +CPU= 23.308
  12:58:45 STEP 2624 H=2624:00 +CPU= 23.404
  12:59:23 STEP 2625 H=2625:00 +CPU= 33.189
  12:59:49 STEP 2626 H=2626:00 +CPU= 21.552
  13:00:13 STEP 2627 H=2627:00 +CPU= 20.652
  13:00:56 STEP 2628 H=2628:00 +CPU= 36.128
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002628
  13:01:23 STEP 2629 H=2629:00 +CPU= 22.023
  13:01:46 STEP 2630 H=2630:00 +CPU= 20.523
  13:02:22 STEP 2631 H=2631:00 +CPU= 31.117
  13:02:49 STEP 2632 H=2632:00 +CPU= 22.401
  13:03:10 STEP 2633 H=2633:00 +CPU= 19.681
  13:03:48 STEP 2634 H=2634:00 +CPU= 32.458
  13:04:14 STEP 2635 H=2635:00 +CPU= 22.437
  13:04:41 STEP 2636 H=2636:00 +CPU= 22.700
  13:05:20 STEP 2637 H=2637:00 +CPU= 33.187
  13:05:49 STEP 2638 H=2638:00 +CPU= 23.157
  13:06:16 STEP 2639 H=2639:00 +CPU= 22.780
  13:07:04 STEP 2640 H=2640:00 +CPU= 39.527
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_85.zip
Uploading the intermediate file: upload_file_85.zip
  13:07:30 STEP 2641 H=2641:00 +CPU= 22.085
  13:07:57 STEP 2642 H=2642:00 +CPU= 22.685
  13:08:37 STEP 2643 H=2643:00 +CPU= 33.426
  13:09:04 STEP 2644 H=2644:00 +CPU= 22.894
  13:09:30 STEP 2645 H=2645:00 +CPU= 22.152
  13:10:08 STEP 2646 H=2646:00 +CPU= 32.665
  13:10:31 STEP 2647 H=2647:00 +CPU= 20.323
  13:10:56 STEP 2648 H=2648:00 +CPU= 21.334
  13:11:32 STEP 2649 H=2649:00 +CPU= 31.604
  13:11:58 STEP 2650 H=2650:00 +CPU= 22.066
  13:12:24 STEP 2651 H=2651:00 +CPU= 22.251
  13:13:08 STEP 2652 H=2652:00 +CPU= 36.964
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002652
  13:13:33 STEP 2653 H=2653:00 +CPU= 22.211
  13:13:59 STEP 2654 H=2654:00 +CPU= 22.426
  13:14:35 STEP 2655 H=2655:00 +CPU= 32.191
  13:15:03 STEP 2656 H=2656:00 +CPU= 24.181
  13:15:30 STEP 2657 H=2657:00 +CPU= 23.266
  13:16:09 STEP 2658 H=2658:00 +CPU= 34.254
  13:16:37 STEP 2659 H=2659:00 +CPU= 24.672
  13:17:04 STEP 2660 H=2660:00 +CPU= 23.192
  13:17:43 STEP 2661 H=2661:00 +CPU= 34.058
  13:18:10 STEP 2662 H=2662:00 +CPU= 22.952
  13:18:39 STEP 2663 H=2663:00 +CPU= 24.200
  13:19:26 STEP 2664 H=2664:00 +CPU= 39.922
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_86.zip
Uploading the intermediate file: upload_file_86.zip
  13:19:54 STEP 2665 H=2665:00 +CPU= 24.100
  13:20:24 STEP 2666 H=2666:00 +CPU= 24.748
  13:21:08 STEP 2667 H=2667:00 +CPU= 35.891
  13:21:37 STEP 2668 H=2668:00 +CPU= 24.907
  13:22:03 STEP 2669 H=2669:00 +CPU= 22.637
  13:22:46 STEP 2670 H=2670:00 +CPU= 36.144
  13:23:14 STEP 2671 H=2671:00 +CPU= 24.052
  13:23:44 STEP 2672 H=2672:00 +CPU= 25.126
  13:24:25 STEP 2673 H=2673:00 +CPU= 35.523
  13:24:55 STEP 2674 H=2674:00 +CPU= 25.318
  13:25:25 STEP 2675 H=2675:00 +CPU= 25.223
  13:26:12 STEP 2676 H=2676:00 +CPU= 40.903
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002676
  13:26:45 STEP 2677 H=2677:00 +CPU= 27.500
  13:27:18 STEP 2678 H=2678:00 +CPU= 26.651
  13:28:10 STEP 2679 H=2679:00 +CPU= 41.441
  13:28:44 STEP 2680 H=2680:00 +CPU= 27.518
  13:29:18 STEP 2681 H=2681:00 +CPU= 27.799
  13:30:09 STEP 2682 H=2682:00 +CPU= 41.957
  13:30:43 STEP 2683 H=2683:00 +CPU= 27.389
  13:31:17 STEP 2684 H=2684:00 +CPU= 27.829
  13:32:04 STEP 2685 H=2685:00 +CPU= 40.224
  13:32:38 STEP 2686 H=2686:00 +CPU= 28.102
  13:33:12 STEP 2687 H=2687:00 +CPU= 27.883
  13:34:10 STEP 2688 H=2688:00 +CPU= 48.295
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_87.zip
Uploading the intermediate file: upload_file_87.zip
  13:34:42 STEP 2689 H=2689:00 +CPU= 26.973
  13:35:16 STEP 2690 H=2690:00 +CPU= 28.065
  13:36:05 STEP 2691 H=2691:00 +CPU= 40.783
  13:36:39 STEP 2692 H=2692:00 +CPU= 28.258
  13:37:12 STEP 2693 H=2693:00 +CPU= 27.685
  13:38:00 STEP 2694 H=2694:00 +CPU= 40.663
  13:38:32 STEP 2695 H=2695:00 +CPU= 27.082
  13:39:06 STEP 2696 H=2696:00 +CPU= 27.956
  13:39:55 STEP 2697 H=2697:00 +CPU= 39.723
  13:40:30 STEP 2698 H=2698:00 +CPU= 28.069
  13:41:03 STEP 2699 H=2699:00 +CPU= 27.178
  13:41:58 STEP 2700 H=2700:00 +CPU= 45.360
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002700
  13:42:32 STEP 2701 H=2701:00 +CPU= 27.698
  13:43:07 STEP 2702 H=2702:00 +CPU= 29.576
  13:43:57 STEP 2703 H=2703:00 +CPU= 41.341
  13:44:32 STEP 2704 H=2704:00 +CPU= 28.286
  13:45:06 STEP 2705 H=2705:00 +CPU= 28.200
  13:45:56 STEP 2706 H=2706:00 +CPU= 41.647
  13:46:28 STEP 2707 H=2707:00 +CPU= 26.811
  13:47:02 STEP 2708 H=2708:00 +CPU= 28.322
  13:47:52 STEP 2709 H=2709:00 +CPU= 41.216
  13:48:26 STEP 2710 H=2710:00 +CPU= 28.644
  13:49:01 STEP 2711 H=2711:00 +CPU= 28.861
  13:49:59 STEP 2712 H=2712:00 +CPU= 48.662
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_88.zip
Uploading the intermediate file: upload_file_88.zip
  13:50:31 STEP 2713 H=2713:00 +CPU= 26.403
  13:51:06 STEP 2714 H=2714:00 +CPU= 28.865
  13:51:55 STEP 2715 H=2715:00 +CPU= 41.030
  13:52:27 STEP 2716 H=2716:00 +CPU= 27.363
  13:53:00 STEP 2717 H=2717:00 +CPU= 27.428
  13:53:49 STEP 2718 H=2718:00 +CPU= 41.142
  13:54:19 STEP 2719 H=2719:00 +CPU= 25.998
  13:54:54 STEP 2720 H=2720:00 +CPU= 28.815
  13:55:42 STEP 2721 H=2721:00 +CPU= 40.884
  13:56:16 STEP 2722 H=2722:00 +CPU= 28.129
  13:56:49 STEP 2723 H=2723:00 +CPU= 27.139
  13:57:44 STEP 2724 H=2724:00 +CPU= 46.651
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002724
  13:58:15 STEP 2725 H=2725:00 +CPU= 25.652
  13:58:45 STEP 2726 H=2726:00 +CPU= 25.473
  13:59:30 STEP 2727 H=2727:00 +CPU= 37.561
  13:59:58 STEP 2728 H=2728:00 +CPU= 24.786
  14:00:27 STEP 2729 H=2729:00 +CPU= 24.233
  14:01:13 STEP 2730 H=2730:00 +CPU= 38.328
  14:01:39 STEP 2731 H=2731:00 +CPU= 23.302
  14:02:07 STEP 2732 H=2732:00 +CPU= 24.135
  14:02:52 STEP 2733 H=2733:00 +CPU= 37.695
  14:03:20 STEP 2734 H=2734:00 +CPU= 23.944
  14:03:51 STEP 2735 H=2735:00 +CPU= 24.982
  14:04:44 STEP 2736 H=2736:00 +CPU= 43.318
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_89.zip
Uploading the intermediate file: upload_file_89.zip
  14:05:12 STEP 2737 H=2737:00 +CPU= 23.106
  14:05:42 STEP 2738 H=2738:00 +CPU= 24.579
  14:06:26 STEP 2739 H=2739:00 +CPU= 37.089
  14:06:54 STEP 2740 H=2740:00 +CPU= 23.933
  14:07:23 STEP 2741 H=2741:00 +CPU= 24.198
  14:08:07 STEP 2742 H=2742:00 +CPU= 37.740
  14:08:35 STEP 2743 H=2743:00 +CPU= 23.185
  14:09:04 STEP 2744 H=2744:00 +CPU= 24.447
  14:09:48 STEP 2745 H=2745:00 +CPU= 37.145
  14:10:17 STEP 2746 H=2746:00 +CPU= 23.857
  14:10:46 STEP 2747 H=2747:00 +CPU= 23.644
  14:11:34 STEP 2748 H=2748:00 +CPU= 40.939
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002748
  14:12:03 STEP 2749 H=2749:00 +CPU= 23.309
  14:12:32 STEP 2750 H=2750:00 +CPU= 24.673
  14:13:17 STEP 2751 H=2751:00 +CPU= 37.028
  14:13:46 STEP 2752 H=2752:00 +CPU= 24.192
  14:14:14 STEP 2753 H=2753:00 +CPU= 24.255
  14:14:57 STEP 2754 H=2754:00 +CPU= 37.620
  14:15:23 STEP 2755 H=2755:00 +CPU= 23.037
  14:15:49 STEP 2756 H=2756:00 +CPU= 24.449
  14:16:31 STEP 2757 H=2757:00 +CPU= 37.325
  14:16:58 STEP 2758 H=2758:00 +CPU= 23.980
  14:17:25 STEP 2759 H=2759:00 +CPU= 24.572
  14:18:13 STEP 2760 H=2760:00 +CPU= 42.982
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_90.zip
Uploading the intermediate file: upload_file_90.zip
  14:18:39 STEP 2761 H=2761:00 +CPU= 23.084
  14:19:08 STEP 2762 H=2762:00 +CPU= 24.627
  14:19:50 STEP 2763 H=2763:00 +CPU= 37.163
  14:20:14 STEP 2764 H=2764:00 +CPU= 23.428
  14:20:39 STEP 2765 H=2765:00 +CPU= 23.335
  14:21:18 STEP 2766 H=2766:00 +CPU= 37.175
  14:21:42 STEP 2767 H=2767:00 +CPU= 22.542
  14:22:07 STEP 2768 H=2768:00 +CPU= 24.090
  14:22:46 STEP 2769 H=2769:00 +CPU= 36.459
  14:23:13 STEP 2770 H=2770:00 +CPU= 23.860
  14:23:41 STEP 2771 H=2771:00 +CPU= 24.214
  14:24:30 STEP 2772 H=2772:00 +CPU= 41.737
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002772
  14:24:54 STEP 2773 H=2773:00 +CPU= 22.785
  14:25:23 STEP 2774 H=2774:00 +CPU= 24.769
  14:26:06 STEP 2775 H=2775:00 +CPU= 36.970
  14:26:34 STEP 2776 H=2776:00 +CPU= 23.699
  14:27:01 STEP 2777 H=2777:00 +CPU= 23.487
  14:27:44 STEP 2778 H=2778:00 +CPU= 37.359
  14:28:12 STEP 2779 H=2779:00 +CPU= 23.147
  14:28:40 STEP 2780 H=2780:00 +CPU= 24.948
  14:29:26 STEP 2781 H=2781:00 +CPU= 37.298
  14:29:55 STEP 2782 H=2782:00 +CPU= 24.109
  14:30:25 STEP 2783 H=2783:00 +CPU= 24.562
  14:31:18 STEP 2784 H=2784:00 +CPU= 42.995
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_91.zip
Uploading the intermediate file: upload_file_91.zip
  14:31:46 STEP 2785 H=2785:00 +CPU= 22.993
  14:32:16 STEP 2786 H=2786:00 +CPU= 24.442
  14:33:02 STEP 2787 H=2787:00 +CPU= 37.077
  14:33:31 STEP 2788 H=2788:00 +CPU= 24.013
  14:34:00 STEP 2789 H=2789:00 +CPU= 23.848
  14:34:46 STEP 2790 H=2790:00 +CPU= 38.079
  14:35:14 STEP 2791 H=2791:00 +CPU= 23.664
  14:35:45 STEP 2792 H=2792:00 +CPU= 24.587
  14:36:29 STEP 2793 H=2793:00 +CPU= 36.902
  14:36:58 STEP 2794 H=2794:00 +CPU= 24.031
  14:37:27 STEP 2795 H=2795:00 +CPU= 24.300
  14:38:16 STEP 2796 H=2796:00 +CPU= 40.744
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002796
  14:38:40 STEP 2797 H=2797:00 +CPU= 22.226
  14:39:06 STEP 2798 H=2798:00 +CPU= 24.276
  14:39:45 STEP 2799 H=2799:00 +CPU= 35.994
  14:40:10 STEP 2800 H=2800:00 +CPU= 23.132
  14:40:38 STEP 2801 H=2801:00 +CPU= 23.711
  14:41:23 STEP 2802 H=2802:00 +CPU= 37.225
  14:41:49 STEP 2803 H=2803:00 +CPU= 23.129
  14:42:21 STEP 2804 H=2804:00 +CPU= 25.699
  14:43:12 STEP 2805 H=2805:00 +CPU= 40.157
  14:43:44 STEP 2806 H=2806:00 +CPU= 25.346
  14:44:18 STEP 2807 H=2807:00 +CPU= 27.001
  14:45:16 STEP 2808 H=2808:00 +CPU= 46.491
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_92.zip
Uploading the intermediate file: upload_file_92.zip
Uploading trickle at timestep: 10108800
  14:45:46 STEP 2809 H=2809:00 +CPU= 24.028
  14:46:21 STEP 2810 H=2810:00 +CPU= 26.679
  14:47:13 STEP 2811 H=2811:00 +CPU= 40.454
  14:47:46 STEP 2812 H=2812:00 +CPU= 25.497
  14:48:15 STEP 2813 H=2813:00 +CPU= 23.804
  14:49:01 STEP 2814 H=2814:00 +CPU= 38.035
  14:49:29 STEP 2815 H=2815:00 +CPU= 23.341
  14:49:58 STEP 2816 H=2816:00 +CPU= 24.433
  14:50:44 STEP 2817 H=2817:00 +CPU= 37.094
  14:51:13 STEP 2818 H=2818:00 +CPU= 23.981
  14:51:41 STEP 2819 H=2819:00 +CPU= 23.570
  14:52:31 STEP 2820 H=2820:00 +CPU= 41.588
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002820
  14:52:59 STEP 2821 H=2821:00 +CPU= 23.146
  14:53:29 STEP 2822 H=2822:00 +CPU= 24.424
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0610
(argv4) batchid: 958
(argv5) wuid: 12175254
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/18
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/18/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/18/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_af179f5e57efa2a19ff4c1024b893661 to: /var/lib/boinc/slots/18/oifs_43r3_ps_0610_1989050100_123_958_12175254.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/18/oifs_43r3_ps_0610_1989050100_123_958_12175254.zip
ic_ancil_file: ic_ancil_12175254
ifsdata_file: ifsdata_12175254
climate_data_file: clim_data_12175254
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d87a761f09f2bb60f129754c0c0b318a to: /var/lib/boinc/slots/18/ic_ancil_12175254.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/18/ic_ancil_12175254.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0398ed82e0806656d6fdc93c14ea537f to: /var/lib/boinc/slots/18/ifsdata/ifsdata_12175254.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/18/ifsdata/ifsdata_12175254.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/18/159l_2/clim_data_12175254.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/18/159l_2/clim_data_12175254.zip
Checking for progress XML file: /var/lib/boinc/slots/18/progress_file_12175254.xml
Opened progress file ok : /var/lib/boinc/slots/18/progress_file_12175254.xml
-- Model is restarting --
Adjusting last_iter, 2823, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/18/progress_file_12175254.xml
last_cpu_time: 64614
upload_file_number: 93
last_iter: 2807
last_upload: 10108800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929
The child process has been launched with process id: 3791
Executing the command: /var/lib/boinc/slots/18/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3791:3791] [20221223:183811:1671820691:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:38:58 STEP 2808 H=2808:00 +CPU= 38.933
  18:39:24 STEP 2809 H=2809:00 +CPU= 24.923
  18:39:50 STEP 2810 H=2810:00 +CPU= 23.957
  18:40:27 STEP 2811 H=2811:00 +CPU= 34.612
  18:40:53 STEP 2812 H=2812:00 +CPU= 24.449
  18:41:18 STEP 2813 H=2813:00 +CPU= 23.697
  18:41:55 STEP 2814 H=2814:00 +CPU= 34.450
  18:42:20 STEP 2815 H=2815:00 +CPU= 23.854
  18:42:46 STEP 2816 H=2816:00 +CPU= 24.351
  18:43:22 STEP 2817 H=2817:00 +CPU= 33.781
  18:43:47 STEP 2818 H=2818:00 +CPU= 24.219
  18:44:12 STEP 2819 H=2819:00 +CPU= 23.137
  18:44:52 STEP 2820 H=2820:00 +CPU= 37.927
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002820
  18:45:17 STEP 2821 H=2821:00 +CPU= 24.020
  18:45:43 STEP 2822 H=2822:00 +CPU= 24.387
  18:46:19 STEP 2823 H=2823:00 +CPU= 34.120
  18:46:45 STEP 2824 H=2824:00 +CPU= 24.587
  18:47:11 STEP 2825 H=2825:00 +CPU= 23.930
  18:47:47 STEP 2826 H=2826:00 +CPU= 34.772
  18:48:12 STEP 2827 H=2827:00 +CPU= 23.592
  18:48:38 STEP 2828 H=2828:00 +CPU= 24.493
  18:49:14 STEP 2829 H=2829:00 +CPU= 33.964
  18:49:39 STEP 2830 H=2830:00 +CPU= 23.854
  18:50:05 STEP 2831 H=2831:00 +CPU= 24.632
  18:50:48 STEP 2832 H=2832:00 +CPU= 40.418
Moving to projects directory: /var/lib/boinc/slots/18/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc/slots/18/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc/slots/18/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175254/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0610_1989050100_123_958_12175254_0_r1089740929_93.zip
Uploading the intermediate file: upload_file_93.zip
  18:51:14 STEP 2833 H=2833:00 +CPU= 23.530
  18:51:39 STEP 2834 H=2834:00 +CPU= 24.099
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:50:44 1531398 22259943 oifs_43r3_ps_0610_1989050100_123_958_12175254_0 10,108,800 64,200 0.0064
09 May 2023 15:48:41 1531398 22259943 oifs_43r3_ps_0610_1989050100_123_958_12175254_0 8,553,600 50,711 0.0059
09 May 2023 15:47:46 1531398 22259943 oifs_43r3_ps_0610_1989050100_123_958_12175254_0 6,480,000 32,503 0.0050
09 May 2023 15:47:35 1531398 22259943 oifs_43r3_ps_0610_1989050100_123_958_12175254_0 5,616,000 26,189 0.0047
09 May 2023 15:47:24 1531398 22259943 oifs_43r3_ps_0610_1989050100_123_958_12175254_0 4,752,000 19,881 0.0042
09 May 2023 15:47:13 1531398 22259943 oifs_43r3_ps_0610_1989050100_123_958_12175254_0 3,888,000 13,568 0.0035
09 May 2023 15:47:07 1531398 22259943 oifs_43r3_ps_0610_1989050100_123_958_12175254_0 3,024,000 7,245 0.0024
09 May 2023 15:46:35 1531398 22259943 oifs_43r3_ps_0610_1989050100_123_958_12175254_0 1,728,000 11,830 0.0068
09 May 2023 15:46:25 1531398 22259943 oifs_43r3_ps_0610_1989050100_123_958_12175254_0 864,000 6,088 0.0070


©2024 climateprediction.net