climateprediction.net home page
Task 22259952

Task 22259952

Name oifs_43r3_ps_0619_1989050100_123_958_12175263_0
Workunit 12175263
Created 21 Dec 2022, 17:02:15 UTC
Sent 21 Dec 2022, 17:10:47 UTC
Report deadline 20 Jan 2023, 17:10:47 UTC
Received 24 Dec 2022, 1:28:53 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1531398
Run time 23 hours 12 min 13 sec
CPU time 18 hours 10 min 36 sec
Validate state Invalid
Credit 11,491.20
Device peak FLOPS 7.99 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,792.79 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,191.72 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
 STEP 2398 H=2398:00 +CPU= 23.389
  21:45:06 STEP 2399 H=2399:00 +CPU= 24.719
  21:45:49 STEP 2400 H=2400:00 +CPU= 40.362
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_80.zip
Uploading the intermediate file: upload_file_80.zip
  21:46:14 STEP 2401 H=2401:00 +CPU= 22.772
  21:46:39 STEP 2402 H=2402:00 +CPU= 24.049
  21:47:15 STEP 2403 H=2403:00 +CPU= 34.374
  21:47:38 STEP 2404 H=2404:00 +CPU= 22.962
  21:48:03 STEP 2405 H=2405:00 +CPU= 24.039
  21:48:41 STEP 2406 H=2406:00 +CPU= 36.206
  21:49:04 STEP 2407 H=2407:00 +CPU= 22.351
  21:49:29 STEP 2408 H=2408:00 +CPU= 23.701
  21:50:06 STEP 2409 H=2409:00 +CPU= 34.853
  21:50:31 STEP 2410 H=2410:00 +CPU= 23.721
  21:50:55 STEP 2411 H=2411:00 +CPU= 23.455
  21:51:36 STEP 2412 H=2412:00 +CPU= 39.487
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002412
  21:52:00 STEP 2413 H=2413:00 +CPU= 22.506
  21:52:24 STEP 2414 H=2414:00 +CPU= 23.587
  21:53:01 STEP 2415 H=2415:00 +CPU= 34.783
  21:53:26 STEP 2416 H=2416:00 +CPU= 23.875
  21:53:50 STEP 2417 H=2417:00 +CPU= 23.093
  21:54:27 STEP 2418 H=2418:00 +CPU= 35.409
  21:54:51 STEP 2419 H=2419:00 +CPU= 23.114
  21:55:17 STEP 2420 H=2420:00 +CPU= 25.031
  21:55:53 STEP 2421 H=2421:00 +CPU= 34.806
  21:56:18 STEP 2422 H=2422:00 +CPU= 23.686
  21:56:44 STEP 2423 H=2423:00 +CPU= 24.536
  21:57:27 STEP 2424 H=2424:00 +CPU= 40.931
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_81.zip
Uploading the intermediate file: upload_file_81.zip
  21:57:52 STEP 2425 H=2425:00 +CPU= 23.427
  21:58:17 STEP 2426 H=2426:00 +CPU= 24.049
  21:58:54 STEP 2427 H=2427:00 +CPU= 35.359
  21:59:20 STEP 2428 H=2428:00 +CPU= 24.360
  21:59:44 STEP 2429 H=2429:00 +CPU= 23.300
  22:00:22 STEP 2430 H=2430:00 +CPU= 35.983
  22:00:46 STEP 2431 H=2431:00 +CPU= 22.913
  22:01:11 STEP 2432 H=2432:00 +CPU= 24.018
  22:01:49 STEP 2433 H=2433:00 +CPU= 36.268
  22:02:14 STEP 2434 H=2434:00 +CPU= 23.659
  22:02:38 STEP 2435 H=2435:00 +CPU= 23.280
  22:03:18 STEP 2436 H=2436:00 +CPU= 38.699
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002436
  22:03:43 STEP 2437 H=2437:00 +CPU= 23.232
  22:04:08 STEP 2438 H=2438:00 +CPU= 24.078
  22:04:44 STEP 2439 H=2439:00 +CPU= 34.592
  22:05:09 STEP 2440 H=2440:00 +CPU= 23.557
  22:05:33 STEP 2441 H=2441:00 +CPU= 23.540
  22:06:11 STEP 2442 H=2442:00 +CPU= 36.445
  22:06:36 STEP 2443 H=2443:00 +CPU= 23.707
  22:07:01 STEP 2444 H=2444:00 +CPU= 23.862
  22:07:38 STEP 2445 H=2445:00 +CPU= 35.514
  22:08:03 STEP 2446 H=2446:00 +CPU= 23.845
  22:08:29 STEP 2447 H=2447:00 +CPU= 24.865
  22:09:11 STEP 2448 H=2448:00 +CPU= 40.192
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_82.zip
Uploading the intermediate file: upload_file_82.zip
  22:09:35 STEP 2449 H=2449:00 +CPU= 22.950
  22:10:01 STEP 2450 H=2450:00 +CPU= 23.933
  22:10:37 STEP 2451 H=2451:00 +CPU= 34.194
  22:11:02 STEP 2452 H=2452:00 +CPU= 24.194
  22:11:25 STEP 2453 H=2453:00 +CPU= 22.549
  22:12:02 STEP 2454 H=2454:00 +CPU= 35.332
  22:12:27 STEP 2455 H=2455:00 +CPU= 23.600
  22:12:51 STEP 2456 H=2456:00 +CPU= 23.914
  22:13:28 STEP 2457 H=2457:00 +CPU= 35.311
  22:13:53 STEP 2458 H=2458:00 +CPU= 23.899
  22:14:17 STEP 2459 H=2459:00 +CPU= 22.948
  22:14:58 STEP 2460 H=2460:00 +CPU= 39.548
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002460
  22:15:23 STEP 2461 H=2461:00 +CPU= 23.183
  22:15:47 STEP 2462 H=2462:00 +CPU= 23.075
  22:16:22 STEP 2463 H=2463:00 +CPU= 34.030
  22:16:47 STEP 2464 H=2464:00 +CPU= 24.036
  22:17:11 STEP 2465 H=2465:00 +CPU= 22.886
  22:17:49 STEP 2466 H=2466:00 +CPU= 36.534
  22:18:14 STEP 2467 H=2467:00 +CPU= 23.893
  22:18:39 STEP 2468 H=2468:00 +CPU= 23.273
  22:19:15 STEP 2469 H=2469:00 +CPU= 34.781
  22:19:40 STEP 2470 H=2470:00 +CPU= 23.807
  22:20:06 STEP 2471 H=2471:00 +CPU= 24.451
  22:20:51 STEP 2472 H=2472:00 +CPU= 40.215
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_83.zip
Uploading the intermediate file: upload_file_83.zip
  22:21:17 STEP 2473 H=2473:00 +CPU= 22.292
  22:21:44 STEP 2474 H=2474:00 +CPU= 23.369
  22:22:26 STEP 2475 H=2475:00 +CPU= 33.611
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0619
(argv4) batchid: 958
(argv5) wuid: 12175263
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/25
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/25/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/25/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_12e847c90efe210604a2d1f0e5af7906 to: /var/lib/boinc/slots/25/oifs_43r3_ps_0619_1989050100_123_958_12175263.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/25/oifs_43r3_ps_0619_1989050100_123_958_12175263.zip
ic_ancil_file: ic_ancil_12175263
ifsdata_file: ifsdata_12175263
climate_data_file: clim_data_12175263
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f585589a0ba9d745e24c020ccc1664fd to: /var/lib/boinc/slots/25/ic_ancil_12175263.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/25/ic_ancil_12175263.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f7c2c5939852b7f30e230b479ceb0a43 to: /var/lib/boinc/slots/25/ifsdata/ifsdata_12175263.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/25/ifsdata/ifsdata_12175263.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/25/159l_2/clim_data_12175263.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/25/159l_2/clim_data_12175263.zip
Checking for progress XML file: /var/lib/boinc/slots/25/progress_file_12175263.xml
Opened progress file ok : /var/lib/boinc/slots/25/progress_file_12175263.xml
-- Model is restarting --
Adjusting last_iter, 2476, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/25/progress_file_12175263.xml
last_cpu_time: 57234
upload_file_number: 84
last_iter: 2471
last_upload: 8899200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959
The child process has been launched with process id: 6325
Executing the command: /var/lib/boinc/slots/25/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:6325:6325] [20221222:232658:1671751618:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0619
(argv4) batchid: 958
(argv5) wuid: 12175263
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/25
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/25/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/25/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_12e847c90efe210604a2d1f0e5af7906 to: /var/lib/boinc/slots/25/oifs_43r3_ps_0619_1989050100_123_958_12175263.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/25/oifs_43r3_ps_0619_1989050100_123_958_12175263.zip
ic_ancil_file: ic_ancil_12175263
ifsdata_file: ifsdata_12175263
climate_data_file: clim_data_12175263
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f585589a0ba9d745e24c020ccc1664fd to: /var/lib/boinc/slots/25/ic_ancil_12175263.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/25/ic_ancil_12175263.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f7c2c5939852b7f30e230b479ceb0a43 to: /var/lib/boinc/slots/25/ifsdata/ifsdata_12175263.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/25/ifsdata/ifsdata_12175263.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/25/159l_2/clim_data_12175263.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/25/159l_2/clim_data_12175263.zip
Checking for progress XML file: /var/lib/boinc/slots/25/progress_file_12175263.xml
Opened progress file ok : /var/lib/boinc/slots/25/progress_file_12175263.xml
-- Model is restarting --
Adjusting last_iter, 2472, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/25/progress_file_12175263.xml
last_cpu_time: 57242
upload_file_number: 84
last_iter: 2471
last_upload: 8899200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959
The child process has been launched with process id: 7017
Executing the command: /var/lib/boinc/slots/25/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:7017:7017] [20221223:122632:1671798392:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:27:18 STEP 2472 H=2472:00 +CPU= 40.828
  12:27:41 STEP 2473 H=2473:00 +CPU= 22.847
  12:28:02 STEP 2474 H=2474:00 +CPU= 21.037
  12:28:29 STEP 2475 H=2475:00 +CPU= 26.038
  12:28:51 STEP 2476 H=2476:00 +CPU= 20.342
  12:29:13 STEP 2477 H=2477:00 +CPU= 20.117
  12:29:45 STEP 2478 H=2478:00 +CPU= 31.378
  12:30:05 STEP 2479 H=2479:00 +CPU= 19.795
  12:30:27 STEP 2480 H=2480:00 +CPU= 20.967
  12:30:59 STEP 2481 H=2481:00 +CPU= 31.185
  12:31:21 STEP 2482 H=2482:00 +CPU= 21.632
  12:31:44 STEP 2483 H=2483:00 +CPU= 21.757
  12:32:30 STEP 2484 H=2484:00 +CPU= 39.677
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002484
  12:32:57 STEP 2485 H=2485:00 +CPU= 23.822
  12:33:29 STEP 2486 H=2486:00 +CPU= 25.078
  12:34:11 STEP 2487 H=2487:00 +CPU= 36.322
  12:34:41 STEP 2488 H=2488:00 +CPU= 25.884
  12:35:08 STEP 2489 H=2489:00 +CPU= 23.681
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:36:07 STEP 2490 H=2490:00 +CPU= 35.703
  12:36:38 STEP 2491 H=2491:00 +CPU= 23.790
  12:37:10 STEP 2492 H=2492:00 +CPU= 24.822
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:38:28 STEP 2493 H=2493:00 +CPU= 32.944
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:39:10 STEP 2494 H=2494:00 +CPU= 22.828
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:39:51 STEP 2495 H=2495:00 +CPU= 23.469
  12:40:40 STEP 2496 H=2496:00 +CPU= 41.436
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_84.zip
Uploading the intermediate file: upload_file_84.zip
  12:41:04 STEP 2497 H=2497:00 +CPU= 21.599
  12:41:33 STEP 2498 H=2498:00 +CPU= 23.887
  12:42:12 STEP 2499 H=2499:00 +CPU= 32.699
  12:42:41 STEP 2500 H=2500:00 +CPU= 23.982
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:43:22 STEP 2501 H=2501:00 +CPU= 22.245
  12:44:04 STEP 2502 H=2502:00 +CPU= 34.062
  12:44:33 STEP 2503 H=2503:00 +CPU= 23.836
  12:45:03 STEP 2504 H=2504:00 +CPU= 24.043
  12:45:44 STEP 2505 H=2505:00 +CPU= 33.680
  12:46:13 STEP 2506 H=2506:00 +CPU= 24.293
  12:46:41 STEP 2507 H=2507:00 +CPU= 22.937
  12:47:20 STEP 2508 H=2508:00 +CPU= 33.931
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002508
  12:47:43 STEP 2509 H=2509:00 +CPU= 19.611
  12:48:09 STEP 2510 H=2510:00 +CPU= 20.853
  12:48:44 STEP 2511 H=2511:00 +CPU= 28.680
  12:49:07 STEP 2512 H=2512:00 +CPU= 19.866
  12:49:30 STEP 2513 H=2513:00 +CPU= 19.383
  12:50:04 STEP 2514 H=2514:00 +CPU= 29.213
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:50:44 STEP 2515 H=2515:00 +CPU= 21.525
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:51:36 STEP 2516 H=2516:00 +CPU= 22.669
  12:52:17 STEP 2517 H=2517:00 +CPU= 33.965
  12:52:47 STEP 2518 H=2518:00 +CPU= 24.274
  12:53:17 STEP 2519 H=2519:00 +CPU= 24.540
  12:54:07 STEP 2520 H=2520:00 +CPU= 40.785
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_85.zip
Uploading the intermediate file: upload_file_85.zip
  12:54:36 STEP 2521 H=2521:00 +CPU= 23.556
  12:55:06 STEP 2522 H=2522:00 +CPU= 24.240
double free or corruption (!prev)
  12:55:46 STEP 2523 H=2523:00 +CPU= 33.709
  12:56:17 STEP 2524 H=2524:00 +CPU= 25.138
  12:56:46 STEP 2525 H=2525:00 +CPU= 23.282
  12:57:27 STEP 2526 H=2526:00 +CPU= 34.670
  12:57:55 STEP 2527 H=2527:00 +CPU= 23.197
  12:58:24 STEP 2528 H=2528:00 +CPU= 23.826
  12:59:04 STEP 2529 H=2529:00 +CPU= 33.513
  12:59:31 STEP 2530 H=2530:00 +CPU= 23.122
  12:59:55 STEP 2531 H=2531:00 +CPU= 20.674
  13:00:37 STEP 2532 H=2532:00 +CPU= 35.248
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002532
  13:01:04 STEP 2533 H=2533:00 +CPU= 21.987
  13:01:30 STEP 2534 H=2534:00 +CPU= 22.082
  13:02:02 STEP 2535 H=2535:00 +CPU= 28.876
  13:02:28 STEP 2536 H=2536:00 +CPU= 22.053
  13:02:52 STEP 2537 H=2537:00 +CPU= 20.768
  13:03:27 STEP 2538 H=2538:00 +CPU= 30.760
  13:03:52 STEP 2539 H=2539:00 +CPU= 20.924
  13:04:19 STEP 2540 H=2540:00 +CPU= 22.769
  13:04:58 STEP 2541 H=2541:00 +CPU= 32.831
  13:05:25 STEP 2542 H=2542:00 +CPU= 22.316
  13:05:53 STEP 2543 H=2543:00 +CPU= 22.725
  13:06:40 STEP 2544 H=2544:00 +CPU= 38.778
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_86.zip
Uploading the intermediate file: upload_file_86.zip
  13:07:06 STEP 2545 H=2545:00 +CPU= 21.658
  13:07:33 STEP 2546 H=2546:00 +CPU= 22.646
  13:08:14 STEP 2547 H=2547:00 +CPU= 33.379
  13:08:41 STEP 2548 H=2548:00 +CPU= 22.051
  13:09:07 STEP 2549 H=2549:00 +CPU= 22.017
  13:09:47 STEP 2550 H=2550:00 +CPU= 33.533
  13:10:10 STEP 2551 H=2551:00 +CPU= 19.923
  13:10:35 STEP 2552 H=2552:00 +CPU= 21.196
  13:11:12 STEP 2553 H=2553:00 +CPU= 31.257
  13:11:36 STEP 2554 H=2554:00 +CPU= 20.983
  13:12:01 STEP 2555 H=2555:00 +CPU= 21.735
  13:12:46 STEP 2556 H=2556:00 +CPU= 37.452
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002556
  13:13:09 STEP 2557 H=2557:00 +CPU= 20.070
  13:13:35 STEP 2558 H=2558:00 +CPU= 22.586
  13:14:12 STEP 2559 H=2559:00 +CPU= 32.463
  13:14:37 STEP 2560 H=2560:00 +CPU= 21.324
  13:15:04 STEP 2561 H=2561:00 +CPU= 22.919
  13:15:44 STEP 2562 H=2562:00 +CPU= 35.331
  13:16:08 STEP 2563 H=2563:00 +CPU= 20.913
  13:16:37 STEP 2564 H=2564:00 +CPU= 24.768
  13:17:17 STEP 2565 H=2565:00 +CPU= 34.443
  13:17:43 STEP 2566 H=2566:00 +CPU= 22.108
  13:18:09 STEP 2567 H=2567:00 +CPU= 23.270
  13:18:59 STEP 2568 H=2568:00 +CPU= 41.444
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_87.zip
Uploading the intermediate file: upload_file_87.zip
  13:19:23 STEP 2569 H=2569:00 +CPU= 20.795
  13:19:53 STEP 2570 H=2570:00 +CPU= 24.315
  13:20:37 STEP 2571 H=2571:00 +CPU= 36.916
  13:21:05 STEP 2572 H=2572:00 +CPU= 22.894
  13:21:34 STEP 2573 H=2573:00 +CPU= 23.868
  13:22:16 STEP 2574 H=2574:00 +CPU= 35.297
  13:22:41 STEP 2575 H=2575:00 +CPU= 22.551
  13:23:12 STEP 2576 H=2576:00 +CPU= 25.243
  13:23:56 STEP 2577 H=2577:00 +CPU= 37.360
  13:24:20 STEP 2578 H=2578:00 +CPU= 22.157
  13:24:51 STEP 2579 H=2579:00 +CPU= 25.203
  13:25:40 STEP 2580 H=2580:00 +CPU= 42.031
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002580
  13:26:06 STEP 2581 H=2581:00 +CPU= 22.941
  13:26:39 STEP 2582 H=2582:00 +CPU= 26.877
  13:27:28 STEP 2583 H=2583:00 +CPU= 39.474
  13:28:02 STEP 2584 H=2584:00 +CPU= 26.526
  13:28:35 STEP 2585 H=2585:00 +CPU= 26.847
  13:29:27 STEP 2586 H=2586:00 +CPU= 42.009
  13:30:00 STEP 2587 H=2587:00 +CPU= 26.214
  13:30:34 STEP 2588 H=2588:00 +CPU= 27.987
  13:31:23 STEP 2589 H=2589:00 +CPU= 40.410
  13:31:56 STEP 2590 H=2590:00 +CPU= 27.138
  13:32:31 STEP 2591 H=2591:00 +CPU= 28.607
  13:33:28 STEP 2592 H=2592:00 +CPU= 47.011
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_88.zip
Uploading the intermediate file: upload_file_88.zip
  13:34:01 STEP 2593 H=2593:00 +CPU= 26.714
  13:34:35 STEP 2594 H=2594:00 +CPU= 27.549
  13:35:26 STEP 2595 H=2595:00 +CPU= 41.414
  13:35:58 STEP 2596 H=2596:00 +CPU= 26.961
  13:36:31 STEP 2597 H=2597:00 +CPU= 27.051
  13:37:22 STEP 2598 H=2598:00 +CPU= 41.966
  13:37:53 STEP 2599 H=2599:00 +CPU= 25.926
  13:38:23 STEP 2600 H=2600:00 +CPU= 26.075
  13:39:14 STEP 2601 H=2601:00 +CPU= 40.629
  13:39:47 STEP 2602 H=2602:00 +CPU= 25.968
  13:40:20 STEP 2603 H=2603:00 +CPU= 26.514
  13:41:16 STEP 2604 H=2604:00 +CPU= 45.247
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002604
  13:41:48 STEP 2605 H=2605:00 +CPU= 26.548
  13:42:21 STEP 2606 H=2606:00 +CPU= 27.141
  13:43:12 STEP 2607 H=2607:00 +CPU= 42.134
  13:43:46 STEP 2608 H=2608:00 +CPU= 27.657
  13:44:18 STEP 2609 H=2609:00 +CPU= 26.401
  13:45:09 STEP 2610 H=2610:00 +CPU= 42.516
  13:45:42 STEP 2611 H=2611:00 +CPU= 26.816
  13:46:14 STEP 2612 H=2612:00 +CPU= 26.513
  13:47:03 STEP 2613 H=2613:00 +CPU= 40.768
  13:47:36 STEP 2614 H=2614:00 +CPU= 27.494
  13:48:09 STEP 2615 H=2615:00 +CPU= 27.525
  13:49:06 STEP 2616 H=2616:00 +CPU= 48.008
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_89.zip
Uploading the intermediate file: upload_file_89.zip
  13:49:39 STEP 2617 H=2617:00 +CPU= 27.436
  13:50:11 STEP 2618 H=2618:00 +CPU= 26.867
  13:51:01 STEP 2619 H=2619:00 +CPU= 40.210
  13:51:34 STEP 2620 H=2620:00 +CPU= 27.784
  13:52:06 STEP 2621 H=2621:00 +CPU= 26.380
  13:52:56 STEP 2622 H=2622:00 +CPU= 41.004
  13:53:28 STEP 2623 H=2623:00 +CPU= 27.415
  13:53:59 STEP 2624 H=2624:00 +CPU= 26.325
  13:54:47 STEP 2625 H=2625:00 +CPU= 40.573
  13:55:21 STEP 2626 H=2626:00 +CPU= 27.757
  13:55:53 STEP 2627 H=2627:00 +CPU= 26.865
  13:56:46 STEP 2628 H=2628:00 +CPU= 44.809
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002628
  13:57:19 STEP 2629 H=2629:00 +CPU= 27.189
  13:57:51 STEP 2630 H=2630:00 +CPU= 27.257
  13:58:37 STEP 2631 H=2631:00 +CPU= 38.598
  13:59:06 STEP 2632 H=2632:00 +CPU= 25.520
  13:59:34 STEP 2633 H=2633:00 +CPU= 23.687
  14:00:17 STEP 2634 H=2634:00 +CPU= 37.602
  14:00:46 STEP 2635 H=2635:00 +CPU= 24.457
  14:01:16 STEP 2636 H=2636:00 +CPU= 24.343
  14:01:55 STEP 2637 H=2637:00 +CPU= 35.081
  14:02:25 STEP 2638 H=2638:00 +CPU= 25.338
  14:02:54 STEP 2639 H=2639:00 +CPU= 24.607
  14:03:45 STEP 2640 H=2640:00 +CPU= 42.149
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_90.zip
Uploading the intermediate file: upload_file_90.zip
  14:04:14 STEP 2641 H=2641:00 +CPU= 24.438
  14:04:44 STEP 2642 H=2642:00 +CPU= 24.392
  14:05:28 STEP 2643 H=2643:00 +CPU= 36.206
  14:05:59 STEP 2644 H=2644:00 +CPU= 25.231
  14:06:26 STEP 2645 H=2645:00 +CPU= 23.560
  14:07:09 STEP 2646 H=2646:00 +CPU= 36.520
  14:07:38 STEP 2647 H=2647:00 +CPU= 24.247
  14:08:07 STEP 2648 H=2648:00 +CPU= 24.241
  14:08:50 STEP 2649 H=2649:00 +CPU= 36.283
  14:09:19 STEP 2650 H=2650:00 +CPU= 24.826
  14:09:47 STEP 2651 H=2651:00 +CPU= 23.781
  14:10:35 STEP 2652 H=2652:00 +CPU= 40.280
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002652
  14:11:04 STEP 2653 H=2653:00 +CPU= 24.252
  14:11:33 STEP 2654 H=2654:00 +CPU= 24.360
  14:12:16 STEP 2655 H=2655:00 +CPU= 36.362
  14:12:46 STEP 2656 H=2656:00 +CPU= 24.703
  14:13:14 STEP 2657 H=2657:00 +CPU= 24.004
  14:13:58 STEP 2658 H=2658:00 +CPU= 37.372
  14:14:26 STEP 2659 H=2659:00 +CPU= 23.875
  14:14:55 STEP 2660 H=2660:00 +CPU= 24.659
  14:15:35 STEP 2661 H=2661:00 +CPU= 36.374
  14:16:02 STEP 2662 H=2662:00 +CPU= 24.676
  14:16:30 STEP 2663 H=2663:00 +CPU= 24.640
  14:17:17 STEP 2664 H=2664:00 +CPU= 41.854
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_91.zip
Uploading the intermediate file: upload_file_91.zip
  14:17:43 STEP 2665 H=2665:00 +CPU= 24.064
  14:18:12 STEP 2666 H=2666:00 +CPU= 24.537
  14:18:52 STEP 2667 H=2667:00 +CPU= 36.167
  14:19:21 STEP 2668 H=2668:00 +CPU= 24.662
  14:19:48 STEP 2669 H=2669:00 +CPU= 23.540
  14:20:25 STEP 2670 H=2670:00 +CPU= 35.688
  14:20:50 STEP 2671 H=2671:00 +CPU= 23.658
  14:21:15 STEP 2672 H=2672:00 +CPU= 24.029
  14:21:53 STEP 2673 H=2673:00 +CPU= 35.593
  14:22:18 STEP 2674 H=2674:00 +CPU= 23.831
  14:22:43 STEP 2675 H=2675:00 +CPU= 24.004
  14:23:30 STEP 2676 H=2676:00 +CPU= 40.377
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002676
  14:23:58 STEP 2677 H=2677:00 +CPU= 23.962
  14:24:26 STEP 2678 H=2678:00 +CPU= 24.949
  14:25:07 STEP 2679 H=2679:00 +CPU= 36.495
  14:25:36 STEP 2680 H=2680:00 +CPU= 24.664
  14:26:03 STEP 2681 H=2681:00 +CPU= 23.604
  14:26:45 STEP 2682 H=2682:00 +CPU= 36.981
  14:27:13 STEP 2683 H=2683:00 +CPU= 23.642
  14:27:42 STEP 2684 H=2684:00 +CPU= 24.500
  14:28:24 STEP 2685 H=2685:00 +CPU= 36.327
  14:28:54 STEP 2686 H=2686:00 +CPU= 24.832
  14:29:25 STEP 2687 H=2687:00 +CPU= 25.210
  14:30:14 STEP 2688 H=2688:00 +CPU= 41.675
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_92.zip
Uploading the intermediate file: upload_file_92.zip
  14:30:43 STEP 2689 H=2689:00 +CPU= 24.011
  14:31:13 STEP 2690 H=2690:00 +CPU= 24.431
  14:31:58 STEP 2691 H=2691:00 +CPU= 36.833
  14:32:29 STEP 2692 H=2692:00 +CPU= 24.381
  14:32:57 STEP 2693 H=2693:00 +CPU= 23.340
  14:33:42 STEP 2694 H=2694:00 +CPU= 37.326
  14:34:10 STEP 2695 H=2695:00 +CPU= 23.701
  14:34:40 STEP 2696 H=2696:00 +CPU= 24.226
  14:35:25 STEP 2697 H=2697:00 +CPU= 37.012
  14:35:55 STEP 2698 H=2698:00 +CPU= 24.629
  14:36:23 STEP 2699 H=2699:00 +CPU= 23.696
  14:37:12 STEP 2700 H=2700:00 +CPU= 40.208
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002700
  14:37:42 STEP 2701 H=2701:00 +CPU= 24.262
  14:38:12 STEP 2702 H=2702:00 +CPU= 24.535
  14:38:50 STEP 2703 H=2703:00 +CPU= 35.685
  14:39:15 STEP 2704 H=2704:00 +CPU= 23.370
  14:39:40 STEP 2705 H=2705:00 +CPU= 23.172
  14:40:21 STEP 2706 H=2706:00 +CPU= 36.726
  14:40:50 STEP 2707 H=2707:00 +CPU= 24.075
  14:41:18 STEP 2708 H=2708:00 +CPU= 24.294
  14:42:02 STEP 2709 H=2709:00 +CPU= 36.223
  14:42:35 STEP 2710 H=2710:00 +CPU= 26.310
  14:43:10 STEP 2711 H=2711:00 +CPU= 27.424
  14:44:05 STEP 2712 H=2712:00 +CPU= 44.413
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_93.zip
Uploading the intermediate file: upload_file_93.zip
Uploading trickle at timestep: 9763200
  14:44:38 STEP 2713 H=2713:00 +CPU= 25.912
  14:45:12 STEP 2714 H=2714:00 +CPU= 26.737
  14:46:00 STEP 2715 H=2715:00 +CPU= 38.065
  14:46:34 STEP 2716 H=2716:00 +CPU= 26.492
  14:47:08 STEP 2717 H=2717:00 +CPU= 25.865
  14:47:56 STEP 2718 H=2718:00 +CPU= 38.692
  14:48:25 STEP 2719 H=2719:00 +CPU= 23.902
  14:48:55 STEP 2720 H=2720:00 +CPU= 24.293
  14:49:39 STEP 2721 H=2721:00 +CPU= 36.168
  14:50:09 STEP 2722 H=2722:00 +CPU= 24.445
  14:50:38 STEP 2723 H=2723:00 +CPU= 23.924
  14:51:28 STEP 2724 H=2724:00 +CPU= 40.293
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002724
  14:51:57 STEP 2725 H=2725:00 +CPU= 24.089
  14:52:27 STEP 2726 H=2726:00 +CPU= 24.700
  14:53:11 STEP 2727 H=2727:00 +CPU= 36.440
  14:53:41 STEP 2728 H=2728:00 +CPU= 23.929
  14:54:06 STEP 2729 H=2729:00 +CPU= 19.339
  14:54:22 STEP 2730 H=2730:00 +CPU= 15.365
  14:54:31 STEP 2731 H=2731:00 +CPU=  9.326
  14:54:41 STEP 2732 H=2732:00 +CPU=  9.966
  14:54:58 STEP 2733 H=2733:00 +CPU= 16.865
  14:55:10 STEP 2734 H=2734:00 +CPU= 11.660
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0619
(argv4) batchid: 958
(argv5) wuid: 12175263
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/25
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/25/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/25/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_12e847c90efe210604a2d1f0e5af7906 to: /var/lib/boinc/slots/25/oifs_43r3_ps_0619_1989050100_123_958_12175263.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/25/oifs_43r3_ps_0619_1989050100_123_958_12175263.zip
ic_ancil_file: ic_ancil_12175263
ifsdata_file: ifsdata_12175263
climate_data_file: clim_data_12175263
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f585589a0ba9d745e24c020ccc1664fd to: /var/lib/boinc/slots/25/ic_ancil_12175263.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/25/ic_ancil_12175263.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f7c2c5939852b7f30e230b479ceb0a43 to: /var/lib/boinc/slots/25/ifsdata/ifsdata_12175263.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/25/ifsdata/ifsdata_12175263.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/25/159l_2/clim_data_12175263.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/25/159l_2/clim_data_12175263.zip
Checking for progress XML file: /var/lib/boinc/slots/25/progress_file_12175263.xml
Opened progress file ok : /var/lib/boinc/slots/25/progress_file_12175263.xml
-- Model is restarting --
Adjusting last_iter, 2734, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/25/progress_file_12175263.xml
last_cpu_time: 64653
upload_file_number: 94
last_iter: 2711
last_upload: 9763200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959
The child process has been launched with process id: 3786
Executing the command: /var/lib/boinc/slots/25/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tweenis:1:1:3786:3786] [20221223:183811:1671820691:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:38:59 STEP 2712 H=2712:00 +CPU= 39.800
  18:39:27 STEP 2713 H=2713:00 +CPU= 25.023
  18:39:53 STEP 2714 H=2714:00 +CPU= 24.173
  18:40:27 STEP 2715 H=2715:00 +CPU= 32.861
  18:40:53 STEP 2716 H=2716:00 +CPU= 24.123
  18:41:18 STEP 2717 H=2717:00 +CPU= 23.308
  18:41:55 STEP 2718 H=2718:00 +CPU= 34.333
  18:42:20 STEP 2719 H=2719:00 +CPU= 23.937
  18:42:46 STEP 2720 H=2720:00 +CPU= 24.514
  18:43:21 STEP 2721 H=2721:00 +CPU= 33.555
  18:43:47 STEP 2722 H=2722:00 +CPU= 24.364
  18:44:12 STEP 2723 H=2723:00 +CPU= 23.454
  18:44:52 STEP 2724 H=2724:00 +CPU= 37.595
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002724
  18:45:17 STEP 2725 H=2725:00 +CPU= 23.868
  18:45:43 STEP 2726 H=2726:00 +CPU= 24.345
  18:46:18 STEP 2727 H=2727:00 +CPU= 34.014
  18:46:45 STEP 2728 H=2728:00 +CPU= 24.999
  18:47:10 STEP 2729 H=2729:00 +CPU= 24.084
  18:47:47 STEP 2730 H=2730:00 +CPU= 34.728
  18:48:12 STEP 2731 H=2731:00 +CPU= 23.553
  18:48:37 STEP 2732 H=2732:00 +CPU= 24.171
  18:49:13 STEP 2733 H=2733:00 +CPU= 34.080
  18:49:39 STEP 2734 H=2734:00 +CPU= 24.910
  18:50:05 STEP 2735 H=2735:00 +CPU= 24.467
  18:50:49 STEP 2736 H=2736:00 +CPU= 40.695
Moving to projects directory: /var/lib/boinc/slots/25/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc/slots/25/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc/slots/25/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12175263/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0619_1989050100_123_958_12175263_0_r1149959_94.zip
Uploading the intermediate file: upload_file_94.zip
  18:51:15 STEP 2737 H=2737:00 +CPU= 23.699
  18:51:40 STEP 2738 H=2738:00 +CPU= 24.079
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:48:41 1531398 22259952 oifs_43r3_ps_0619_1989050100_123_958_12175263_0 8,208,000 50,663 0.0062
09 May 2023 15:47:46 1531398 22259952 oifs_43r3_ps_0619_1989050100_123_958_12175263_0 6,048,000 31,717 0.0052
09 May 2023 15:47:35 1531398 22259952 oifs_43r3_ps_0619_1989050100_123_958_12175263_0 5,184,000 25,559 0.0049
09 May 2023 15:47:24 1531398 22259952 oifs_43r3_ps_0619_1989050100_123_958_12175263_0 4,320,000 19,402 0.0045
09 May 2023 15:47:13 1531398 22259952 oifs_43r3_ps_0619_1989050100_123_958_12175263_0 3,456,000 13,250 0.0038
09 May 2023 15:47:07 1531398 22259952 oifs_43r3_ps_0619_1989050100_123_958_12175263_0 2,592,000 7,046 0.0027
09 May 2023 15:46:40 1531398 22259952 oifs_43r3_ps_0619_1989050100_123_958_12175263_0 1,728,000 12,262 0.0071
09 May 2023 15:46:29 1531398 22259952 oifs_43r3_ps_0619_1989050100_123_958_12175263_0 864,000 6,740 0.0078


©2024 climateprediction.net