climateprediction.net home page
Task 22270226

Task 22270226

Name oifs_43r3_ps_0304_1999050100_123_968_12184948_0
Workunit 12184948
Created 22 Dec 2022, 18:28:27 UTC
Sent 15 Jan 2023, 20:23:13 UTC
Report deadline 14 Feb 2023, 20:23:13 UTC
Received 18 Jan 2023, 18:29:12 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1478188
Run time 19 hours 9 min 10 sec
CPU time 18 hours 32 min 2 sec
Validate state Invalid
Credit 9,072.00
Device peak FLOPS 4.20 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,402.76 MB
Peak swap size 4,978.08 MB
Peak disk usage 3,646.07 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
ctory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001488
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001464
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001464
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001464
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001476
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001476
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001476
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_61.zip
Uploading the intermediate file: upload_file_61.zip
  20:58:37 STEP 1489 H=1489:00 +CPU= 28.647
Uploading trickle at timestep: 5356800
  20:59:09 STEP 1490 H=1490:00 +CPU= 31.543
  20:59:58 STEP 1491 H=1491:00 +CPU= 47.868
  21:00:30 STEP 1492 H=1492:00 +CPU= 32.222
  21:01:01 STEP 1493 H=1493:00 +CPU= 30.735
  21:01:52 STEP 1494 H=1494:00 +CPU= 50.288
  21:02:23 STEP 1495 H=1495:00 +CPU= 30.839
  21:02:54 STEP 1496 H=1496:00 +CPU= 30.747
  21:03:42 STEP 1497 H=1497:00 +CPU= 47.468
  21:04:13 STEP 1498 H=1498:00 +CPU= 31.235
  21:04:44 STEP 1499 H=1499:00 +CPU= 29.847
  21:05:37 STEP 1500 H=1500:00 +CPU= 53.040
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001500
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001500
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001500
  21:06:09 STEP 1501 H=1501:00 +CPU= 31.263
  21:06:39 STEP 1502 H=1502:00 +CPU= 29.522
  21:07:27 STEP 1503 H=1503:00 +CPU= 47.465
  21:07:58 STEP 1504 H=1504:00 +CPU= 31.207
  21:08:28 STEP 1505 H=1505:00 +CPU= 29.800
  21:09:19 STEP 1506 H=1506:00 +CPU= 50.043
  21:09:49 STEP 1507 H=1507:00 +CPU= 29.649
  21:10:21 STEP 1508 H=1508:00 +CPU= 31.633
  21:11:10 STEP 1509 H=1509:00 +CPU= 49.007
  21:11:42 STEP 1510 H=1510:00 +CPU= 31.308
  21:12:13 STEP 1511 H=1511:00 +CPU= 31.046
  21:13:09 STEP 1512 H=1512:00 +CPU= 55.970
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001512
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001512
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001512
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001488
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001488
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001488
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001500
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001500
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001500
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_62.zip
Uploading the intermediate file: upload_file_62.zip
  21:13:40 STEP 1513 H=1513:00 +CPU= 30.145
  21:14:11 STEP 1514 H=1514:00 +CPU= 30.630
  21:15:00 STEP 1515 H=1515:00 +CPU= 48.975
  21:15:31 STEP 1516 H=1516:00 +CPU= 30.241
  21:16:01 STEP 1517 H=1517:00 +CPU= 30.091
  21:16:50 STEP 1518 H=1518:00 +CPU= 48.640
  21:17:20 STEP 1519 H=1519:00 +CPU= 29.094
  21:17:51 STEP 1520 H=1520:00 +CPU= 31.327
  21:18:40 STEP 1521 H=1521:00 +CPU= 48.399
  21:19:11 STEP 1522 H=1522:00 +CPU= 31.037
  21:19:42 STEP 1523 H=1523:00 +CPU= 29.962
  21:20:36 STEP 1524 H=1524:00 +CPU= 53.634
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001524
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001524
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001524
  21:21:06 STEP 1525 H=1525:00 +CPU= 30.311
  21:21:37 STEP 1526 H=1526:00 +CPU= 30.476
  21:22:26 STEP 1527 H=1527:00 +CPU= 48.350
  21:22:57 STEP 1528 H=1528:00 +CPU= 30.746
  21:23:26 STEP 1529 H=1529:00 +CPU= 28.937
  21:24:16 STEP 1530 H=1530:00 +CPU= 48.997
  21:24:46 STEP 1531 H=1531:00 +CPU= 29.862
  21:25:17 STEP 1532 H=1532:00 +CPU= 31.073
  21:26:05 STEP 1533 H=1533:00 +CPU= 47.720
  21:26:36 STEP 1534 H=1534:00 +CPU= 30.192
  21:27:07 STEP 1535 H=1535:00 +CPU= 30.762
  21:28:04 STEP 1536 H=1536:00 +CPU= 56.258
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001536
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001536
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001536
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001512
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001512
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001512
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001524
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001524
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001524
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_63.zip
Uploading the intermediate file: upload_file_63.zip
  21:28:35 STEP 1537 H=1537:00 +CPU= 30.719
  21:29:06 STEP 1538 H=1538:00 +CPU= 30.929
  21:29:55 STEP 1539 H=1539:00 +CPU= 48.532
  21:30:27 STEP 1540 H=1540:00 +CPU= 31.648
  21:30:57 STEP 1541 H=1541:00 +CPU= 29.896
  21:31:48 STEP 1542 H=1542:00 +CPU= 50.403
  21:32:19 STEP 1543 H=1543:00 +CPU= 30.500
  21:32:51 STEP 1544 H=1544:00 +CPU= 31.381
  21:33:39 STEP 1545 H=1545:00 +CPU= 47.895
  21:34:11 STEP 1546 H=1546:00 +CPU= 31.135
  21:34:41 STEP 1547 H=1547:00 +CPU= 29.881
  21:35:35 STEP 1548 H=1548:00 +CPU= 53.327
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001548
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001548
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001548
  21:36:05 STEP 1549 H=1549:00 +CPU= 30.348
  21:36:37 STEP 1550 H=1550:00 +CPU= 31.729
  21:37:25 STEP 1551 H=1551:00 +CPU= 47.162
  21:37:57 STEP 1552 H=1552:00 +CPU= 31.726
  21:38:27 STEP 1553 H=1553:00 +CPU= 29.681
  21:39:17 STEP 1554 H=1554:00 +CPU= 50.040
  21:39:47 STEP 1555 H=1555:00 +CPU= 29.636
  21:40:19 STEP 1556 H=1556:00 +CPU= 31.521
  21:41:08 STEP 1557 H=1557:00 +CPU= 48.750
  21:41:39 STEP 1558 H=1558:00 +CPU= 31.022
  21:42:11 STEP 1559 H=1559:00 +CPU= 31.319
  21:43:09 STEP 1560 H=1560:00 +CPU= 57.096
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001560
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001560
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001560
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001536
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001536
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001536
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001548
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001548
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001548
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_64.zip
Uploading the intermediate file: upload_file_64.zip
  21:43:39 STEP 1561 H=1561:00 +CPU= 29.665
  21:44:10 STEP 1562 H=1562:00 +CPU= 31.097
  21:44:59 STEP 1563 H=1563:00 +CPU= 48.190
  21:45:30 STEP 1564 H=1564:00 +CPU= 30.444
  21:45:59 STEP 1565 H=1565:00 +CPU= 29.466
  21:46:49 STEP 1566 H=1566:00 +CPU= 49.152
  21:47:19 STEP 1567 H=1567:00 +CPU= 29.784
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.587] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4412MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.620] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.620] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.620] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.620] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.620] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4412 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.620] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                CLOUD_LAYER 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223917:1673995157:41304.621] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 CLOUDSC 
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223918:1673995158:41305.628] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:50498:50498] [20230117:223918:1673995158:41305.628] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001021F36  cloudsc_                 1068  cloudsc.F90
oifs_43r3_model.e  0000000001014D57  cloud_layer_              131  cloud_layer.F90
oifs_43r3_model.e  0000000000F10C1B  callpar_                 1011  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0304
(argv4) batchid: 968
(argv5) wuid: 12184948
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /data/boinc/boinc-client/slots/23
Project directory is: /data/boinc/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948
..mkdir for temp folder for results failed
Copying: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /data/boinc/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /data/boinc/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_363f329e3f9995b0ad4e29c296b93b38 to: /data/boinc/boinc-client/slots/23/oifs_43r3_ps_0304_1999050100_123_968_12184948.zip
Unzipping the namelist zip file: /data/boinc/boinc-client/slots/23/oifs_43r3_ps_0304_1999050100_123_968_12184948.zip
ic_ancil_file: ic_ancil_12184948
ifsdata_file: ifsdata_12184948
climate_data_file: clim_data_12184948
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_20504c40489c6109352abe3407c4fae8 to: /data/boinc/boinc-client/slots/23/ic_ancil_12184948.zip
Unzipping the IC ancils zip file: /data/boinc/boinc-client/slots/23/ic_ancil_12184948.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_7fa72421af36ba59b198767f22f7816e to: /data/boinc/boinc-client/slots/23/ifsdata/ifsdata_12184948.zip
Unzipping the ifsdata_zip file: /data/boinc/boinc-client/slots/23/ifsdata/ifsdata_12184948.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /data/boinc/boinc-client/slots/23/159l_2/clim_data_12184948.zip
Unzipping the climate data zip file: /data/boinc/boinc-client/slots/23/159l_2/clim_data_12184948.zip
Checking for progress XML file: /data/boinc/boinc-client/slots/23/progress_file_12184948.xml
Opened progress file ok : /data/boinc/boinc-client/slots/23/progress_file_12184948.xml
-- Model is restarting --
Adjusting last_iter, 1568, to previous model restart step.
Creating progress file: /data/boinc/boinc-client/slots/23/progress_file_12184948.xml
last_cpu_time: 53394
upload_file_number: 65
last_iter: 1559
last_upload: 5616000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995
The child process has been launched with process id: 13633
Executing the command: /data/boinc/boinc-client/slots/23/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:victor-Precision-7920-Tower:1:1:13633:13633] [20230118:085734:1674032254:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:08:29 STEP 1560 H=1560:00 +CPU= 57.456
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:36:23 STEP 1561 H=1561:00 +CPU= 30.703
  09:36:54 STEP 1562 H=1562:00 +CPU= 30.577
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:39:44 STEP 1563 H=1563:00 +CPU= 48.438
  09:40:15 STEP 1564 H=1564:00 +CPU= 30.400
  09:40:46 STEP 1565 H=1565:00 +CPU= 30.466
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:19:10 STEP 1566 H=1566:00 +CPU= 48.505
  10:19:39 STEP 1567 H=1567:00 +CPU= 29.173
  10:20:11 STEP 1568 H=1568:00 +CPU= 31.666
  10:21:00 STEP 1569 H=1569:00 +CPU= 47.977
  10:21:31 STEP 1570 H=1570:00 +CPU= 30.534
  10:22:02 STEP 1571 H=1571:00 +CPU= 30.606
  10:22:56 STEP 1572 H=1572:00 +CPU= 53.349
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001572
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001572
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001572
  10:23:26 STEP 1573 H=1573:00 +CPU= 30.535
  10:23:57 STEP 1574 H=1574:00 +CPU= 30.551
  10:24:45 STEP 1575 H=1575:00 +CPU= 47.131
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:38:08 STEP 1576 H=1576:00 +CPU= 30.666
  10:38:39 STEP 1577 H=1577:00 +CPU= 30.713
  10:39:29 STEP 1578 H=1578:00 +CPU= 49.324
  10:39:59 STEP 1579 H=1579:00 +CPU= 29.608
  10:40:30 STEP 1580 H=1580:00 +CPU= 30.517
  10:41:19 STEP 1581 H=1581:00 +CPU= 48.528
  10:41:50 STEP 1582 H=1582:00 +CPU= 30.053
  10:42:21 STEP 1583 H=1583:00 +CPU= 30.969
  10:43:20 STEP 1584 H=1584:00 +CPU= 57.353
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001584
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001584
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001584
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001560
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001560
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001560
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001572
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001572
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001572
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_65.zip
Uploading the intermediate file: upload_file_65.zip
  10:43:43 STEP 1585 H=1585:00 +CPU= 22.620
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:04:51 STEP 1586 H=1586:00 +CPU= 30.931
  12:05:40 STEP 1587 H=1587:00 +CPU= 48.143
  12:06:11 STEP 1588 H=1588:00 +CPU= 30.821
  12:06:43 STEP 1589 H=1589:00 +CPU= 31.023
  12:07:32 STEP 1590 H=1590:00 +CPU= 48.344
  12:08:02 STEP 1591 H=1591:00 +CPU= 29.968
  12:08:33 STEP 1592 H=1592:00 +CPU= 31.215
  12:09:22 STEP 1593 H=1593:00 +CPU= 48.128
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:22:42 STEP 1594 H=1594:00 +CPU= 30.564
  12:23:13 STEP 1595 H=1595:00 +CPU= 31.066
  12:24:08 STEP 1596 H=1596:00 +CPU= 53.195
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001596
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001596
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001596
  12:24:38 STEP 1597 H=1597:00 +CPU= 30.400
  12:25:10 STEP 1598 H=1598:00 +CPU= 31.161
  12:25:59 STEP 1599 H=1599:00 +CPU= 48.014
  12:26:31 STEP 1600 H=1600:00 +CPU= 31.929
  12:27:03 STEP 1601 H=1601:00 +CPU= 30.536
  12:27:53 STEP 1602 H=1602:00 +CPU= 49.885
  12:28:25 STEP 1603 H=1603:00 +CPU= 30.856
  12:28:57 STEP 1604 H=1604:00 +CPU= 32.127
  12:29:47 STEP 1605 H=1605:00 +CPU= 48.634
  12:30:20 STEP 1606 H=1606:00 +CPU= 32.259
  12:30:50 STEP 1607 H=1607:00 +CPU= 29.812
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:43:32 STEP 1608 H=1608:00 +CPU= 56.319
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001608
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001608
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001608
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001584
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001584
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001584
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001596
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001596
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001596
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_66.zip
Uploading the intermediate file: upload_file_66.zip
  12:44:03 STEP 1609 H=1609:00 +CPU= 30.678
  12:44:36 STEP 1610 H=1610:00 +CPU= 31.762
  12:45:24 STEP 1611 H=1611:00 +CPU= 47.860
  12:45:56 STEP 1612 H=1612:00 +CPU= 31.100
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:39:34 STEP 1613 H=1613:00 +CPU= 29.469
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:58:06 STEP 1614 H=1614:00 +CPU= 49.902
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:02:40 STEP 1615 H=1615:00 +CPU= 31.524
  14:03:13 STEP 1616 H=1616:00 +CPU= 31.932
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:17:29 STEP 1617 H=1617:00 +CPU= 47.144
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:20:00 STEP 1618 H=1618:00 +CPU= 30.762
  14:20:31 STEP 1619 H=1619:00 +CPU= 30.221
  14:21:25 STEP 1620 H=1620:00 +CPU= 52.729
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001620
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001620
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001620
  14:21:54 STEP 1621 H=1621:00 +CPU= 29.537
  14:22:26 STEP 1622 H=1622:00 +CPU= 31.022
  14:23:14 STEP 1623 H=1623:00 +CPU= 47.300
  14:23:46 STEP 1624 H=1624:00 +CPU= 31.241
  14:24:16 STEP 1625 H=1625:00 +CPU= 29.863
  14:25:06 STEP 1626 H=1626:00 +CPU= 48.934
  14:25:36 STEP 1627 H=1627:00 +CPU= 29.674
  14:26:07 STEP 1628 H=1628:00 +CPU= 31.395
  14:26:56 STEP 1629 H=1629:00 +CPU= 47.636
  14:27:26 STEP 1630 H=1630:00 +CPU= 30.259
  14:27:58 STEP 1631 H=1631:00 +CPU= 31.047
  14:28:57 STEP 1632 H=1632:00 +CPU= 57.927
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001632
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001632
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001632
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001608
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001608
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001608
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001620
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001620
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001620
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_67.zip
Uploading the intermediate file: upload_file_67.zip
  14:29:27 STEP 1633 H=1633:00 +CPU= 29.619
  14:29:58 STEP 1634 H=1634:00 +CPU= 30.398
  14:30:45 STEP 1635 H=1635:00 +CPU= 47.043
  14:31:17 STEP 1636 H=1636:00 +CPU= 31.044
  14:31:47 STEP 1637 H=1637:00 +CPU= 29.732
  14:32:37 STEP 1638 H=1638:00 +CPU= 48.815
  14:33:06 STEP 1639 H=1639:00 +CPU= 29.432
  14:33:38 STEP 1640 H=1640:00 +CPU= 31.000
  14:34:26 STEP 1641 H=1641:00 +CPU= 47.372
  14:34:57 STEP 1642 H=1642:00 +CPU= 30.969
  14:35:28 STEP 1643 H=1643:00 +CPU= 29.888
  14:36:21 STEP 1644 H=1644:00 +CPU= 52.546
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001644
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001644
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001644
  14:36:51 STEP 1645 H=1645:00 +CPU= 29.868
  14:37:23 STEP 1646 H=1646:00 +CPU= 31.158
  14:38:10 STEP 1647 H=1647:00 +CPU= 47.207
  14:38:42 STEP 1648 H=1648:00 +CPU= 30.819
  14:39:11 STEP 1649 H=1649:00 +CPU= 29.462
  14:40:00 STEP 1650 H=1650:00 +CPU= 48.151
  14:40:31 STEP 1651 H=1651:00 +CPU= 30.400
  14:41:02 STEP 1652 H=1652:00 +CPU= 30.897
  14:41:51 STEP 1653 H=1653:00 +CPU= 47.709
  14:42:21 STEP 1654 H=1654:00 +CPU= 30.463
  14:42:52 STEP 1655 H=1655:00 +CPU= 30.700
  14:43:50 STEP 1656 H=1656:00 +CPU= 56.393
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001656
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001656
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001656
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001632
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001632
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001632
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001644
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001644
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001644
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_68.zip
Uploading the intermediate file: upload_file_68.zip
  14:44:20 STEP 1657 H=1657:00 +CPU= 29.802
  14:44:50 STEP 1658 H=1658:00 +CPU= 30.062
  14:45:39 STEP 1659 H=1659:00 +CPU= 47.479
  14:46:09 STEP 1660 H=1660:00 +CPU= 30.442
  14:46:39 STEP 1661 H=1661:00 +CPU= 29.468
  14:47:28 STEP 1662 H=1662:00 +CPU= 48.605
  14:47:58 STEP 1663 H=1663:00 +CPU= 29.687
  14:48:30 STEP 1664 H=1664:00 +CPU= 30.967
  14:49:17 STEP 1665 H=1665:00 +CPU= 47.165
  14:49:48 STEP 1666 H=1666:00 +CPU= 30.095
  14:50:18 STEP 1667 H=1667:00 +CPU= 29.399
  14:51:10 STEP 1668 H=1668:00 +CPU= 52.078
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001668
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001668
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001668
  14:51:41 STEP 1669 H=1669:00 +CPU= 30.157
  14:52:12 STEP 1670 H=1670:00 +CPU= 30.637
  14:53:00 STEP 1671 H=1671:00 +CPU= 48.032
  14:53:32 STEP 1672 H=1672:00 +CPU= 31.064
  14:54:02 STEP 1673 H=1673:00 +CPU= 29.408
  14:54:50 STEP 1674 H=1674:00 +CPU= 47.934
  14:55:19 STEP 1675 H=1675:00 +CPU= 28.774
  14:55:51 STEP 1676 H=1676:00 +CPU= 30.855
  14:56:38 STEP 1677 H=1677:00 +CPU= 46.694
  14:57:08 STEP 1678 H=1678:00 +CPU= 30.243
  14:57:39 STEP 1679 H=1679:00 +CPU= 30.652
  14:58:36 STEP 1680 H=1680:00 +CPU= 55.965
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001680
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001680
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001680
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001656
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001656
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001656
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001668
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001668
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001668
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_69.zip
Uploading the intermediate file: upload_file_69.zip
  14:59:06 STEP 1681 H=1681:00 +CPU= 29.836
  14:59:37 STEP 1682 H=1682:00 +CPU= 30.572
  15:00:25 STEP 1683 H=1683:00 +CPU= 46.608
  15:00:56 STEP 1684 H=1684:00 +CPU= 30.942
  15:01:25 STEP 1685 H=1685:00 +CPU= 29.111
  15:02:15 STEP 1686 H=1686:00 +CPU= 48.922
  15:02:45 STEP 1687 H=1687:00 +CPU= 30.233
  15:03:17 STEP 1688 H=1688:00 +CPU= 30.832
  15:04:04 STEP 1689 H=1689:00 +CPU= 46.580
  15:04:35 STEP 1690 H=1690:00 +CPU= 30.658
  15:05:05 STEP 1691 H=1691:00 +CPU= 29.958
  15:05:58 STEP 1692 H=1692:00 +CPU= 52.467
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001692
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001692
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001692
  15:06:27 STEP 1693 H=1693:00 +CPU= 29.116
  15:06:58 STEP 1694 H=1694:00 +CPU= 30.587
  15:07:46 STEP 1695 H=1695:00 +CPU= 47.400
  15:08:16 STEP 1696 H=1696:00 +CPU= 29.706
  15:08:46 STEP 1697 H=1697:00 +CPU= 29.335
  15:09:35 STEP 1698 H=1698:00 +CPU= 48.620
  15:10:05 STEP 1699 H=1699:00 +CPU= 28.918
  15:10:36 STEP 1700 H=1700:00 +CPU= 31.117
  15:11:22 STEP 1701 H=1701:00 +CPU= 45.915
  15:11:53 STEP 1702 H=1702:00 +CPU= 30.020
  15:12:23 STEP 1703 H=1703:00 +CPU= 30.234
  15:13:20 STEP 1704 H=1704:00 +CPU= 55.869
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001704
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001704
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001704
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001680
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001680
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001680
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001692
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001692
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001692
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_70.zip
Uploading the intermediate file: upload_file_70.zip
  15:13:50 STEP 1705 H=1705:00 +CPU= 29.294
  15:14:21 STEP 1706 H=1706:00 +CPU= 30.495
  15:15:08 STEP 1707 H=1707:00 +CPU= 46.854
  15:15:39 STEP 1708 H=1708:00 +CPU= 30.936
  15:16:09 STEP 1709 H=1709:00 +CPU= 29.721
  15:16:58 STEP 1710 H=1710:00 +CPU= 48.211
  15:17:28 STEP 1711 H=1711:00 +CPU= 29.550
  15:17:58 STEP 1712 H=1712:00 +CPU= 30.250
  15:18:46 STEP 1713 H=1713:00 +CPU= 47.270
  15:19:17 STEP 1714 H=1714:00 +CPU= 30.554
  15:19:47 STEP 1715 H=1715:00 +CPU= 29.690
  15:20:40 STEP 1716 H=1716:00 +CPU= 51.949
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001716
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001716
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001716
  15:21:09 STEP 1717 H=1717:00 +CPU= 29.329
  15:21:40 STEP 1718 H=1718:00 +CPU= 30.139
  15:22:28 STEP 1719 H=1719:00 +CPU= 47.237
  15:22:59 STEP 1720 H=1720:00 +CPU= 30.668
  15:23:28 STEP 1721 H=1721:00 +CPU= 29.366
  15:24:17 STEP 1722 H=1722:00 +CPU= 48.112
  15:24:47 STEP 1723 H=1723:00 +CPU= 29.111
  15:25:17 STEP 1724 H=1724:00 +CPU= 29.820
  15:26:03 STEP 1725 H=1725:00 +CPU= 45.456
  15:26:34 STEP 1726 H=1726:00 +CPU= 30.640
  15:27:04 STEP 1727 H=1727:00 +CPU= 30.512
  15:28:02 STEP 1728 H=1728:00 +CPU= 56.703
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001728
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001728
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001728
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001704
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001704
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001704
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001716
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001716
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001716
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_71.zip
Uploading the intermediate file: upload_file_71.zip
  15:28:31 STEP 1729 H=1729:00 +CPU= 29.244
  15:29:03 STEP 1730 H=1730:00 +CPU= 30.988
  15:29:51 STEP 1731 H=1731:00 +CPU= 47.727
  15:30:21 STEP 1732 H=1732:00 +CPU= 30.376
  15:30:52 STEP 1733 H=1733:00 +CPU= 29.903
  15:31:41 STEP 1734 H=1734:00 +CPU= 48.964
  15:32:11 STEP 1735 H=1735:00 +CPU= 29.332
  15:32:43 STEP 1736 H=1736:00 +CPU= 31.468
  15:33:31 STEP 1737 H=1737:00 +CPU= 47.773
  15:34:02 STEP 1738 H=1738:00 +CPU= 30.655
  15:34:32 STEP 1739 H=1739:00 +CPU= 29.701
  15:35:25 STEP 1740 H=1740:00 +CPU= 52.807
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001740
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001740
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001740
  15:35:55 STEP 1741 H=1741:00 +CPU= 29.392
  15:36:26 STEP 1742 H=1742:00 +CPU= 30.622
  15:37:14 STEP 1743 H=1743:00 +CPU= 47.106
  15:37:46 STEP 1744 H=1744:00 +CPU= 31.346
  15:38:15 STEP 1745 H=1745:00 +CPU= 28.799
  15:39:04 STEP 1746 H=1746:00 +CPU= 48.242
  15:39:34 STEP 1747 H=1747:00 +CPU= 29.554
  15:40:05 STEP 1748 H=1748:00 +CPU= 30.598
  15:40:51 STEP 1749 H=1749:00 +CPU= 46.128
  15:41:22 STEP 1750 H=1750:00 +CPU= 30.574
  15:41:53 STEP 1751 H=1751:00 +CPU= 30.188
  15:42:49 STEP 1752 H=1752:00 +CPU= 56.008
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001752
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001752
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001752
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001728
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001728
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001728
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001740
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001740
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001740
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_72.zip
Uploading the intermediate file: upload_file_72.zip
  15:43:19 STEP 1753 H=1753:00 +CPU= 29.070
  15:43:50 STEP 1754 H=1754:00 +CPU= 30.715
  15:44:37 STEP 1755 H=1755:00 +CPU= 46.742
  15:45:08 STEP 1756 H=1756:00 +CPU= 30.823
  15:45:39 STEP 1757 H=1757:00 +CPU= 30.025
  15:46:28 STEP 1758 H=1758:00 +CPU= 48.356
  15:46:57 STEP 1759 H=1759:00 +CPU= 29.571
  15:47:28 STEP 1760 H=1760:00 +CPU= 30.587
  15:48:16 STEP 1761 H=1761:00 +CPU= 46.727
  15:48:46 STEP 1762 H=1762:00 +CPU= 30.250
  15:49:16 STEP 1763 H=1763:00 +CPU= 29.324
  15:50:09 STEP 1764 H=1764:00 +CPU= 52.064
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001764
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001764
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001764
  15:50:38 STEP 1765 H=1765:00 +CPU= 29.117
  15:51:09 STEP 1766 H=1766:00 +CPU= 30.755
  15:51:57 STEP 1767 H=1767:00 +CPU= 47.128
  15:52:28 STEP 1768 H=1768:00 +CPU= 30.814
  15:52:58 STEP 1769 H=1769:00 +CPU= 29.571
  15:53:47 STEP 1770 H=1770:00 +CPU= 48.674
  15:54:16 STEP 1771 H=1771:00 +CPU= 28.820
  15:54:47 STEP 1772 H=1772:00 +CPU= 29.964
  15:55:34 STEP 1773 H=1773:00 +CPU= 47.169
  15:56:06 STEP 1774 H=1774:00 +CPU= 31.130
  15:56:37 STEP 1775 H=1775:00 +CPU= 30.835
  15:57:34 STEP 1776 H=1776:00 +CPU= 56.220
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001776
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001776
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001776
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001752
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001752
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001752
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001764
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001764
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001764
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_73.zip
Uploading the intermediate file: upload_file_73.zip
  15:58:04 STEP 1777 H=1777:00 +CPU= 29.548
  15:58:35 STEP 1778 H=1778:00 +CPU= 30.494
  15:59:22 STEP 1779 H=1779:00 +CPU= 46.686
  15:59:52 STEP 1780 H=1780:00 +CPU= 29.725
  16:00:22 STEP 1781 H=1781:00 +CPU= 29.712
  16:01:11 STEP 1782 H=1782:00 +CPU= 48.335
  16:01:42 STEP 1783 H=1783:00 +CPU= 29.967
  16:02:13 STEP 1784 H=1784:00 +CPU= 30.904
  16:03:01 STEP 1785 H=1785:00 +CPU= 47.305
  16:03:32 STEP 1786 H=1786:00 +CPU= 30.561
  16:04:01 STEP 1787 H=1787:00 +CPU= 29.204
  16:04:55 STEP 1788 H=1788:00 +CPU= 52.353
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001788
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001788
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001788
  16:05:25 STEP 1789 H=1789:00 +CPU= 30.165
  16:05:56 STEP 1790 H=1790:00 +CPU= 30.827
  16:06:43 STEP 1791 H=1791:00 +CPU= 46.338
  16:07:14 STEP 1792 H=1792:00 +CPU= 30.688
  16:07:44 STEP 1793 H=1793:00 +CPU= 29.227
  16:08:32 STEP 1794 H=1794:00 +CPU= 48.069
  16:09:01 STEP 1795 H=1795:00 +CPU= 28.853
  16:09:32 STEP 1796 H=1796:00 +CPU= 30.038
  16:10:20 STEP 1797 H=1797:00 +CPU= 47.539
  16:10:51 STEP 1798 H=1798:00 +CPU= 30.927
  16:11:21 STEP 1799 H=1799:00 +CPU= 29.923
  16:12:17 STEP 1800 H=1800:00 +CPU= 54.991
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001800
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001800
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001800
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001776
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001776
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001776
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001788
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001788
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001788
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_74.zip
Uploading the intermediate file: upload_file_74.zip
Uploading trickle at timestep: 6480000
  16:12:48 STEP 1801 H=1801:00 +CPU= 29.828
  16:13:18 STEP 1802 H=1802:00 +CPU= 30.331
  16:14:05 STEP 1803 H=1803:00 +CPU= 46.055
  16:14:37 STEP 1804 H=1804:00 +CPU= 31.077
  16:15:06 STEP 1805 H=1805:00 +CPU= 28.580
  16:15:54 STEP 1806 H=1806:00 +CPU= 47.566
  16:16:24 STEP 1807 H=1807:00 +CPU= 29.940
  16:16:55 STEP 1808 H=1808:00 +CPU= 30.486
  16:17:42 STEP 1809 H=1809:00 +CPU= 47.155
  16:18:14 STEP 1810 H=1810:00 +CPU= 31.112
  16:18:44 STEP 1811 H=1811:00 +CPU= 29.708
  16:19:37 STEP 1812 H=1812:00 +CPU= 52.315
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001812
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001812
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001812
  16:20:06 STEP 1813 H=1813:00 +CPU= 29.363
  16:20:37 STEP 1814 H=1814:00 +CPU= 30.570
  16:21:25 STEP 1815 H=1815:00 +CPU= 47.428
  16:21:57 STEP 1816 H=1816:00 +CPU= 30.908
  16:22:27 STEP 1817 H=1817:00 +CPU= 29.561
  16:23:16 STEP 1818 H=1818:00 +CPU= 48.563
  16:23:46 STEP 1819 H=1819:00 +CPU= 29.652
  16:24:16 STEP 1820 H=1820:00 +CPU= 30.022
  16:25:04 STEP 1821 H=1821:00 +CPU= 47.701
  16:25:35 STEP 1822 H=1822:00 +CPU= 30.425
  16:26:06 STEP 1823 H=1823:00 +CPU= 30.211
  16:27:03 STEP 1824 H=1824:00 +CPU= 56.599
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001824
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001824
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001824
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001800
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001800
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001800
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001812
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001812
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001812
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_75.zip
Uploading the intermediate file: upload_file_75.zip
  16:27:33 STEP 1825 H=1825:00 +CPU= 29.351
  16:28:04 STEP 1826 H=1826:00 +CPU= 30.724
  16:28:52 STEP 1827 H=1827:00 +CPU= 47.346
  16:29:24 STEP 1828 H=1828:00 +CPU= 31.611
  16:29:55 STEP 1829 H=1829:00 +CPU= 30.729
  16:30:46 STEP 1830 H=1830:00 +CPU= 49.820
  16:31:16 STEP 1831 H=1831:00 +CPU= 30.305
  16:31:47 STEP 1832 H=1832:00 +CPU= 30.418
  16:32:35 STEP 1833 H=1833:00 +CPU= 47.475
  16:33:06 STEP 1834 H=1834:00 +CPU= 30.891
  16:33:36 STEP 1835 H=1835:00 +CPU= 29.496
  16:34:30 STEP 1836 H=1836:00 +CPU= 53.265
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001836
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001836
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001836
  16:34:59 STEP 1837 H=1837:00 +CPU= 29.134
  16:35:31 STEP 1838 H=1838:00 +CPU= 30.986
  16:36:18 STEP 1839 H=1839:00 +CPU= 47.068
  16:36:49 STEP 1840 H=1840:00 +CPU= 30.290
  16:37:19 STEP 1841 H=1841:00 +CPU= 29.530
  16:38:08 STEP 1842 H=1842:00 +CPU= 48.638
  16:38:38 STEP 1843 H=1843:00 +CPU= 29.312
  16:39:08 STEP 1844 H=1844:00 +CPU= 30.256
  16:39:57 STEP 1845 H=1845:00 +CPU= 47.724
  16:40:28 STEP 1846 H=1846:00 +CPU= 30.682
  16:40:59 STEP 1847 H=1847:00 +CPU= 30.878
  16:41:56 STEP 1848 H=1848:00 +CPU= 56.339
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001848
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001848
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001848
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001824
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001824
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001824
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001836
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001836
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001836
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_76.zip
Uploading the intermediate file: upload_file_76.zip
  16:42:26 STEP 1849 H=1849:00 +CPU= 29.758
  16:42:58 STEP 1850 H=1850:00 +CPU= 31.003
  16:43:45 STEP 1851 H=1851:00 +CPU= 46.788
  16:44:16 STEP 1852 H=1852:00 +CPU= 30.758
  16:44:46 STEP 1853 H=1853:00 +CPU= 29.597
  16:45:35 STEP 1854 H=1854:00 +CPU= 48.626
  16:46:06 STEP 1855 H=1855:00 +CPU= 30.156
  16:46:37 STEP 1856 H=1856:00 +CPU= 30.911
  16:47:25 STEP 1857 H=1857:00 +CPU= 47.158
  16:47:56 STEP 1858 H=1858:00 +CPU= 31.014
  16:48:26 STEP 1859 H=1859:00 +CPU= 29.980
  16:49:21 STEP 1860 H=1860:00 +CPU= 53.757
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001860
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001860
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001860
  16:49:52 STEP 1861 H=1861:00 +CPU= 30.702
  16:50:24 STEP 1862 H=1862:00 +CPU= 31.671
  16:51:12 STEP 1863 H=1863:00 +CPU= 47.800
  16:51:44 STEP 1864 H=1864:00 +CPU= 31.060
  16:52:14 STEP 1865 H=1865:00 +CPU= 29.540
  16:53:03 STEP 1866 H=1866:00 +CPU= 48.891
  16:53:34 STEP 1867 H=1867:00 +CPU= 30.018
  16:54:04 STEP 1868 H=1868:00 +CPU= 29.899
  16:54:51 STEP 1869 H=1869:00 +CPU= 46.877
  16:55:22 STEP 1870 H=1870:00 +CPU= 30.381
  16:55:53 STEP 1871 H=1871:00 +CPU= 31.109
  16:56:53 STEP 1872 H=1872:00 +CPU= 58.362
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001872
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001872
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001872
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001848
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001848
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001848
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001860
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001860
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001860
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_77.zip
Uploading the intermediate file: upload_file_77.zip
  16:57:23 STEP 1873 H=1873:00 +CPU= 29.612
  16:57:54 STEP 1874 H=1874:00 +CPU= 30.708
  16:58:42 STEP 1875 H=1875:00 +CPU= 48.152
  16:59:13 STEP 1876 H=1876:00 +CPU= 30.518
  16:59:43 STEP 1877 H=1877:00 +CPU= 29.994
  17:00:32 STEP 1878 H=1878:00 +CPU= 48.239
  17:01:02 STEP 1879 H=1879:00 +CPU= 29.653
  17:01:34 STEP 1880 H=1880:00 +CPU= 31.332
  17:02:22 STEP 1881 H=1881:00 +CPU= 47.150
  17:02:53 STEP 1882 H=1882:00 +CPU= 30.585
  17:03:23 STEP 1883 H=1883:00 +CPU= 29.984
  17:04:17 STEP 1884 H=1884:00 +CPU= 53.133
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001884
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001884
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001884
  17:04:47 STEP 1885 H=1885:00 +CPU= 30.346
  17:05:19 STEP 1886 H=1886:00 +CPU= 31.552
  17:06:07 STEP 1887 H=1887:00 +CPU= 47.288
  17:06:39 STEP 1888 H=1888:00 +CPU= 31.135
  17:07:09 STEP 1889 H=1889:00 +CPU= 29.625
  17:07:58 STEP 1890 H=1890:00 +CPU= 49.006
  17:08:28 STEP 1891 H=1891:00 +CPU= 29.870
  17:08:59 STEP 1892 H=1892:00 +CPU= 30.624
  17:09:48 STEP 1893 H=1893:00 +CPU= 48.102
  17:10:20 STEP 1894 H=1894:00 +CPU= 31.683
  17:10:52 STEP 1895 H=1895:00 +CPU= 31.481
  17:11:50 STEP 1896 H=1896:00 +CPU= 57.158
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001896
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001896
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001896
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001872
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001872
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001872
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001884
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001884
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001884
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_78.zip
Uploading the intermediate file: upload_file_78.zip
  17:12:20 STEP 1897 H=1897:00 +CPU= 29.633
  17:12:51 STEP 1898 H=1898:00 +CPU= 30.793
  17:13:40 STEP 1899 H=1899:00 +CPU= 48.449
  17:14:12 STEP 1900 H=1900:00 +CPU= 31.433
  17:14:43 STEP 1901 H=1901:00 +CPU= 31.104
  17:15:34 STEP 1902 H=1902:00 +CPU= 49.908
  17:16:04 STEP 1903 H=1903:00 +CPU= 29.896
  17:16:36 STEP 1904 H=1904:00 +CPU= 31.339
  17:17:24 STEP 1905 H=1905:00 +CPU= 47.867
  17:17:56 STEP 1906 H=1906:00 +CPU= 31.070
  17:18:26 STEP 1907 H=1907:00 +CPU= 30.173
  17:19:20 STEP 1908 H=1908:00 +CPU= 53.222
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001908
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001908
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001908
  17:19:51 STEP 1909 H=1909:00 +CPU= 30.789
  17:20:21 STEP 1910 H=1910:00 +CPU= 29.884
  17:21:11 STEP 1911 H=1911:00 +CPU= 48.443
  17:21:42 STEP 1912 H=1912:00 +CPU= 31.279
  17:22:12 STEP 1913 H=1913:00 +CPU= 29.621
  17:23:02 STEP 1914 H=1914:00 +CPU= 49.181
  17:23:32 STEP 1915 H=1915:00 +CPU= 30.331
  17:24:04 STEP 1916 H=1916:00 +CPU= 31.251
  17:24:52 STEP 1917 H=1917:00 +CPU= 47.635
  17:25:25 STEP 1918 H=1918:00 +CPU= 32.388
  17:25:56 STEP 1919 H=1919:00 +CPU= 30.985
  17:26:54 STEP 1920 H=1920:00 +CPU= 57.046
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMGGhq0f+001920
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMSHhq0f+001920
Moving to projects directory: /data/boinc/boinc-client/slots/23/ICMUAhq0f+001920
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001896
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001896
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001896
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMGGhq0f+001908
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMSHhq0f+001908
Adding to the zip: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184948/ICMUAhq0f+001908
Zipping up the intermediate file: /data/boinc/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0304_1999050100_123_968_12184948_0_r212459995_79.zip
Uploading the intermediate file: upload_file_79.zip
  17:27:25 STEP 1921 H=1921:00 +CPU= 30.285
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:26:46 1478188 22270226 oifs_43r3_ps_0304_1999050100_123_968_12184948_0 6,480,000 62,253 0.0096
09 May 2023 16:24:58 1478188 22270226 oifs_43r3_ps_0304_1999050100_123_968_12184948_0 5,356,800 50,471 0.0094
09 May 2023 16:24:45 1478188 22270226 oifs_43r3_ps_0304_1999050100_123_968_12184948_0 4,492,800 42,012 0.0094
09 May 2023 16:24:35 1478188 22270226 oifs_43r3_ps_0304_1999050100_123_968_12184948_0 3,628,800 34,066 0.0094
09 May 2023 16:23:51 1478188 22270226 oifs_43r3_ps_0304_1999050100_123_968_12184948_0 2,678,400 24,433 0.0091
09 May 2023 16:22:42 1478188 22270226 oifs_43r3_ps_0304_1999050100_123_968_12184948_0 1,814,400 16,425 0.0091
09 May 2023 16:22:31 1478188 22270226 oifs_43r3_ps_0304_1999050100_123_968_12184948_0 950,400 8,373 0.0088


©2024 cpdn.org