climateprediction.net home page
Task 22285376

Task 22285376

Name oifs_43r3_ps_0576_2013050100_123_982_12199220_0
Workunit 12199220
Created 24 Dec 2022, 7:56:08 UTC
Sent 20 Jan 2023, 12:58:00 UTC
Report deadline 21 Mar 2023, 12:58:00 UTC
Received 21 Jan 2023, 13:15:58 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1538317
Run time 17 hours 15 min 54 sec
CPU time 16 hours 44 min 30 sec
Validate state Invalid
Credit 12,700.80
Device peak FLOPS 5.68 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.19 MB
Peak swap size 4,978.08 MB
Peak disk usage 3,647.78 MB
Stderr
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
b/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_100.zip
Uploading the intermediate file: upload_file_100.zip
  09:53:39 STEP 2425 H=2425:00 +CPU= 17.568
  09:53:58 STEP 2426 H=2426:00 +CPU= 17.976
  09:54:27 STEP 2427 H=2427:00 +CPU= 28.631
  09:54:45 STEP 2428 H=2428:00 +CPU= 18.133
  09:55:03 STEP 2429 H=2429:00 +CPU= 17.504
  09:55:33 STEP 2430 H=2430:00 +CPU= 29.062
  09:55:51 STEP 2431 H=2431:00 +CPU= 17.577
  09:56:09 STEP 2432 H=2432:00 +CPU= 18.088
  09:56:38 STEP 2433 H=2433:00 +CPU= 28.693
  09:56:57 STEP 2434 H=2434:00 +CPU= 18.094
  09:57:14 STEP 2435 H=2435:00 +CPU= 17.537
  09:57:47 STEP 2436 H=2436:00 +CPU= 31.670
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002436
  09:58:05 STEP 2437 H=2437:00 +CPU= 17.617
  09:58:23 STEP 2438 H=2438:00 +CPU= 18.033
  09:58:52 STEP 2439 H=2439:00 +CPU= 28.594
  09:59:11 STEP 2440 H=2440:00 +CPU= 18.051
  09:59:28 STEP 2441 H=2441:00 +CPU= 17.512
  09:59:58 STEP 2442 H=2442:00 +CPU= 29.064
  10:00:16 STEP 2443 H=2443:00 +CPU= 17.645
  10:00:34 STEP 2444 H=2444:00 +CPU= 17.971
  10:01:03 STEP 2445 H=2445:00 +CPU= 28.652
  10:01:22 STEP 2446 H=2446:00 +CPU= 17.990
  10:01:40 STEP 2447 H=2447:00 +CPU= 17.982
  10:02:18 STEP 2448 H=2448:00 +CPU= 33.232
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_101.zip
Uploading the intermediate file: upload_file_101.zip
  10:02:36 STEP 2449 H=2449:00 +CPU= 17.520
  10:02:55 STEP 2450 H=2450:00 +CPU= 18.073
  10:03:24 STEP 2451 H=2451:00 +CPU= 28.675
  10:03:42 STEP 2452 H=2452:00 +CPU= 18.063
  10:04:00 STEP 2453 H=2453:00 +CPU= 17.535
  10:04:30 STEP 2454 H=2454:00 +CPU= 29.016
  10:04:48 STEP 2455 H=2455:00 +CPU= 17.655
  10:05:06 STEP 2456 H=2456:00 +CPU= 18.099
  10:05:35 STEP 2457 H=2457:00 +CPU= 28.531
  10:05:54 STEP 2458 H=2458:00 +CPU= 18.149
  10:06:12 STEP 2459 H=2459:00 +CPU= 17.631
  10:06:44 STEP 2460 H=2460:00 +CPU= 31.625
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002460
  10:07:02 STEP 2461 H=2461:00 +CPU= 17.572
  10:07:20 STEP 2462 H=2462:00 +CPU= 18.061
  10:07:49 STEP 2463 H=2463:00 +CPU= 28.584
  10:08:08 STEP 2464 H=2464:00 +CPU= 17.981
  10:08:26 STEP 2465 H=2465:00 +CPU= 17.618
  10:08:55 STEP 2466 H=2466:00 +CPU= 29.089
  10:09:13 STEP 2467 H=2467:00 +CPU= 17.549
  10:09:31 STEP 2468 H=2468:00 +CPU= 18.039
  10:10:00 STEP 2469 H=2469:00 +CPU= 28.532
  10:10:19 STEP 2470 H=2470:00 +CPU= 18.019
  10:10:37 STEP 2471 H=2471:00 +CPU= 18.071
  10:11:13 STEP 2472 H=2472:00 +CPU= 33.231
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_102.zip
Uploading the intermediate file: upload_file_102.zip
  10:11:31 STEP 2473 H=2473:00 +CPU= 17.564
  10:11:50 STEP 2474 H=2474:00 +CPU= 18.166
  10:12:19 STEP 2475 H=2475:00 +CPU= 28.687
  10:12:37 STEP 2476 H=2476:00 +CPU= 18.120
  10:12:55 STEP 2477 H=2477:00 +CPU= 17.554
  10:13:25 STEP 2478 H=2478:00 +CPU= 29.298
  10:13:43 STEP 2479 H=2479:00 +CPU= 17.670
  10:14:01 STEP 2480 H=2480:00 +CPU= 18.033
  10:14:31 STEP 2481 H=2481:00 +CPU= 28.628
  10:14:49 STEP 2482 H=2482:00 +CPU= 18.135
  10:15:07 STEP 2483 H=2483:00 +CPU= 17.635
  10:15:39 STEP 2484 H=2484:00 +CPU= 31.625
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002484
  10:15:57 STEP 2485 H=2485:00 +CPU= 17.648
  10:16:16 STEP 2486 H=2486:00 +CPU= 18.096
  10:16:45 STEP 2487 H=2487:00 +CPU= 28.604
  10:17:03 STEP 2488 H=2488:00 +CPU= 18.179
  10:17:21 STEP 2489 H=2489:00 +CPU= 17.620
  10:17:51 STEP 2490 H=2490:00 +CPU= 29.155
  10:18:09 STEP 2491 H=2491:00 +CPU= 17.582
  10:18:27 STEP 2492 H=2492:00 +CPU= 18.112
  10:18:56 STEP 2493 H=2493:00 +CPU= 28.534
  10:19:15 STEP 2494 H=2494:00 +CPU= 18.039
  10:19:33 STEP 2495 H=2495:00 +CPU= 18.093
  10:20:10 STEP 2496 H=2496:00 +CPU= 33.226
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_103.zip
Uploading the intermediate file: upload_file_103.zip
  10:20:28 STEP 2497 H=2497:00 +CPU= 17.631
  10:20:46 STEP 2498 H=2498:00 +CPU= 18.070
  10:21:16 STEP 2499 H=2499:00 +CPU= 28.568
  10:21:34 STEP 2500 H=2500:00 +CPU= 18.094
  10:21:52 STEP 2501 H=2501:00 +CPU= 17.666
  10:22:22 STEP 2502 H=2502:00 +CPU= 29.223
  10:22:40 STEP 2503 H=2503:00 +CPU= 17.559
  10:22:58 STEP 2504 H=2504:00 +CPU= 18.119
  10:23:27 STEP 2505 H=2505:00 +CPU= 28.539
  10:23:46 STEP 2506 H=2506:00 +CPU= 18.090
  10:24:04 STEP 2507 H=2507:00 +CPU= 17.674
  10:24:36 STEP 2508 H=2508:00 +CPU= 31.726
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002508
  10:24:54 STEP 2509 H=2509:00 +CPU= 17.694
  10:25:13 STEP 2510 H=2510:00 +CPU= 18.166
  10:25:42 STEP 2511 H=2511:00 +CPU= 28.522
  10:26:00 STEP 2512 H=2512:00 +CPU= 18.034
  10:26:18 STEP 2513 H=2513:00 +CPU= 17.606
  10:26:48 STEP 2514 H=2514:00 +CPU= 29.162
  10:27:05 STEP 2515 H=2515:00 +CPU= 17.560
  10:27:24 STEP 2516 H=2516:00 +CPU= 18.081
  10:27:53 STEP 2517 H=2517:00 +CPU= 28.626
  10:28:11 STEP 2518 H=2518:00 +CPU= 18.028
  10:28:30 STEP 2519 H=2519:00 +CPU= 18.293
  10:29:05 STEP 2520 H=2520:00 +CPU= 33.089
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_104.zip
Uploading the intermediate file: upload_file_104.zip
  10:29:23 STEP 2521 H=2521:00 +CPU= 17.708
Uploading trickle at timestep: 9072000
  10:29:41 STEP 2522 H=2522:00 +CPU= 18.164
  10:30:11 STEP 2523 H=2523:00 +CPU= 28.766
  10:30:29 STEP 2524 H=2524:00 +CPU= 18.356
  10:30:47 STEP 2525 H=2525:00 +CPU= 17.650
  10:31:17 STEP 2526 H=2526:00 +CPU= 29.327
  10:31:35 STEP 2527 H=2527:00 +CPU= 17.660
  10:31:54 STEP 2528 H=2528:00 +CPU= 18.185
  10:32:23 STEP 2529 H=2529:00 +CPU= 28.731
  10:32:41 STEP 2530 H=2530:00 +CPU= 18.113
  10:33:00 STEP 2531 H=2531:00 +CPU= 17.690
  10:33:32 STEP 2532 H=2532:00 +CPU= 31.711
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002532
  10:33:50 STEP 2533 H=2533:00 +CPU= 17.676
  10:34:08 STEP 2534 H=2534:00 +CPU= 18.214
  10:34:37 STEP 2535 H=2535:00 +CPU= 28.560
  10:34:56 STEP 2536 H=2536:00 +CPU= 18.162
  10:35:14 STEP 2537 H=2537:00 +CPU= 17.656
  10:35:44 STEP 2538 H=2538:00 +CPU= 29.241
  10:36:02 STEP 2539 H=2539:00 +CPU= 17.685
  10:36:20 STEP 2540 H=2540:00 +CPU= 18.117
  10:36:49 STEP 2541 H=2541:00 +CPU= 28.532
  10:37:08 STEP 2542 H=2542:00 +CPU= 18.054
  10:37:26 STEP 2543 H=2543:00 +CPU= 18.170
  10:38:01 STEP 2544 H=2544:00 +CPU= 33.174
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_105.zip
Uploading the intermediate file: upload_file_105.zip
  10:38:20 STEP 2545 H=2545:00 +CPU= 17.626
  10:38:38 STEP 2546 H=2546:00 +CPU= 18.044
  10:39:08 STEP 2547 H=2547:00 +CPU= 29.095
  10:39:27 STEP 2548 H=2548:00 +CPU= 18.469
  10:39:45 STEP 2549 H=2549:00 +CPU= 17.651
  10:40:15 STEP 2550 H=2550:00 +CPU= 29.408
  10:40:33 STEP 2551 H=2551:00 +CPU= 17.701
  10:40:51 STEP 2552 H=2552:00 +CPU= 18.011
  10:41:20 STEP 2553 H=2553:00 +CPU= 28.736
  10:41:39 STEP 2554 H=2554:00 +CPU= 18.146
  10:41:57 STEP 2555 H=2555:00 +CPU= 17.508
  10:42:29 STEP 2556 H=2556:00 +CPU= 31.679
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002556
  10:42:47 STEP 2557 H=2557:00 +CPU= 17.666
  10:43:05 STEP 2558 H=2558:00 +CPU= 18.066
  10:43:35 STEP 2559 H=2559:00 +CPU= 28.792
  10:43:53 STEP 2560 H=2560:00 +CPU= 18.089
  10:44:11 STEP 2561 H=2561:00 +CPU= 17.615
  10:44:41 STEP 2562 H=2562:00 +CPU= 29.256
  10:44:59 STEP 2563 H=2563:00 +CPU= 17.710
  10:45:17 STEP 2564 H=2564:00 +CPU= 18.081
  10:45:46 STEP 2565 H=2565:00 +CPU= 28.662
  10:46:05 STEP 2566 H=2566:00 +CPU= 18.055
  10:46:23 STEP 2567 H=2567:00 +CPU= 18.058
  10:46:59 STEP 2568 H=2568:00 +CPU= 33.153
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_106.zip
Uploading the intermediate file: upload_file_106.zip
  10:47:17 STEP 2569 H=2569:00 +CPU= 17.554
  10:47:36 STEP 2570 H=2570:00 +CPU= 17.960
  10:48:05 STEP 2571 H=2571:00 +CPU= 28.568
  10:48:23 STEP 2572 H=2572:00 +CPU= 18.111
  10:48:41 STEP 2573 H=2573:00 +CPU= 17.514
  10:49:11 STEP 2574 H=2574:00 +CPU= 29.166
  10:49:28 STEP 2575 H=2575:00 +CPU= 17.534
  10:49:47 STEP 2576 H=2576:00 +CPU= 17.978
  10:50:16 STEP 2577 H=2577:00 +CPU= 28.554
  10:50:34 STEP 2578 H=2578:00 +CPU= 18.049
  10:50:52 STEP 2579 H=2579:00 +CPU= 17.401
  10:51:24 STEP 2580 H=2580:00 +CPU= 31.531
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002580
  10:51:42 STEP 2581 H=2581:00 +CPU= 17.602
  10:52:00 STEP 2582 H=2582:00 +CPU= 17.831
  10:52:29 STEP 2583 H=2583:00 +CPU= 28.560
  10:52:48 STEP 2584 H=2584:00 +CPU= 17.998
  10:53:05 STEP 2585 H=2585:00 +CPU= 17.112
  10:53:35 STEP 2586 H=2586:00 +CPU= 29.114
  10:53:53 STEP 2587 H=2587:00 +CPU= 17.566
  10:54:11 STEP 2588 H=2588:00 +CPU= 18.026
  10:54:40 STEP 2589 H=2589:00 +CPU= 28.426
  10:54:58 STEP 2590 H=2590:00 +CPU= 18.074
  10:55:17 STEP 2591 H=2591:00 +CPU= 17.946
  10:55:54 STEP 2592 H=2592:00 +CPU= 33.230
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_107.zip
Uploading the intermediate file: upload_file_107.zip
  10:56:13 STEP 2593 H=2593:00 +CPU= 17.546
  10:56:31 STEP 2594 H=2594:00 +CPU= 18.044
  10:57:00 STEP 2595 H=2595:00 +CPU= 28.511
  10:57:19 STEP 2596 H=2596:00 +CPU= 18.113
  10:57:36 STEP 2597 H=2597:00 +CPU= 17.499
  10:58:06 STEP 2598 H=2598:00 +CPU= 29.104
  10:58:24 STEP 2599 H=2599:00 +CPU= 17.632
  10:58:42 STEP 2600 H=2600:00 +CPU= 17.947
  10:59:11 STEP 2601 H=2601:00 +CPU= 28.579
  10:59:30 STEP 2602 H=2602:00 +CPU= 18.043
  10:59:48 STEP 2603 H=2603:00 +CPU= 17.581
  11:00:20 STEP 2604 H=2604:00 +CPU= 31.514
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002604
  11:00:38 STEP 2605 H=2605:00 +CPU= 17.519
  11:00:56 STEP 2606 H=2606:00 +CPU= 17.983
  11:01:25 STEP 2607 H=2607:00 +CPU= 28.564
  11:01:44 STEP 2608 H=2608:00 +CPU= 18.166
  11:02:01 STEP 2609 H=2609:00 +CPU= 17.425
  11:02:31 STEP 2610 H=2610:00 +CPU= 29.157
  11:02:49 STEP 2611 H=2611:00 +CPU= 17.563
  11:03:07 STEP 2612 H=2612:00 +CPU= 17.966
  11:03:36 STEP 2613 H=2613:00 +CPU= 28.546
  11:03:55 STEP 2614 H=2614:00 +CPU= 18.086
  11:04:13 STEP 2615 H=2615:00 +CPU= 17.909
  11:04:49 STEP 2616 H=2616:00 +CPU= 33.150
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_108.zip
Uploading the intermediate file: upload_file_108.zip
  11:05:07 STEP 2617 H=2617:00 +CPU= 17.582
  11:05:26 STEP 2618 H=2618:00 +CPU= 18.134
  11:05:55 STEP 2619 H=2619:00 +CPU= 28.782
  11:06:14 STEP 2620 H=2620:00 +CPU= 18.625
  11:06:33 STEP 2621 H=2621:00 +CPU= 18.183
  11:07:03 STEP 2622 H=2622:00 +CPU= 29.674
  11:07:21 STEP 2623 H=2623:00 +CPU= 18.033
  11:07:40 STEP 2624 H=2624:00 +CPU= 18.462
  11:08:10 STEP 2625 H=2625:00 +CPU= 29.418
  11:08:29 STEP 2626 H=2626:00 +CPU= 18.497
  11:08:47 STEP 2627 H=2627:00 +CPU= 18.233
  11:09:20 STEP 2628 H=2628:00 +CPU= 32.003
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002628
  11:09:38 STEP 2629 H=2629:00 +CPU= 17.825
  11:09:57 STEP 2630 H=2630:00 +CPU= 18.381
  11:10:27 STEP 2631 H=2631:00 +CPU= 29.037
  11:10:45 STEP 2632 H=2632:00 +CPU= 18.566
  11:11:04 STEP 2633 H=2633:00 +CPU= 18.051
  11:11:34 STEP 2634 H=2634:00 +CPU= 29.956
  11:11:53 STEP 2635 H=2635:00 +CPU= 18.054
  11:12:11 STEP 2636 H=2636:00 +CPU= 18.301
  11:12:41 STEP 2637 H=2637:00 +CPU= 29.268
  11:13:01 STEP 2638 H=2638:00 +CPU= 18.971
  11:13:20 STEP 2639 H=2639:00 +CPU= 18.509
  11:13:58 STEP 2640 H=2640:00 +CPU= 33.404
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_109.zip
Uploading the intermediate file: upload_file_109.zip
  11:14:16 STEP 2641 H=2641:00 +CPU= 17.846
  11:14:35 STEP 2642 H=2642:00 +CPU= 18.578
  11:15:05 STEP 2643 H=2643:00 +CPU= 29.322
  11:15:24 STEP 2644 H=2644:00 +CPU= 18.400
  11:15:42 STEP 2645 H=2645:00 +CPU= 18.127
  11:16:13 STEP 2646 H=2646:00 +CPU= 29.545
  11:16:31 STEP 2647 H=2647:00 +CPU= 17.831
  11:16:50 STEP 2648 H=2648:00 +CPU= 18.847
  11:17:17 STEP 2649 H=2649:00 +CPU= 26.882
  11:17:34 STEP 2650 H=2650:00 +CPU= 16.514
  11:17:53 STEP 2651 H=2651:00 +CPU= 18.058
  11:18:23 STEP 2652 H=2652:00 +CPU= 29.310
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002652
  11:18:39 STEP 2653 H=2653:00 +CPU= 16.065
  11:18:58 STEP 2654 H=2654:00 +CPU= 18.695
  11:19:27 STEP 2655 H=2655:00 +CPU= 28.275
  11:19:51 STEP 2656 H=2656:00 +CPU= 23.545
  11:20:17 STEP 2657 H=2657:00 +CPU= 25.281
  11:20:53 STEP 2658 H=2658:00 +CPU= 35.439
  11:21:07 STEP 2659 H=2659:00 +CPU= 13.867
  11:21:23 STEP 2660 H=2660:00 +CPU= 15.328
  11:21:51 STEP 2661 H=2661:00 +CPU= 27.730
  11:22:18 STEP 2662 H=2662:00 +CPU= 25.920
  11:22:44 STEP 2663 H=2663:00 +CPU= 25.953
  11:23:32 STEP 2664 H=2664:00 +CPU= 44.755
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_110.zip
Uploading the intermediate file: upload_file_110.zip
  11:23:57 STEP 2665 H=2665:00 +CPU= 25.083
  11:24:23 STEP 2666 H=2666:00 +CPU= 24.741
  11:24:45 STEP 2667 H=2667:00 +CPU= 21.124
  11:25:02 STEP 2668 H=2668:00 +CPU= 16.779
  11:25:20 STEP 2669 H=2669:00 +CPU= 17.982
  11:25:50 STEP 2670 H=2670:00 +CPU= 28.968
  11:26:09 STEP 2671 H=2671:00 +CPU= 18.600
  11:26:26 STEP 2672 H=2672:00 +CPU= 16.856
  11:26:55 STEP 2673 H=2673:00 +CPU= 28.898
  11:27:13 STEP 2674 H=2674:00 +CPU= 17.697
  11:27:32 STEP 2675 H=2675:00 +CPU= 18.530
  11:28:05 STEP 2676 H=2676:00 +CPU= 32.408
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002676
  11:28:24 STEP 2677 H=2677:00 +CPU= 18.399
  11:28:43 STEP 2678 H=2678:00 +CPU= 18.841
  11:29:13 STEP 2679 H=2679:00 +CPU= 29.549
  11:29:33 STEP 2680 H=2680:00 +CPU= 18.984
  11:29:51 STEP 2681 H=2681:00 +CPU= 18.294
  11:30:22 STEP 2682 H=2682:00 +CPU= 29.979
  11:30:41 STEP 2683 H=2683:00 +CPU= 18.448
  11:31:00 STEP 2684 H=2684:00 +CPU= 18.696
  11:31:30 STEP 2685 H=2685:00 +CPU= 29.778
  11:31:50 STEP 2686 H=2686:00 +CPU= 19.071
  11:32:09 STEP 2687 H=2687:00 +CPU= 18.991
  11:32:47 STEP 2688 H=2688:00 +CPU= 34.178
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_111.zip
Uploading the intermediate file: upload_file_111.zip
  11:33:06 STEP 2689 H=2689:00 +CPU= 18.345
  11:33:25 STEP 2690 H=2690:00 +CPU= 18.843
  11:33:55 STEP 2691 H=2691:00 +CPU= 29.490
  11:34:15 STEP 2692 H=2692:00 +CPU= 19.415
  11:34:34 STEP 2693 H=2693:00 +CPU= 18.781
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0576
(argv4) batchid: 982
(argv5) wuid: 12199220
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c837241c7951667d05b39bbc4812bf43 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0576_2013050100_123_982_12199220.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0576_2013050100_123_982_12199220.zip
ic_ancil_file: ic_ancil_12199220
ifsdata_file: ifsdata_12199220
climate_data_file: clim_data_12199220
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_664bc9e12ca318fb783a2a093a9e002c to: /var/lib/boinc-client/slots/6/ic_ancil_12199220.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12199220.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_429251a49c4f549b3ec89669ada7ac7f to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12199220.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12199220.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12199220.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12199220.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12199220.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12199220.xml
-- Model is restarting --
Adjusting last_iter, 2693, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12199220.xml
last_cpu_time: 55987
upload_file_number: 112
last_iter: 2687
last_upload: 9676800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861
The child process has been launched with process id: 3415
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:antix1vbox:1:1:3415:3415] [20230121:114420:1674301460:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:44:57 STEP 2688 H=2688:00 +CPU= 33.440
  11:45:16 STEP 2689 H=2689:00 +CPU= 18.235
  11:45:35 STEP 2690 H=2690:00 +CPU= 18.588
  11:46:05 STEP 2691 H=2691:00 +CPU= 29.407
  11:46:24 STEP 2692 H=2692:00 +CPU= 18.431
  11:46:42 STEP 2693 H=2693:00 +CPU= 18.153
  11:47:13 STEP 2694 H=2694:00 +CPU= 29.998
  11:47:32 STEP 2695 H=2695:00 +CPU= 18.364
  11:47:51 STEP 2696 H=2696:00 +CPU= 18.734
  11:48:20 STEP 2697 H=2697:00 +CPU= 29.113
  11:48:39 STEP 2698 H=2698:00 +CPU= 18.302
  11:48:57 STEP 2699 H=2699:00 +CPU= 17.928
  11:49:28 STEP 2700 H=2700:00 +CPU= 29.985
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002700
  11:49:44 STEP 2701 H=2701:00 +CPU= 16.213
  11:50:00 STEP 2702 H=2702:00 +CPU= 15.383
  11:50:23 STEP 2703 H=2703:00 +CPU= 22.702
  11:50:36 STEP 2704 H=2704:00 +CPU= 13.058
  11:50:49 STEP 2705 H=2705:00 +CPU= 12.700
  11:51:10 STEP 2706 H=2706:00 +CPU= 20.740
  11:51:27 STEP 2707 H=2707:00 +CPU= 15.985
  11:51:46 STEP 2708 H=2708:00 +CPU= 18.886
  11:52:16 STEP 2709 H=2709:00 +CPU= 29.542
  11:52:36 STEP 2710 H=2710:00 +CPU= 19.062
  11:52:55 STEP 2711 H=2711:00 +CPU= 18.991
  11:53:38 STEP 2712 H=2712:00 +CPU= 34.251
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_112.zip
Uploading the intermediate file: upload_file_112.zip
  11:53:56 STEP 2713 H=2713:00 +CPU= 17.780
  11:54:11 STEP 2714 H=2714:00 +CPU= 14.401
  11:54:38 STEP 2715 H=2715:00 +CPU= 26.505
  11:54:57 STEP 2716 H=2716:00 +CPU= 19.018
  11:55:16 STEP 2717 H=2717:00 +CPU= 18.648
  11:55:47 STEP 2718 H=2718:00 +CPU= 30.374
  11:56:06 STEP 2719 H=2719:00 +CPU= 18.660
  11:56:26 STEP 2720 H=2720:00 +CPU= 19.032
  11:56:55 STEP 2721 H=2721:00 +CPU= 28.450
  11:57:10 STEP 2722 H=2722:00 +CPU= 15.003
  11:57:23 STEP 2723 H=2723:00 +CPU= 12.662
  11:57:53 STEP 2724 H=2724:00 +CPU= 29.831
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002724
  11:58:12 STEP 2725 H=2725:00 +CPU= 18.511
  11:58:32 STEP 2726 H=2726:00 +CPU= 19.088
  11:59:02 STEP 2727 H=2727:00 +CPU= 29.782
  11:59:21 STEP 2728 H=2728:00 +CPU= 18.963
  11:59:40 STEP 2729 H=2729:00 +CPU= 18.674
  12:00:10 STEP 2730 H=2730:00 +CPU= 28.663
  12:00:23 STEP 2731 H=2731:00 +CPU= 13.342
  12:00:41 STEP 2732 H=2732:00 +CPU= 17.499
  12:01:11 STEP 2733 H=2733:00 +CPU= 29.643
  12:01:31 STEP 2734 H=2734:00 +CPU= 19.052
  12:01:51 STEP 2735 H=2735:00 +CPU= 19.093
  12:02:36 STEP 2736 H=2736:00 +CPU= 34.205
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_113.zip
Uploading the intermediate file: upload_file_113.zip
  12:02:54 STEP 2737 H=2737:00 +CPU= 18.298
  12:03:11 STEP 2738 H=2738:00 +CPU= 16.793
  12:03:34 STEP 2739 H=2739:00 +CPU= 21.907
  12:03:53 STEP 2740 H=2740:00 +CPU= 18.944
  12:04:12 STEP 2741 H=2741:00 +CPU= 18.621
  12:04:43 STEP 2742 H=2742:00 +CPU= 30.142
  12:05:02 STEP 2743 H=2743:00 +CPU= 18.545
  12:05:21 STEP 2744 H=2744:00 +CPU= 19.095
  12:05:51 STEP 2745 H=2745:00 +CPU= 29.561
  12:06:09 STEP 2746 H=2746:00 +CPU= 17.710
  12:06:24 STEP 2747 H=2747:00 +CPU= 14.799
  12:06:50 STEP 2748 H=2748:00 +CPU= 25.344
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002748
  12:07:09 STEP 2749 H=2749:00 +CPU= 18.472
  12:07:28 STEP 2750 H=2750:00 +CPU= 18.981
  12:07:59 STEP 2751 H=2751:00 +CPU= 29.583
  12:08:18 STEP 2752 H=2752:00 +CPU= 19.079
  12:08:37 STEP 2753 H=2753:00 +CPU= 18.546
  12:09:07 STEP 2754 H=2754:00 +CPU= 29.614
  12:09:24 STEP 2755 H=2755:00 +CPU= 16.365
  12:09:38 STEP 2756 H=2756:00 +CPU= 13.716
  12:10:08 STEP 2757 H=2757:00 +CPU= 29.559
  12:10:27 STEP 2758 H=2758:00 +CPU= 18.849
  12:10:49 STEP 2759 H=2759:00 +CPU= 19.083
  12:11:33 STEP 2760 H=2760:00 +CPU= 34.433
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_114.zip
Uploading the intermediate file: upload_file_114.zip
  12:11:52 STEP 2761 H=2761:00 +CPU= 18.669
  12:12:12 STEP 2762 H=2762:00 +CPU= 18.941
  12:12:37 STEP 2763 H=2763:00 +CPU= 24.677
  12:12:52 STEP 2764 H=2764:00 +CPU= 15.074
  12:13:11 STEP 2765 H=2765:00 +CPU= 18.442
  12:13:42 STEP 2766 H=2766:00 +CPU= 30.110
  12:14:01 STEP 2767 H=2767:00 +CPU= 18.555
  12:14:20 STEP 2768 H=2768:00 +CPU= 19.013
  12:14:50 STEP 2769 H=2769:00 +CPU= 29.612
  12:15:09 STEP 2770 H=2770:00 +CPU= 18.564
  12:15:26 STEP 2771 H=2771:00 +CPU= 16.371
  12:15:51 STEP 2772 H=2772:00 +CPU= 24.499
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002772
  12:16:10 STEP 2773 H=2773:00 +CPU= 18.488
  12:16:29 STEP 2774 H=2774:00 +CPU= 18.776
  12:16:59 STEP 2775 H=2775:00 +CPU= 29.647
  12:17:19 STEP 2776 H=2776:00 +CPU= 19.038
  12:17:37 STEP 2777 H=2777:00 +CPU= 18.414
  12:18:08 STEP 2778 H=2778:00 +CPU= 30.095
  12:18:27 STEP 2779 H=2779:00 +CPU= 18.068
  12:18:42 STEP 2780 H=2780:00 +CPU= 14.664
  12:19:10 STEP 2781 H=2781:00 +CPU= 27.796
  12:19:29 STEP 2782 H=2782:00 +CPU= 18.954
  12:19:49 STEP 2783 H=2783:00 +CPU= 19.198
  12:20:28 STEP 2784 H=2784:00 +CPU= 34.140
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_115.zip
Uploading the intermediate file: upload_file_115.zip
  12:20:47 STEP 2785 H=2785:00 +CPU= 18.559
  12:21:06 STEP 2786 H=2786:00 +CPU= 19.013
  12:21:34 STEP 2787 H=2787:00 +CPU= 27.126
  12:21:47 STEP 2788 H=2788:00 +CPU= 13.456
  12:22:00 STEP 2789 H=2789:00 +CPU= 12.608
  12:22:31 STEP 2790 H=2790:00 +CPU= 29.828
  12:22:49 STEP 2791 H=2791:00 +CPU= 18.384
  12:23:09 STEP 2792 H=2792:00 +CPU= 18.998
  12:23:39 STEP 2793 H=2793:00 +CPU= 29.660
  12:23:59 STEP 2794 H=2794:00 +CPU= 19.157
  12:24:17 STEP 2795 H=2795:00 +CPU= 18.434
  12:24:48 STEP 2796 H=2796:00 +CPU= 29.732
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002796
  12:25:01 STEP 2797 H=2797:00 +CPU= 13.550
  12:25:21 STEP 2798 H=2798:00 +CPU= 18.785
  12:25:51 STEP 2799 H=2799:00 +CPU= 29.536
  12:26:10 STEP 2800 H=2800:00 +CPU= 19.001
  12:26:29 STEP 2801 H=2801:00 +CPU= 18.231
  12:27:00 STEP 2802 H=2802:00 +CPU= 30.206
  12:27:18 STEP 2803 H=2803:00 +CPU= 18.457
  12:27:38 STEP 2804 H=2804:00 +CPU= 18.887
  12:28:02 STEP 2805 H=2805:00 +CPU= 24.265
  12:28:19 STEP 2806 H=2806:00 +CPU= 15.654
  12:28:38 STEP 2807 H=2807:00 +CPU= 18.827
  12:29:18 STEP 2808 H=2808:00 +CPU= 34.121
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_116.zip
Uploading the intermediate file: upload_file_116.zip
  12:29:37 STEP 2809 H=2809:00 +CPU= 18.556
  12:29:56 STEP 2810 H=2810:00 +CPU= 18.921
  12:30:26 STEP 2811 H=2811:00 +CPU= 29.674
  12:30:46 STEP 2812 H=2812:00 +CPU= 18.988
  12:31:03 STEP 2813 H=2813:00 +CPU= 16.687
  12:31:26 STEP 2814 H=2814:00 +CPU= 23.388
  12:31:45 STEP 2815 H=2815:00 +CPU= 18.381
  12:32:04 STEP 2816 H=2816:00 +CPU= 18.714
  12:32:35 STEP 2817 H=2817:00 +CPU= 29.611
  12:32:54 STEP 2818 H=2818:00 +CPU= 19.123
  12:33:13 STEP 2819 H=2819:00 +CPU= 18.307
  12:33:46 STEP 2820 H=2820:00 +CPU= 32.595
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002820
  12:34:04 STEP 2821 H=2821:00 +CPU= 18.003
  12:34:21 STEP 2822 H=2822:00 +CPU= 16.485
  12:34:44 STEP 2823 H=2823:00 +CPU= 22.419
  12:35:03 STEP 2824 H=2824:00 +CPU= 18.860
  12:35:22 STEP 2825 H=2825:00 +CPU= 18.187
  12:35:53 STEP 2826 H=2826:00 +CPU= 30.214
  12:36:12 STEP 2827 H=2827:00 +CPU= 18.422
  12:36:31 STEP 2828 H=2828:00 +CPU= 18.659
  12:37:01 STEP 2829 H=2829:00 +CPU= 29.759
  12:37:21 STEP 2830 H=2830:00 +CPU= 18.736
  12:37:38 STEP 2831 H=2831:00 +CPU= 16.650
  12:38:10 STEP 2832 H=2832:00 +CPU= 29.160
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_117.zip
Uploading the intermediate file: upload_file_117.zip
  12:38:29 STEP 2833 H=2833:00 +CPU= 18.459
  12:38:48 STEP 2834 H=2834:00 +CPU= 18.864
  12:39:19 STEP 2835 H=2835:00 +CPU= 29.853
  12:39:38 STEP 2836 H=2836:00 +CPU= 18.953
  12:39:57 STEP 2837 H=2837:00 +CPU= 18.299
  12:40:27 STEP 2838 H=2838:00 +CPU= 29.854
  12:40:44 STEP 2839 H=2839:00 +CPU= 16.195
  12:40:57 STEP 2840 H=2840:00 +CPU= 13.752
  12:41:26 STEP 2841 H=2841:00 +CPU= 27.611
  12:41:45 STEP 2842 H=2842:00 +CPU= 19.088
  12:42:04 STEP 2843 H=2843:00 +CPU= 18.396
  12:42:37 STEP 2844 H=2844:00 +CPU= 32.723
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002844
  12:42:56 STEP 2845 H=2845:00 +CPU= 18.623
  12:43:16 STEP 2846 H=2846:00 +CPU= 19.030
  12:43:46 STEP 2847 H=2847:00 +CPU= 29.292
  12:44:02 STEP 2848 H=2848:00 +CPU= 16.042
  12:44:16 STEP 2849 H=2849:00 +CPU= 13.383
  12:44:49 STEP 2850 H=2850:00 +CPU= 31.069
  12:45:09 STEP 2851 H=2851:00 +CPU= 19.892
  12:45:31 STEP 2852 H=2852:00 +CPU= 20.541
  12:46:04 STEP 2853 H=2853:00 +CPU= 32.522
  12:46:27 STEP 2854 H=2854:00 +CPU= 21.113
  12:46:49 STEP 2855 H=2855:00 +CPU= 21.479
  12:47:31 STEP 2856 H=2856:00 +CPU= 37.239
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861_118.zip
Uploading the intermediate file: upload_file_118.zip
  12:47:48 STEP 2857 H=2857:00 +CPU= 17.184
  12:48:07 STEP 2858 H=2858:00 +CPU= 17.630
  12:48:40 STEP 2859 H=2859:00 +CPU= 32.083
  12:49:03 STEP 2860 H=2860:00 +CPU= 22.093
  12:49:25 STEP 2861 H=2861:00 +CPU= 21.293
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0576
(argv4) batchid: 982
(argv5) wuid: 12199220
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199220
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c837241c7951667d05b39bbc4812bf43 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0576_2013050100_123_982_12199220.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0576_2013050100_123_982_12199220.zip
ic_ancil_file: ic_ancil_12199220
ifsdata_file: ifsdata_12199220
climate_data_file: clim_data_12199220
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_664bc9e12ca318fb783a2a093a9e002c to: /var/lib/boinc-client/slots/6/ic_ancil_12199220.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12199220.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_429251a49c4f549b3ec89669ada7ac7f to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12199220.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12199220.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12199220.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12199220.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12199220.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12199220.xml
-- Model is restarting --
Adjusting last_iter, 2862, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12199220.xml
last_cpu_time: 59776
upload_file_number: 119
last_iter: 2855
last_upload: 10281600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0576_2013050100_123_982_12199220_0_r1054347861
The child process has been launched with process id: 12311
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:antix1vbox:1:1:12311:12311] [20230121:125048:1674305448:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:51:35 STEP 2856 H=2856:00 +CPU= 41.998
  12:51:58 STEP 2857 H=2857:00 +CPU= 21.715
  12:52:16 STEP 2858 H=2858:00 +CPU= 18.068
  12:52:43 STEP 2859 H=2859:00 +CPU= 26.066
  12:53:05 STEP 2860 H=2860:00 +CPU= 21.672
  12:53:29 STEP 2861 H=2861:00 +CPU= 22.389
  12:54:06 STEP 2862 H=2862:00 +CPU= 35.892
  12:54:28 STEP 2863 H=2863:00 +CPU= 21.526
  12:54:50 STEP 2864 H=2864:00 +CPU= 21.827
  12:55:27 STEP 2865 H=2865:00 +CPU= 34.118
  12:55:50 STEP 2866 H=2866:00 +CPU= 22.212
  12:56:12 STEP 2867 H=2867:00 +CPU= 21.352
  12:56:42 STEP 2868 H=2868:00 +CPU= 28.818
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+002868
  12:56:56 STEP 2869 H=2869:00 +CPU= 14.274
  12:57:16 STEP 2870 H=2870:00 +CPU= 19.201
  12:57:51 STEP 2871 H=2871:00 +CPU= 34.374
  12:58:14 STEP 2872 H=2872:00 +CPU= 22.230
  12:58:38 STEP 2873 H=2873:00 +CPU= 23.035
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:33:01 1538317 22285376 oifs_43r3_ps_0576_2013050100_123_982_12199220_0 9,072,000 52,166 0.0058
09 May 2023 16:32:55 1538317 22285376 oifs_43r3_ps_0576_2013050100_123_982_12199220_0 8,208,000 46,940 0.0057
09 May 2023 16:32:43 1538317 22285376 oifs_43r3_ps_0576_2013050100_123_982_12199220_0 7,344,000 41,735 0.0057
09 May 2023 16:32:37 1538317 22285376 oifs_43r3_ps_0576_2013050100_123_982_12199220_0 6,480,000 36,613 0.0057
09 May 2023 16:32:26 1538317 22285376 oifs_43r3_ps_0576_2013050100_123_982_12199220_0 5,616,000 31,508 0.0056
09 May 2023 16:32:20 1538317 22285376 oifs_43r3_ps_0576_2013050100_123_982_12199220_0 4,752,000 26,388 0.0056
09 May 2023 16:32:09 1538317 22285376 oifs_43r3_ps_0576_2013050100_123_982_12199220_0 3,888,000 21,248 0.0055
09 May 2023 16:32:03 1538317 22285376 oifs_43r3_ps_0576_2013050100_123_982_12199220_0 3,024,000 15,900 0.0053
09 May 2023 16:31:50 1538317 22285376 oifs_43r3_ps_0576_2013050100_123_982_12199220_0 1,641,600 8,176 0.0050


©2024 cpdn.org