climateprediction.net home page
Task 12787054

Task 12787054

Name hadcm3n_o65v_1900_40_007203318_2
Workunit 7401598
Created 8 Apr 2011, 10:25:56 UTC
Sent 8 Apr 2011, 10:26:04 UTC
Report deadline 8 Jul 2011, 17:53:15 UTC
Received 21 Apr 2011, 18:30:40 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 1073113
Run time 7 days 0 hours 15 min 23 sec
CPU time 5 days 5 hours 49 min 58 sec
Validate state Invalid
Credit 3,110.40
Device peak FLOPS 1.38 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr
<core_client_version>6.10.17</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1
forrtl: No space left on device
forrtl: severe (38): error during write, unit -1, file stdout
Image              PC        Routine            Line        Source             
hadcm3n_um_6.07_i  0848EB7D  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0848D975  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0845F3CF  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0841F90D  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0841F257  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  08450B99  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0844E937  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  083009BD  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0838EDB1  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0839BDF8  Unknown               Unknown  Unknown
libc.so.6          F7653BD6  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0804CB11  Unknown               Unknown  Unknown
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4568, iMonCtr=1
Model crash detected, will try to restart...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1
forrtl: No space left on device
forrtl: severe (38): error during write, unit -1, file stdout
Image              PC        Routine            Line        Source             
hadcm3n_um_6.07_i  0848EB7D  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0848D975  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0845F3CF  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0841F90D  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0841F257  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  08450B99  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0844E937  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  083009BD  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0838EDB1  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0839BDF8  Unknown               Unknown  Unknown
libc.so.6          F7583BD6  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0804CB11  Unknown               Unknown  Unknown
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4568, iMonCtr=1
Model crash detected, will try to restart...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1
forrtl: No space left on device
forrtl: severe (38): error during write, unit -1, file stdout
Image              PC        Routine            Line        Source             
hadcm3n_um_6.07_i  0848EB7D  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0848D975  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0845F3CF  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0841F90D  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0841F257  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  08450B99  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0844E937  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  083009BD  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0838EDB1  Unknown               Unknown  Unknown
hadcm3n_uSuspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1
forrtl: No space left on device
forrtl: severe (38): error during write, unit -1, file stdout
Image              PC        Routine            Line        Source             
hadcm3n_um_6.07_i  0848EB7D  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0848D975  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0845F3CF  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0841F90D  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0841F257  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  08450B99  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0844E937  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  083009BD  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0838EDB1  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0839BDF8  Unknown               Unknown  Unknown
libc.so.6          F761BBD6  Unknown               Unknown  Unknown
hadcm3n_um_6.07_i  0804CB11  Unknown               Unknown  Unknown
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=869, iMonCtr=1
Model crash detected, will try to restart...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0967a488 ***
======= Backtrace: =========
/lib32/libc.so.6(+0x6c231)[0xf74cb231]
/lib32/libc.so.6(+0x6dab8)[0xf74ccab8]
/lib32/libc.so.6(cfree+0x6d)[0xf74cfb9d]
/usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf76b9a91]
/usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf76b9aed]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib32/libc.so.6(__libc_start_main+0xe6)[0xf7475bd6]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]
======= Memory map: ========
08048000-080e3000 r-xp 00000000 fb:03 82603                              /apps/BOINC/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e3000-080e4000 rwxp 0009b000 fb:03 82603                              /apps/BOINC/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e4000-0813b000 rwxp 00000000 00:00 0 
09624000-0968b000 rwxp 00000000 00:00 0                                  [heap]
f6e00000-f6e21000 rwxp 00000000 00:00 0 
f6e21000-f6f00000 ---p 00000000 00:00 0 
f6fde000-f7459000 rwxs 00000000 fb:03 163904                             /apps/BOINC/slots/15/133330
f7459000-f745a000 ---p 00000000 00:00 0 
f745a000-f745f000 rwxp 00000000 00:00 0 
f745f000-f75b2000 r-xp 00000000 fb:00 83973                              /lib32/libc-2.11.1.so
f75b2000-f75b3000 ---p 00153000 fb:00 83973                              /lib32/libc-2.11.1.so
f75b3000-f75b5000 r-xp 00153000 fb:00 83973                              /lib32/libc-2.11.1.so
f75b5000-f75b6000 rwxp 00155000 fb:00 83973                              /lib32/libc-2.11.1.so
f75b6000-f75b9000 rwxp 00000000 00:00 0 
f75b9000-f75d6000 r-xp 00000000 fb:01 16637                              /usr/lib32/libgcc_s.so.1
f75d6000-f75d7000 r-xp 0001c000 fb:01 16637                              /usr/lib32/libgcc_s.so.1
f75d7000-f75d8000 rwxp 0001d000 fb:01 16637                              /usr/lib32/libgcc_s.so.1
f75d8000-f75fc000 r-xp 00000000 fb:00 83977                              /lib32/libm-2.11.1.so
f75fc000-f75fd000 r-xp 00023000 fb:00 83977                              /lib32/libm-2.11.1.so
f75fd000-f75fe000 rwxp 00024000 fb:00 83977                              /lib32/libm-2.11.1.so
f75fe000-f76e7000 r-xp 00000000 fb:01 16638                              /usr/lib32/libstdc++.so.6.0.13
f76e7000-f76e8000 ---p 000e9000 fb:01 16638                              /usr/lib32/libstdc++.so.6.0.13
f76e8000-f76ec000 r-xp 000e9000 fb:01 16638                              /usr/lib32/libstdc++.so.6.0.13
f76ec000-f76ed000 rwxp 000ed000 fb:01 16638                              /usr/lib32/libstdc++.so.6.0.13
f76ed000-f76f4000 rwxp 00000000 00:00 0 
f76f4000-f76f6000 r-xp 00000000 fb:00 83976                              /lib32/libdl-2.11.1.so
f76f6000-f76f7000 r-xp 00001000 fb:00 83976                              /lib32/libdl-2.11.1.so
f76f7000-f76f8000 rwxp 00002000 fb:00 83976                              /lib32/libdl-2.11.1.so
f76f8000-f76f9000 rwxp 00000000 00:00 0 
f76f9000-f770e000 r-xp 00000000 fb:00 83987                              /lib32/libpthread-2.11.1.so
f770e000-f770f000 r-xp 00014000 fb:00 83987                              /lib32/libpthread-2.11.1.so
f770f000-f7710000 rwxp 00015000 fb:00 83987                              /lib32/libpthread-2.11.1.so
f7710000-f7713000 rwxp 00000000 00:00 0 
f7713000-f7715000 rwxs 00000000 fb:03 163901                             /apps/BOINC/slots/15/boinc_mmap_file
f7715000-f7717000 rwxp 00000000 00:00 0 
f7717000-f7718000 r-xp 00000000 00:00 0                                  [vdso]
f7718000-f7734000 r-xp 00000000 fb:00 83970                              /lib32/ld-2.11.1.so
f7734000-f7735000 r-xp 0001b000 fb:00 83970                              /lib32/ld-2.11.1.so
f7735000-f7736000 rwxp 0001c000 fb:00 83970                              /lib32/ld-2.11.1.so
fff0e000-fff7f000 rw-p 00000000 00:00 0                                  [stack]
SIGABRT: abort called
Stack trace (19 frames):
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df]
[0xf7717400]
[0xf7717422]
/lib32/libc.so.6(gsignal+0x51)[0xf7489921]
/lib32/libc.so.6(abort+0x182)[0xf748cd52]
/lib32/libc.so.6(+0x6213d)[0xf74c113d]
/lib32/libc.so.6(+0x6c231)[0xf74cb231]
/lib32/libc.so.6(+0x6dab8)[0xf74ccab8]
/lib32/libc.so.6(cfree+0x6d)[0xf74cfb9d]
/usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf76b9a91]
/usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf76b9aed]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib32/libc.so.6(__libc_start_main+0xe6)[0xf7475bd6]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
21 Apr 2011 18:21:59 1073113 12787054 hadcm3n_o65v_1900_40_007203318_2 259,200 452,990 1.7476
20 Apr 2011 21:27:43 1073113 12787054 hadcm3n_o65v_1900_40_007203318_2 233,280 391,792 1.6795
20 Apr 2011 16:01:53 1073113 12787054 hadcm3n_o65v_1900_40_007203318_2 207,360 332,287 1.6025
20 Apr 2011 16:01:55 1073113 12787054 hadcm3n_o65v_1900_40_007203318_2 181,440 274,701 1.5140
20 Apr 2011 16:01:54 1073113 12787054 hadcm3n_o65v_1900_40_007203318_2 155,520 217,576 1.3990
20 Apr 2011 16:01:55 1073113 12787054 hadcm3n_o65v_1900_40_007203318_2 129,600 159,990 1.2345
20 Apr 2011 16:01:53 1073113 12787054 hadcm3n_o65v_1900_40_007203318_2 103,680 101,988 0.9837
20 Apr 2011 16:01:55 1073113 12787054 hadcm3n_o65v_1900_40_007203318_2 77,760 43,106 0.5543
20 Apr 2011 16:01:53 1073113 12787054 hadcm3n_o65v_1900_40_007203318_2 51,840 50,075 0.9660
11 Apr 2011 09:25:07 1073113 12787054 hadcm3n_o65v_1900_40_007203318_2 25,920 73,199 2.8240


©2024 climateprediction.net