climateprediction.net home page
Task 12746489

Task 12746489

Name hadcm3n_o5n0_1900_40_007202639_0
Workunit 7400919
Created 28 Mar 2011, 14:14:22 UTC
Sent 29 Mar 2011, 18:11:03 UTC
Report deadline 29 Jun 2011, 1:38:14 UTC
Received 26 Apr 2011, 21:10:15 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 984843
Run time 10 days 11 hours 24 min 18 sec
CPU time 9 days 2 hours 6 min 24 sec
Validate state Invalid
Credit 6,220.80
Device peak FLOPS 3.23 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-apple-darwin
Stderr 
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
12:34:55 (1283): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
OPEN:  File Creation Failed: No space left on device
OPEN:  Unable to Open File dataout/o5n0ka.dac1820 for Read/Write

Model crashed: DUMPCTL : Fail to open output dump - may already exist                                                                                                                                                                                                          tmp/pipe_dummy                                                                  2048    
forrtl: No space left on device
forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_o5n0_1900_40_007202639/dataout/stdout_um.txt
Image              PC        Routine            Line        Called boinc_finish
15:02:29 (9570): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
22:47:28 (640): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
22:47:29 (640): No heartbeat from core client for 30 sec - exiting
22:47:30 (640): No heartbeat from core client for 30 sec - exiting

Model crashed: TEMPHIST: Write ERROR on history file for namelistNLCFILES                                                                                                                                                                                                      tmp/pipe_dummy                                                                  2048    
23:06:46 (8858): No heartbeat from core client for 30 sec - exiting
23:06:48 (8858): No heartbeat from core client for 30 sec - exiting
23:06:49 (8858): No heartbeat from core client for 30 sec - exiting
23:06:50 (8858): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Tue Apr 26 23:07:56 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x970cf156 __kill + 10
1   libSystem.B.dylib                   0x97161899 raise + 26
2   libSystem.B.dylib                   0x971779b8 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x9706e1ae mach_wait_until + 10
1   libSystem.B.dylib                   0x970f5809 nanosleep + 345
2   libSystem.B.dylib                   0x970f56aa usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x9709b7fd _pthread_start + 345
6   libSystem.B.dylib                   0x9709b682 thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc9e8 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x90996000 - 0x909a4fff /usr/lib/libz.1.dylib
0x92748000 - 0x9274bfff /usr/lib/system/libmathCommon.A.dylib
0x9706d000 - 0x97214fff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9487, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Tue Apr 26 23:08:02 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x970cf156 __kill + 10
1   libSystem.B.dylib                   0x97161899 raise + 26
2   libSystem.B.dylib                   0x971779b8 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x9706e1ae mach_wait_until + 10
1   libSystem.B.dylib                   0x970f5809 nanosleep + 345
2   libSystem.B.dylib                   0x970f56aa usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x9709b7fd _pthread_start + 345
6   libSystem.B.dylib                   0x9709b682 thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc9e8 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x90996000 - 0x909a4fff /usr/lib/libz.1.dylib
0x92748000 - 0x9274bfff /usr/lib/system/libmathCommon.A.dylib
0x9706d000 - 0x97214fff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9487, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Tue Apr 26 23:08:05 2011

Thread 0 Crashed:
23:08:27 (9487): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Tue Apr 26 23:08:57 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x970cf156 __kill + 10
1   libSystem.B.dylib                   0x97161899 raise + 26
2   libSystem.B.dylib                   0x971779b8 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x9706e1ae mach_wait_until + 10
1   libSystem.B.dylib                   0x970f5809 nanosleep + 345
2   libSystem.B.dylib                   0x970f56aa usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x9709b7fd _pthread_start + 345
6   libSystem.B.dylib                   0x9709b682 thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc9e8 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x90996000 - 0x909a4fff /usr/lib/libz.1.dylib
0x92748000 - 0x9274bfff /usr/lib/system/libmathCommon.A.dylib
0x9706d000 - 0x97214fff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9831, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Tue Apr 26 23:09:01 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x970cf156 __kill + 10
1   libSystem.B.dylib                   0x97161899 raise + 26
2   libSystem.B.dylib                   0x971779b8 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x9706e1ae mach_wait_until + 10
1   libSystem.B.dylib                   0x970f5809 nanosleep + 345
2   libSystem.B.dylib                   0x970f56aa usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x9709b7fd _pthread_start + 345
6   libSystem.B.dylib                   0x9709b682 thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc9e8 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x90996000 - 0x909a4fff /usr/lib/libz.1.dylib
0x92748000 - 0x9274bfff /usr/lib/system/libmathCommon.A.dylib
0x9706d000 - 0x97214fff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9831, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Tue Apr 26 23:09:03 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x970cf156 __kill + 10
1   libSystem.B.dylib                   0x97161899 raise + 26
2   libSystem.B.dylib                   0x971779b8 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x9706e1ae mach_wait_until + 10
1   libSystem.B.dylib                   0x970f5809 nanosleep + 345
2   libSystem.B.dylib                   0x970f56aa usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x9709b7fd _pthread_start + 345
6   libSystem.B.dylib                   0x9709b682 thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc9e8 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x90996000 - 0x909a4fff /usr/lib/libz.1.dylib
0x92748000 - 0x9274bfff /usr/lib/system/libmathCommon.A.dylib
0x9706d000 - 0x97214fff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9831, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Tue Apr 26 23:09:06 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x970cf156 __kill + 10
1   libSystem.B.dylib                   0x97161899 raise + 26
2   libSystem.B.dylib                   0x971779b8 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x9706e1ae mach_wait_until + 10
1   libSystem.B.dylib                   0x970f5809 nanosleep + 345
2   libSystem.B.dylib                   0x970f56aa usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x9709b7fd _pthread_start + 345
6   libSystem.B.dylib                   0x9709b682 thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc9e8 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x90996000 - 0x909a4fff /usr/lib/libz.1.dylib
0x92748000 - 0x9274bfff /usr/lib/system/libmathCommon.A.dylib
0x9706d000 - 0x97214fff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9831, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
22 Apr 2011 07:39:19 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 518,400 754,323 1.4551
21 Apr 2011 05:58:44 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 492,480 722,302 1.4667
20 Apr 2011 19:59:31 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 466,560 687,384 1.4733
20 Apr 2011 15:53:10 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 440,640 642,863 1.4589
20 Apr 2011 15:53:10 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 414,720 602,786 1.4535
20 Apr 2011 15:53:10 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 388,800 561,688 1.4447
20 Apr 2011 15:53:10 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 362,880 519,453 1.4315
20 Apr 2011 15:53:10 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 336,960 478,526 1.4201
20 Apr 2011 15:53:10 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 311,040 438,761 1.4106
13 Apr 2011 10:43:57 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 285,120 400,628 1.4051
11 Apr 2011 05:36:26 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 259,200 356,063 1.3737
10 Apr 2011 19:47:14 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 233,280 323,281 1.3858
10 Apr 2011 09:59:57 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 207,360 290,263 1.3998
10 Apr 2011 00:14:41 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 181,440 257,278 1.4180
09 Apr 2011 14:29:39 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 155,520 224,235 1.4418
09 Apr 2011 04:46:02 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 129,600 191,198 1.4753
08 Apr 2011 18:55:22 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 103,680 157,772 1.5217
08 Apr 2011 05:18:48 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 77,760 122,727 1.5783
07 Apr 2011 18:10:54 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 51,840 84,877 1.6373
06 Apr 2011 08:49:23 984843 12746489 hadcm3n_o5n0_1900_40_007202639_0 25,920 39,027 1.5057


©2024 cpdn.org