climateprediction.net home page
Task 12835089

Task 12835089

Name hadcm3n_p7pn_1900_40_007227579_1
Workunit 7425819
Created 26 Apr 2011, 15:40:33 UTC
Sent 26 Apr 2011, 15:59:51 UTC
Report deadline 26 Jul 2011, 23:27:02 UTC
Received 18 May 2011, 15:05:36 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1061085
Run time 14 days 12 hours 26 min 53 sec
CPU time 9 days 23 hours 13 min 20 sec
Validate state Invalid
Credit 5,598.72
Device peak FLOPS 2.44 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-apple-darwin
Stderr 
<core_client_version>6.10.21</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
23:22:22 (58488): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
00:57:34 (1047): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
00:57:35 (1047): No heartbeat from core client for 30 sec - exiting
00:57:36 (1047): No heartbeat from core client for 30 sec - exiting
00:57:37 (1047): No heartbeat from core client for 30 sec - exiting
00:57:38 (1047): No heartbeat from core client for 30 sec - exiting
00:57:39 (1047): No heartbeat from core client for 30 sec - exiting
00:57:40 (1047): No heartbeat from core client for 30 sec - exiting
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Mon May 16 00:58:41 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x9808b156 ___kill + 10
1   libSystem.B.dylib                   0x9811d899 _raise + 26
2   libSystem.B.dylib                   0x981339b8 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x9802a1ae _mach_wait_until + 10
1   libSystem.B.dylib                   0x980b1809 _nanosleep + 345
2   libSystem.B.dylib                   0x980b16aa _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x980577fd __pthread_start + 345
6   libSystem.B.dylib                   0x98057682 _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca28 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x926d8000 - 0x926dbfff /usr/lib/system/libmathCommon.A.dylib
0x98029000 - 0x981d0fff /usr/lib/libSystem.B.dylib
0x99902000 - 0x99910fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=300, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Mon May 16 00:58:44 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x9808b156 ___kill + 10
1   libSystem.B.dylib                   0x9811d899 _raise + 26
2   libSystem.B.dylib                   0x981339b8 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x9802a1ae _mach_wait_until + 10
1   libSystem.B.dylib                   0x980b1809 _nanosleep + 345
2   libSystem.B.dylib                   0x980b16aa _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x980577fd __pthread_start + 345
6   libSystem.B.dylib                   0x98057682 _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca28 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x926d8000 - 0x926dbfff /usr/lib/system/libmathCommon.A.dylib
0x98029000 - 0x981d0fff /usr/lib/libSystem.B.dylib
0x99902000 - 0x99910fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=300, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Mon May 16 00:58:48 2011

Thread 0 Crashed:
CPDN Monitor - Quit request from BOINC...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Tue May 17 08:05:16 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x9808b156 ___kill + 10
1   libSystem.B.dylib                   0x9811d899 _raise + 26
2   libSystem.B.dylib                   0x981339b8 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x9802a1ae _mach_wait_until + 10
1   libSystem.B.dylib                   0x980b1809 _nanosleep + 345
2   libSystem.B.dylib                   0x980b16aa _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x980577fd __pthread_start + 345
6   libSystem.B.dylib                   0x98057682 _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca28 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x926d8000 - 0x926dbfff /usr/lib/system/libmathCommon.A.dylib
0x98029000 - 0x981d0fff /usr/lib/libSystem.B.dylib
0x99902000 - 0x99910fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=74042, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Tue May 17 08:05:20 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x9808b156 ___kill + 10
1   libSystem.B.dylib                   0x9811d899 _raise + 26
2   libSystem.B.dylib                   0x981339b8 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x9802a1ae _mach_wait_until + 10
1   libSystem.B.dylib                   0x980b1809 _nanosleep + 345
2   libSystem.B.dylib                   0x980b16aa _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x980577fd __pthread_start + 345
6   libSystem.B.dylib                   0x98057682 _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca28 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x926d8000 - 0x926dbfff /usr/lib/system/libmathCommon.A.dylib
0x98029000 - 0x981d0fff /usr/lib/libSystem.B.dylib
0x99902000 - 0x99910fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=74042, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Tue May 17 08:05:23 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x9808b156 ___kill + 10
1   libSystem.B.dylib                   0x9811d899 _raise + 26
2   libSystem.B.dylib                   0x981339b8 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x9802a1ae _mach_wait_until + 10
1   libSystem.B.dylib                   0x980b1809 _nanosleep + 345
2   libSystem.B.dylib                   0x980b16aa _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x980577fd __pthread_start + 345
6   libSystem.B.dylib                   0x98057682 _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca28 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x926d8000 - 0x926dbfff /usr/lib/system/libmathCommon.A.dylib
0x98029000 - 0x981d0fff /usr/lib/libSystem.B.dylib
0x99902000 - 0x99910fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=74042, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Tue May 17 08:05:27 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x9808b156 ___kill + 10
1   libSystem.B.dylib                   0x9811d899 _raise + 26
2   libSystem.B.dylib                   0x981339b8 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x9802a1ae _mach_wait_until + 10
1   libSystem.B.dylib                   0x980b1809 _nanosleep + 345
2   libSystem.B.dylib                   0x980b16aa _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x980577fd __pthread_start + 345
6   libSystem.B.dylib                   0x98057682 _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca28 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x926d8000 - 0x926dbfff /usr/lib/system/libmathCommon.A.dylib
0x98029000 - 0x981d0fff /usr/lib/libSystem.B.dylib
0x99902000 - 0x99910fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=74042, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
16 May 2011 02:39:07 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 466,560 853,525 1.8294
15 May 2011 00:11:27 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 440,640 806,822 1.8310
13 May 2011 23:22:10 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 414,720 759,444 1.8312
12 May 2011 20:38:29 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 388,800 712,010 1.8313
11 May 2011 19:30:25 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 362,880 664,549 1.8313
10 May 2011 19:34:50 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 336,960 617,647 1.8330
09 May 2011 06:44:44 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 311,040 570,234 1.8333
05 May 2011 22:24:02 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 285,120 523,372 1.8356
05 May 2011 03:11:17 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 259,200 475,736 1.8354
04 May 2011 08:36:15 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 233,280 428,118 1.8352
03 May 2011 13:58:03 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 207,360 381,135 1.8380
02 May 2011 09:10:01 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 181,440 333,582 1.8385
01 May 2011 10:21:37 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 155,520 286,541 1.8425
30 Apr 2011 16:09:29 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 129,600 238,654 1.8415
29 Apr 2011 21:48:56 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 103,680 192,186 1.8536
29 Apr 2011 03:40:07 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 77,760 143,845 1.8499
28 Apr 2011 10:25:57 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 51,840 95,726 1.8466
27 Apr 2011 15:53:33 1061085 12835089 hadcm3n_p7pn_1900_40_007227579_1 25,920 48,036 1.8532


©2024 cpdn.org