Task 16819032

Name	hadam3p_eu_e3nm_2013_1_008846525_1
Workunit	8992454
Created	20 Jul 2014, 22:44:41 UTC
Sent	20 Jul 2014, 23:08:39 UTC
Report deadline	3 Jul 2015, 4:28:39 UTC
Received	22 Jul 2014, 16:48:30 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1298482
Run time	1 days 5 hours 38 min 58 sec
CPU time	1 days 4 hours 36 min 9 sec
Validate state	Invalid
Credit	995.30
Device peak FLOPS	3.52 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Europe v6.09 windows_intelx86
Stderr	<core_client_version>7.2.26</core_client_version> <![CDATA[ <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11428, selfPID=11428, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9604, selfPID=9604, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12240, selfPID=12240, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x74AAC42D Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x74AAC42D Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 07/22/14 12:33:56 Install Directory : C:\Program Files (x86)\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : LoadLibraryA( C:\Program Files (x86)\BOINC\\dbghelp.dll ): GetLastError = 126 Loaded Library : dbghelp.dll LoadLibraryA( C:\Program Files (x86)\BOINC\\symsrv.dll ): GetLastError = 126 LoadLibraryA( symsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files (x86)\BOINC\\srcsrv.dll ): GetLastError = 126 LoadLibraryA( srcsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll ****************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 07/22/14 12:33:56 Install Directory : C:\Program Files (x86)\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : LoadLibraryA( C:\Program Files (x86)\BOINC\\dbghelp.dll ): GetLastError = 126 Loaded Library : dbghelp.dll LoadLibraryA( C:\Program Files (x86)\BOINC\\symsrv.dll ): GetLastError = 126 LoadLibraryA( symsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files (x86)\BOINC\\srcsrv.dll ): GetLastError = 126 LoadLibraryA( srcsrv.dll ): GetLastError = 126 LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadam3p_eu_e3nm_2013_1_008846525;C:\ProgramData\BOINC\projects\climateprediction.net Debugger EnginM : 4.0.5.0 oymboal S00eb0000 h: C:\ProgramData\BOINC\projects\climta\climateprediction.netm3p_eam3p_eu_um_6.09008846525;inteC:\ProxramDexe (-nosymbols- Symclbmatepredictio n.n Lin d PDB Filename : d:\cpdn\cpdnboinc\cpModLoad: 00b50000 008d3000 Cc\orogramDatac.uk_cCppdonects\climatu_um_6.09_windows_intelx86.pdb 6. 9_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : d:\cpdn\cpdnboinc\cpdnprecis\testMprojects\reqs.comlab.ox.00aCc.uk_cpdn\SyadrmW34\ntdll._6.l (6.1nd7601_intelx86.pdb 1 247) (-exported- Symbols Loaded) Linked PDB Filename : wntdll.pdb M dLoad: 76ed0000 00180000 C:\1Wi.d7600.ys6W8O (w4inn_rdm.0l90ll7(1.1-.27605.5) 7) (-e Cxmpany N-me : MScrosoft yormorbtion s P odoadud) c Ltnked PD Filenam e : wntd : licrlso.tp di n doFsi Oe Vpreiont ng y tem : 6o.1.d7u600.16385 (wicnt_rVems090i13-12o5) n C p an : y.1.7 00.1638me 5 : Microsoft Corporation Product Name : Microsoft® Windows® OpMoadLoad:yste75ccm00 0 0 Produc0 1ersion 1 0: 6.1.0600016 3:8Win ows\syswow64\kernel32.dll (6.1.7601.18409) (-exported- Symbols Loaded) Linked PDB Filename : wkernel32.pdb File Versi5cc0000 : 6.1.7601.d1ws\syswow64\ker_gelr.1403l03-2144) 1. 8409mp) ( Nnosemb b: sMLiaded) ofinked PDB File n eroduct Namnel 2 pdb : Ficrosoert® Windows® Opera6.1.7601.1m40 Product Version03-214 ) Com01any Name : Microsoft Corporation Product Name : Microsoft® Windows® MpodLting 74ata00 00 000duct Versioindow s\s 6.w.7601.KERNEL B ASE.dll (6.1.7601.18409) (-exported- Symbols Loaded) Linked PDB FilenModL ad:w74aa0000 e.00470 0 Filie Vesrsyswow64\KERNELBASE1.dl6 (1.184060 (win70p) (-nr.1y4030s- 2Symbo s o dCo) an L Nkeme DB F l name : wkernelbase.opdb File Version Name 6.1.76: .Mi015oswon7sp Windows® Operating Systemp n N me : Microsoft Corporati1.18 409 P oduct Name : Microsoft® Windows® OperatinM System 7691000d uct 00000 C:\ W nd : s\.1yswow64\USER3 2 .dll (6.1.7601.17514) (-exported- SymboodL Loade76 0000 00100000 C:\WBn Filename : Uwuser3dll (6. 1.7601.17514) (-nosymbols- Symbols Loaded)14 (wiLn7ksp1_rB Filename : wu Compan Name : : C6o1r7601.17514 Product Name 9 50) M icrosoft ® Windo s® Operating oft temp ion P Product Name : Micros.1ft® Wi n dows® Operating System Product VeMsodLoad: 76a40000 600090000 C \ Windows\ of thwe Proce2.dlSt(6ist.cs6: 18 5) I/O Operati- Symbols ersaded Re L Write:P0, Othernam e O wgdi32.pdb File V Rerd: 0, Write: 0, Other 0 - Paged Poolsion - Qu : 6.1.7601.age: 0(, QuotaPeakPagedPoolU33)e: 0 QuompaNonPagedP l e 0, QicrtosoakN CoPporation Produ c t Vamrtu a Memory Uicroesoft VirtualSize: 0, Ping SystemSi z e: 0 - Pagersion U : agefileUsage: 5 PeakPagefileUsage: 0 - Working Set Size - WorkingSetSize: 0, PeakWorMiogdLoad:e 75 e20000 ltCount: C: \ Windows\mpyof thread ID 1o676 \LPK.dll (6t.1.7zed1.18*** - Infor(a-exp or ted- S: BaolsPrioaityd: Normainked PDB Fi:lNonrmel, : wrlep Tipdb 000000, User VTirsion0.0 0 , W6.1.Time: 0.00070001 . 1- 77 (winled _gdr.1306n R-153rd - easCompany t Of MemorN (C++ Exceptione (0xe 0 6 7363) at address tx7CorAC4rat2on Product Name : Microsoft® Windows® Operating System Product Ver-ion giste r - : 6.1.7601.18177 efde000 ecx=00000003 edx=00000000 esi=0Mfd4e48 ed75000000000 e0p=9d000c42d esnp=1190yfw8cc4ebUS=P10.dll (s=0023 ss1=08454 d(-e02brted- Sy2b fs=L0053 s=002b DB Filename0000206s 0.pdb t Fck e Cersion P RetAd : 1.0gs to Child 454 (win7sp11190f91c 00e71793 e06d7363 00000001 00000003 1190f948 KERNE_BgdE!+0x0 r .140424-1533) 19 0f9ompany 00emee b 1 : Microsoft Corpdrc io0n09 haoduct Na_um_6.09_ wind Microsoft(R) Un iscribe Unico11 script processor 00Product85bersion 0eab3f4 hadrm3p_eu_um_ .09_windows_inte!+0x0 1190f9f8 M0od0Load: 76090000 000ac0 0 0271cf0do79\s5ed6ow h4\msvcrte._dll .09_w.7601.17744) (-exp rted- Symbols 190fa40 7 5cd338a 7efde00Fil1190fa : msvcrf72 7e Filadrm3siou_um : 7.0.7601.17744 ( in7sp1_gd11901215 76f09 72 7eCdo00p 6e8acfme 00000 0 0 : Microsoft Corporati n Product Na0fa f45: Microsoaft® deindowffffpeff 76f97428 ntd ll!RtlInuct lizrExcon o :n 7.0.7 601.177411 0faa4 00000000 00e708ea 7efde000 00000000 00000000 ntdll!RtlInitiMlizLExcd: io6170000 000a 0 000 C:\Wiof threadwow 64\ADVAPI32e: In(ti.1.7601.1847) ( -exported- Symb otasus:oaded) r o Ltinkeod PlDB Filenta: N mal,a, Kern32.Timb : 0.00000V,erseion : 6.1.W600t.Time: 0.00000t m .090713-125Registers - eax=005) Company Na00 0 000000Miedx=so0t000000ratii=n3 Prod euict0000000 eiN=76eemed 9 : eicrp=osaf5®f 7a8 ows=®a52f 1Op ce=r02a ns Syst=m 02 bProdu t V rsiod s : 6.1.70002b6 385 s =002b fs=0053 gs=002b odL ad: 5 8 000 efl=00100020 C :- Callndows\SysaOW64\skcho tCdllh(6ld.EB00P1 3RetAd) (rgs tp Chiold ted- Sy3bols2f81Lo74aded4a5) 0 00 Lin0ed P000F0000l0e 3a5amfe2c 00 fb f ntdll!Z DslayExeccuost.pdiob x0 3File V5er2fsio20 n0 : 6.10760b2f4 385 0win7_rta5.090713-12d338a 00000000 KEame : M icrosoft Corporation Produ 00000000 a 78 Microsoft00000000 ® Oprmrating System Pors_inte!+0xsio n a5 6.18.7600.172 0 00 0000 4548cdd9 00000000 0000ModLoad: 749a0000 000f0000 C:\Windows\sys0ow64\kPCRne432d!l+ x0 1.a52f878 76f09f45 00dfb2e0 00000000 ffffffff 76f9747601.18205!RtlIn)t a(-exExorted- Syambols Lo aded) L2fn89d PDB Filename : wrpcrt4.pdb File Version 0 : 6 0.700000 385 (win0 0000009071000005) Company tdll! tlInitializeExceptionCharin+0x0 Debut Nasse crosoft® Windows® Operating Syst e n Product deow Name :6.1 .7 0 .1638o5 ModLoad: 748 f0000i00060000 C:\Windows \syswow6o4\SspiCli.dll (6 1Exit1ng..496 ) (-exported- Symbols Loaded) Linked PDB Filename : wsspicli.pdb File Version : 6.1.7601.18496 (win7sp1_gdr.140605-0343) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7601.18496 ModLoad: 748e0000 0000c000 C:\Windows\syswow64\CRYPTBASE.dll (6.1.7600.16385) (-nosymbols- Symbols Loaded) Linked PDB Filename : cryptbase.pdb File Version : 6.1.7600.16385 (win7_rtm.090713-1255) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7600.16385 ModLoad: 19ac0000 000c8000 C:\Windows\system32\OPENGL32.dll (6.1.7600.16385) (-nosymbols- Symbols Loaded) Linked PDB Filename : opengl32.pdb File Version : 6.1.7600.16385 (win7_rtm.090713-1255) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7600.16385 ModLoad: 00080000 00022000 C:\Windows\system32\GLU32.dll (6.1.7600.16385) (-nosymbols- Symbols Loaded) Linked PDB Filename : glu32.pdb File Version : 6.1.7600.16385 (win7_rtm.090713-1255) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7600.16385 ModLoad: 728d0000 000e7000 C:\Windows\system32\DDRAW.dll (6.1.7600.16385) (-nosymbols- Symbols Loaded) Linked PDB Filename : ddraw.pdb File Version : 6.1.7600.16385 (win7_rtm.090713-1255) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7600.16385 ModLoad: 729c0000 00006000 C:\Windows\system32\DCIMAN32.dll (6.1.7601.18177) (-nosymbols- Symbols Loaded) Linked PDB Filename : dciman32.pdb File Version : 6.1.7601.18177 (win7sp1_gdr.130605-1534) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7601.18177 ModLoad: 75ee0000 0019d000 C:\Windows\syswow64\SETUPAPI.dll (6.1.7601.17514) (-nosymbols- Symbols Loaded) Linked PDB Filename : setupapi.pdb File Version : 6.1.7600.16385 (win7_rtm.090713-1255) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7600.16385 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 0, Write: 0, Other 0 - I/O Transfers Counters - Read: 0, Write: 0, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0 QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0 - Virtual Memory Usage - VirtualSize: 0, PeakVirtualSize: 0 - Pagefile Usage - PagefileUsage: 0, PeakPagefileUsage: 0 - Working Set Size - WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0 * Dump of thread ID 16828 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x74AAC42D - Registers - eax=12fbf690 ebx=00000000 ecx=00000003 edx=00000000 esi=01366b48 edi=00321c7c eip=74aac42d esp=12fbf690 ebp=12fbf6e0 cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000216 - Callstack - ChildEBP RetAddr Args to Child 12fbf6e0 011ff613 e06d7363 00000001 00000003 12fbf70c KERNELBASE!RaiseException+0x0 12fbf718 011fc835 12fbf728 0129a130 0126b7bc 012963f4 hadam3p_eu_um_6.09_windows_inte!+0x0 12fbf734 00eb9bc5 00fd9f80 00000000 00000000 7efde000 hadam3p_eu_um_6.09_windows_inte!+0x0 12fbf7a8 011fe638 00000009 00321c58 00321db8 7aaee99b hadam3p_eu_um_6.09_windows_inte!+0x0 12fbf7f0 75cd338a 7efde000 12fbf83c 76f09f72 7efde000 hadam3p_eu_um_6.09_windows_inte!+0x0 12fbf7fc 76f09f72 7efde000 6de1cd9d 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 12fbf83c 76f09f45 011fe68f 7efde000 ffffffff 76f97428 ntdll!RtlInitializeExceptionChain+0x0 12fbf854 00000000 011fe68f 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 * Dump of thread ID 17356 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 - Registers - eax=01189340 ebx=00000000 ecx=00000000 edx=00000000 esi=3cb4f888 edi=00000000 eip=76eefd91 esp=3cb4f844 ebp=3cb4f8ac cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246 - Callstack - ChildEBP RetAddr Args to Child 3cb4f8ac 74ab44a5 00000064 00000000 3cb4f8c8 01189354 ntdll!ZwDelayExecution+0x0 3cb4f8bc 01189354 00000064 3cb4f8d4 75cd338a 00000000 KERNELBASE!Sleep+0x0 3cb4f8c8 75cd338a 00000000 3cb4f914 76f09f72 00000000 hadam3p_eu_um_6.09_windows_inte!+0x0 3cb4f8d4 76f09f72 00000000 43aeccb5 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 3cb4f914 76f09f45 01189340 00000000 ffffffff 76f97428 ntdll!RtlInitializeExceptionChain+0x0 3cb4f92c 00000000 01189340 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 * Debug Message Dump * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12456, selfPID=10760, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_eu_e3nm_2013_1_008846525_1_6.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_e3nm_2013_1_008846525_1_7.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_e3nm_2013_1_008846525_1_8.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_e3nm_2013_1_008846525_1_9.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_e3nm_2013_1_008846525_1_10.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_e3nm_2013_1_008846525_1_11.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_e3nm_2013_1_008846525_1_12.zip</file_name> <error_code>-161</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Jul 2014 13:08:18	1298482	16819032	hadam3p_eu_e3nm_2013_1_008846525_1	57,696	92,683	1.6064
22 Jul 2014 13:08:18	1298482	16819032	hadam3p_eu_e3nm_2013_1_008846525_1	46,176	74,090	1.6045
22 Jul 2014 13:08:18	1298482	16819032	hadam3p_eu_e3nm_2013_1_008846525_1	34,656	55,431	1.5995
21 Jul 2014 11:09:20	1298482	16819032	hadam3p_eu_e3nm_2013_1_008846525_1	23,136	37,129	1.6048
21 Jul 2014 06:01:56	1298482	16819032	hadam3p_eu_e3nm_2013_1_008846525_1	11,616	18,509	1.5934