[JIRA] Commented: (SSPCPP-401) IIS App Pool Crash
jlaweave@idp.protectnetwork.org (JIRA)
noreply at shibboleth.net
Tue Nov 1 21:53:26 GMT 2011
[ https://issues.shibboleth.net/jira/browse/SSPCPP-401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13683#comment-13683 ]
jlaweave at idp.protectnetwork.org commented on SSPCPP-401:
--------------------------------------------------------
Hi Scott,
I have been corresponding with Microsoft PSS regarding this issue (and passed along your observations) and they have replied with the following observations. Can you take a look at their synopsis and let me know if that helps with resolving the issue or if you can give us any idea as to what we can do on our end to help?
This issues is really becoming a problem for us. Let me know if you need any additional information or if you have any other suggestions on how to fix the problem?
//Microsoft PSS response:
I am from the IIS/ASP.NET escalation team and have taken ownership of your support case. I have been assisting Anjum behind the scenes on the issue so I am already familiar with the issue and the concerns that the developer from Shibboleth has raised. I have analyzed the latest crash dumps with the new version of the filter and the problem is still the same as we saw previously. I will first speak to the issues raised by the developer.
<Shibboleth Dev>
Here, IIS is shutting down my filter and probably unloading it from memory. But it also lets the call into the IIS server support function to send the response to the client proceed. When that happens, the static string constant I passed into the function is gone from memory by the time the call happens.
</Shibboleth Dev>
The developer is misinterpreting part of the call stack so his assertions about what is happening are incorrect. The debugger is building the call stack using exported methods since we do not have the correct symbols for the component. He is partially correct in that his filter throws an exception and goes into the catch block however the calls back into his filter are not to GetFilterVersion and TerminateFilter. These are calls back into catch block of the filter but because we do not have symbols the debugger is showing the wrong methods. Notice the offset (highlighted in yellow). These offsets show where we are in the function. These are very large offsets and the functions are not even that long so we can say for sure that these aren’t the correct methods. What is happening here is that the C++ catch code is creating the exception object to pass back into his handler and then his handler calls into ServerSupportFunction to send the response (as he has acknowledged).
0:006> kL
Child-SP RetAddr Call Site
00000000`0165bf70 00000001`8000184e filter!W3_FILTER_CONTEXT::ServerSupportFunction+0x174
00000000`0165c180 00000001`80011426 isapi_shib!TerminateFilter+0x50e
00000000`0165c1c0 00000000`74d48a40 isapi_shib!GetFilterVersion+0x2746
00000000`0165c200 00000000`74d40463 msvcr100!_CallSettingFrame+0x20
00000000`0165c230 00000000`77730c21 msvcr100!__CxxCallCatchBlock+0xeb
00000000`0165c2f0 00000001`8000cf46 ntdll!RcFrameConsolidation+0x3
00000000`0165ecd0 000007fe`f58817e4 isapi_shib!HttpFilterProc+0x3b6
00000000`0165f120 000007fe`f5881e01 filter!W3_FILTER_CONTEXT::NotifyFilters+0x178
00000000`0165f340 000007fe`f5daa185 filter!GlobalDoWork+0x351
00000000`0165f570 000007fe`f5daab24 iiscore!W3_CONTEXT::SetupStateMachine+0x685
00000000`0165fa80 000007fe`f64c10d2 iiscore!W3_MAIN_CONTEXT::OnNewRequest+0x1b0
00000000`0165fab0 000007fe`f64c109c w3dt!UL_NATIVE_REQUEST::DoWork+0x126
00000000`0165fb10 000007fe`f6391fba w3dt!OverlappedCompletionRoutine+0x1c
00000000`0165fb40 000007fe`f6392024 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x7a
00000000`0165fb90 000007fe`f63920a1 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x34
00000000`0165fbc0 00000000`774d652d w3tp!THREAD_MANAGER::ThreadManagerThread+0x61
00000000`0165fbf0 00000000`7770c521 kernel32!BaseThreadInitThunk+0xd
00000000`0165fc20 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
<Shibboleth Dev>
Something that is supposed to be set for every request by IIS is apparently not set. My logging is unfortunately not sufficient in that release to tell which piece of information it is, but the set of possibilities is small.
</Shibboleth Dev>
We have looked at the disassembly and this is not really the case. We have traced the exception to be coming from the following call in the ISAPI code.
res = stf.getServiceProvider().doAuthentication(stf);
The exception that is getting raised is matching ERROR_NO_DATA so this is giving the false reading that there is a missing header or server variable. This is something that Shibboleth support will have to help you track down. While this exception is the trigger for the AV that is crashing the process, it is a separate issue from the actual AV which I will speak to now.
The AV is happening because of an incorrect cast of a pointer to a DWORD which is causing the incorrect pointer to be used in the ServerSupportFunction call. The operation that causes the AV is seen here where we are dereferencing the pointer in register RDI. RDI is supposed to contain the pointer to the string the ISAPI filter is attempting to send in the response.
filter!W3_FILTER_CONTEXT::ServerSupportFunction+0x174:
000007fe`f588bc14 f2ae repne scas byte ptr [rdi]
As we see below the pointer value in RDI is 0000000080012798
0:006> r
rax=0000000000000000 rbx=000000000112a0f0 rcx=ffffffffffffffff
rdx=000000000165c030 rsi=000000000112b0f8 rdi=0000000080012798
rip=000007fef588bc14 rsp=000000000165bf70 rbp=000007fef588f600
r8=0000000000000100 r9=0000000080012798 r10=0000000000000008
r11=000000000165c170 r12=0000000000000001 r13=0000000080012798
r14=000000018001278c r15=0000000180011400
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010244
filter!W3_FILTER_CONTEXT::ServerSupportFunction+0x174:
000007fe`f588bc14 f2ae repne scas byte ptr [rdi]
If we dump out the contents of what’s at RDI we see
0:006> dc @rdi
00000000`80012798 ???????? ???????? ???????? ???????? ????????????????
00000000`800127a8 ???????? ???????? ???????? ???????? ????????????????
00000000`800127b8 ???????? ???????? ???????? ???????? ????????????????
00000000`800127c8 ???????? ???????? ???????? ???????? ????????????????
00000000`800127d8 ???????? ???????? ???????? ???????? ????????????????
00000000`800127e8 ???????? ???????? ???????? ???????? ????????????????
00000000`800127f8 ???????? ???????? ???????? ???????? ????????????????
00000000`80012808 ???????? ???????? ???????? ???????? ????????????????
And if we check the memory usage at that address we see it is free memory that has not been allocated yet so this is not a valid pointer.
0:006> !address @rdi
Usage: Free
Base Address: 00000000`7fff0000
End Address: 00000000`ff4a0000
Region Size: 00000000`7f4b0000
Type: 00000000
State: 00010000 MEM_FREE
Protect: 00000001 PAGE_NOACCESS
On a hunch we checked what’s at 0000000180012798 which would be the address if the value wasn’t truncated by the invalid cast. Here we see clearly the string that the developer was intending to return.
0:006> dc 0000000180012798
00000001`80012798 6e6e6f43 69746365 203a6e6f 736f6c63 Connection: clos
00000001`800127a8 430a0d65 65746e6f 542d746e 3a657079 e..Content-Type:
00000001`800127b8 78657420 74682f74 0a0d6c6d 00000a0d text/html......
00000001`800127c8 00000000 00000000 4d54483c 483c3e4c ........<HTML><H
00000001`800127d8 3e444145 5449543c 533e454c 62626968 EAD><TITLE>Shibb
00000001`800127e8 74656c6f 69462068 7265746c 72724520 oleth Filter Err
00000001`800127f8 2f3c726f 4c544954 2f3c3e45 44414548 or</TITLE></HEAD
00000001`80012808 4f423c3e 3c3e5944 533e3148 62626968 ><BODY><H1>Shibb
0:006> d
00000001`80012818 74656c6f 69462068 7265746c 72724520 oleth Filter Err
00000001`80012828 2f3c726f 003e3148 4f422f3c 3c3e5944 or</H1>.</BODY><
00000001`80012838 4d54482f 00003e4c 4d54483c 483c3e4c /HTML>..<HTML><H
00000001`80012848 3e444145 5449543c 533e454c 62626968 EAD><TITLE>Shibb
00000001`80012858 74656c6f 72452068 3c726f72 5449542f oleth Error</TIT
00000001`80012868 3c3e454c 4145482f 423c3e44 3e59444f LE></HEAD><BODY>
00000001`80012878 3e31483c 62696853 656c6f62 45206874 <H1>Shibboleth E
00000001`80012888 726f7272 31482f3c 0000003e 00000000 rror</H1>.......
0:006> d
00000001`80012898 69727473 7420676e 6c206f6f 00676e6f string too long.
00000001`800128a8 61766e69 2064696c 69727473 7020676e invalid string p
00000001`800128b8 7469736f 006e6f69 800144d0 00000001 osition..D......
00000001`800128c8 80002b10 00000001 8000f7dc 00000001 .+..............
00000001`800128d8 8000f7e2 00000001 80003d70 00000001 ........p=......
00000001`800128e8 800027f0 00000001 8000f7e8 00000001 .'..............
00000001`800128f8 80002750 00000001 8000f7ee 00000001 P'..............
00000001`80012908 8000f7f4 00000001 8000f7fa 00000001 ..............
Below highlighted in yellow is where the bug is:
DWORD WriteClientError(PHTTP_FILTER_CONTEXT pfc, const char* msg)
{
LogEvent(nullptr, EVENTLOG_ERROR_TYPE, 2100, nullptr, msg);
static const char* ctype="Connection: close\r\nContent-Type: text/html\r\n\r\n";
pfc->ServerSupportFunction(pfc,SF_REQ_SEND_RESPONSE_HEADER,"200 OK",(DWORD)ctype,0);
static const char* xmsg="<HTML><HEAD><TITLE>Shibboleth Filter Error</TITLE></HEAD><BODY>"
"<H1>Shibboleth Filter Error</H1>";
We found same problem in several other places in the vendors code. This is a problem because a DWORD will only hold a 32bit address on x86 or x64. The vendor should probably use a UINT_PTR instead
http://msdn.microsoft.com/en-us/library/aa384242(v=VS.85).aspx
You will need to take this up with the developer for a fix in their ISAPI filter. Please let me know if you have any questions or concerns.
> IIS App Pool Crash
> ------------------
>
> Key: SSPCPP-401
> URL: https://issues.shibboleth.net/jira/browse/SSPCPP-401
> Project: Shibboleth SP - C++
> Issue Type: Bug
> Security Level: Standard(Standard bug, may impact functionality but does not represent a security vulnerability )
> Components: Error Handling
> Affects Versions: 2.4.3
> Environment: Windows 2008R2 (SP1 + current patches), IIS 7.5 and .NET 3.5 framework
> Reporter: jlaweave at idp.protectnetwork.org
> Assignee: Scott Cantor
> Labels: Session
> Fix For: 2.5
>
> Original Estimate: 1 day, 4 hours
> Remaining Estimate: 1 day, 4 hours
>
> Our IIS web service keeps crashing with the following error:
> A process serving application pool 'app.Production' suffered a fatal communication error with the Windows Process Activation Service. The process id was '2576'. The data field contains the error number.
> A debug of the crash shows an access violation that appears to be caused by Shibboleth as referenced from a support request we opened with Microsoft (below):
> //From Microsoft PSS
> I have finished analyzing the dumps you sent Friday and the crash is being caused by the Shibboleth ISAPI filter. In the crashing call stack below you can see that isapi_shib is calling into our ServerSupportFunction. When I look at the instruction that is causing the AV in our code, I can see that the Shibboleth component is passing in a bad pointer for the ul1 parameter, see http://msdn.microsoft.com/en-us/library/aa503395.aspx.
> 0:030> kL
> Child-SP RetAddr Call Site
> 00000000`0b23b970 00000001`8000174e filter!W3_FILTER_CONTEXT::ServerSupportFunction+0x174
> 00000000`0b23bb80 00000001`8000c566 isapi_shib!TerminateFilter+0x45e
> 00000000`0b23bbc0 00000000`745e6f60 isapi_shib!GetFilterVersion+0x2896
> 00000000`0b23bc00 00000000`745b3b3c msvcr90!_CallSettingFrame+0x20
> 00000000`0b23bc30 00000000`77990c21 msvcr90!__CxxCallCatchBlock+0xfc
> 00000000`0b23bd00 00000001`80007ff2 ntdll!RcFrameConsolidation+0x3
> 00000000`0b23ea20 000007fe`f67e17e4 isapi_shib!HttpFilterProc+0x2d2
> 00000000`0b23eec0 000007fe`f67e1e01 filter!W3_FILTER_CONTEXT::NotifyFilters+0x178
> 00000000`0b23f0e0 000007fe`f8f6a185 filter!GlobalDoWork+0x351
> 00000000`0b23f310 000007fe`f8f6ab24 iiscore!W3_CONTEXT::SetupStateMachine+0x685
> 00000000`0b23f820 000007fe`fb4310d2 iiscore!W3_MAIN_CONTEXT::OnNewRequest+0x1b0
> 00000000`0b23f850 000007fe`fb43109c w3dt!UL_NATIVE_REQUEST::DoWork+0x126
> 00000000`0b23f8b0 000007fe`f8b01fba w3dt!OverlappedCompletionRoutine+0x1c
> 00000000`0b23f8e0 000007fe`f8b02024 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x7a
> 00000000`0b23f930 000007fe`f8b020a1 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x34
> 00000000`0b23f960 00000000`7783652d w3tp!THREAD_MANAGER::ThreadManagerThread+0x61
> 00000000`0b23f990 00000000`7796c521 kernel32!BaseThreadInitThunk+0xd
> 00000000`0b23f9c0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
> As you can see from below the AV is caused by an attempted read from 8000e6f0.
> 0:030> .exr -1
> ExceptionAddress: 000007fef67ebc14 (filter!W3_FILTER_CONTEXT::ServerSupportFunction+0x0000000000000174)
> ExceptionCode: c0000005 (Access violation)
> ExceptionFlags: 00000000
> NumberParameters: 2
> Parameter[0]: 0000000000000000
> Parameter[1]: 000000008000e6f0
> Attempt to read from address 000000008000e6f0
> The memory at the referenced address is free and is Page_Protect
> 0:030> !address 000000008000e6f0
> Usage: Free
> Base Address: 00000000`7fff0000
> End Address: 00000000`ffb00000
> Region Size: 00000000`7fb10000
> Type: 00000000
> State: 00010000 MEM_FREE
> Protect: 00000001 PAGE_NOACCESS
> 0:030> lmvm isapi_shib
> start end module name
> 00000001`80000000 00000001`80020000 isapi_shib (export symbols) isapi_shib.dll
> Loaded symbol image file: isapi_shib.dll
> Image path: D:\opt\shibboleth-sp\lib\shibboleth\isapi_shib.dll
> Image name: isapi_shib.dll
> Timestamp: Sun Jul 03 17:00:27 2011 (4E10D86B)
> CheckSum: 00026390
> ImageSize: 00020000
> File version: 2.4.3.0
> Product version: 2.4.3.0
> File flags: 0 (Mask 3F)
> File OS: 40004 NT Win32
> File type: 2.0 Dll
> File date: 00000000.00000000
> Translations: 0409.04b0
> CompanyName: UCAID
> ProductName: Shibboleth 2.4.3
> InternalName: isapi_shib
> OriginalFilename: isapi_shib.dll
> ProductVersion: 2, 4, 3, 0
> FileVersion: 2, 4, 3, 0
> PrivateBuild: 2, 4, 3, 0
> SpecialBuild: 2, 4, 3, 0
> FileDescription: Shibboleth ISAPI Filter / Extension
> LegalCopyright: Copyright © 2011 UCAID
> LegalTrademarks: Copyright © 2011 UCAID
> Comments: Copyright © 2011 UCAID
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the commits
mailing list