IIS App Pool Crash

[Joe Lawson]

Greetings,


We're running Windows Server 2008R2/IIS 7.5 and Shibboleth v2.4.3 and, in a nutshell, we are seeing Shibboleth involved in quite a few second chance access violations (and subsequent web service crash) of late.

I invoked some diagnostic help from Microsoft PSS in September and their synopsis of the problem is included below. I have since upgraded to the current version of Shibboleth, v2.4.3, but the *same* problems continue.

Is anyone on the list aware of any similar problems or offer any insight on what we should do to resolve the problem?


Joe


//Microsoft PSS response

I have finished analyzing the dumps you sent Friday and the crash is being caused by the Shibboleth ISAPI filter. In the crashing call stack below you can see that isapi_shib is calling into our ServerSupportFunction. When I look at the instruction that is causing the AV in our code, I can see that the Shibboleth component is passing in a bad pointer for the ul1 parameter, see http://msdn.microsoft.com/en-us/library/aa503395.aspx.

0:030> kL
Child-SP RetAddr Call Site
00000000`0b23b970 00000001`8000174e filter!W3_FILTER_CONTEXT::ServerSupportFunction+0x174
00000000`0b23bb80 00000001`8000c566 isapi_shib!TerminateFilter+0x45e
00000000`0b23bbc0 00000000`745e6f60 isapi_shib!GetFilterVersion+0x2896
00000000`0b23bc00 00000000`745b3b3c msvcr90!_CallSettingFrame+0x20
00000000`0b23bc30 00000000`77990c21 msvcr90!__CxxCallCatchBlock+0xfc
00000000`0b23bd00 00000001`80007ff2 ntdll!RcFrameConsolidation+0x3
00000000`0b23ea20 000007fe`f67e17e4 isapi_shib!HttpFilterProc+0x2d2
00000000`0b23eec0 000007fe`f67e1e01 filter!W3_FILTER_CONTEXT::NotifyFilters+0x178
00000000`0b23f0e0 000007fe`f8f6a185 filter!GlobalDoWork+0x351
00000000`0b23f310 000007fe`f8f6ab24 iiscore!W3_CONTEXT::SetupStateMachine+0x685
00000000`0b23f820 000007fe`fb4310d2 iiscore!W3_MAIN_CONTEXT::OnNewRequest+0x1b0
00000000`0b23f850 000007fe`fb43109c w3dt!UL_NATIVE_REQUEST::DoWork+0x126
00000000`0b23f8b0 000007fe`f8b01fba w3dt!OverlappedCompletionRoutine+0x1c
00000000`0b23f8e0 000007fe`f8b02024 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x7a
00000000`0b23f930 000007fe`f8b020a1 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x34
00000000`0b23f960 00000000`7783652d w3tp!THREAD_MANAGER::ThreadManagerThread+0x61
00000000`0b23f990 00000000`7796c521 kernel32!BaseThreadInitThunk+0xd
00000000`0b23f9c0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

As you can see from below the AV is caused by an attempted read from 8000e6f0.
0:030> .exr -1
ExceptionAddress: 000007fef67ebc14 (filter!W3_FILTER_CONTEXT::ServerSupportFunction+0x0000000000000174)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 000000008000e6f0
Attempt to read from address 000000008000e6f0

The memory at the referenced address is free and is Page_Protect
0:030> !address 000000008000e6f0
Usage: Free
Base Address: 00000000`7fff0000
End Address: 00000000`ffb00000
Region Size: 00000000`7fb10000
Type: 00000000
State: 00010000 MEM_FREE
Protect: 00000001 PAGE_NOACCESS

0:030> lmvm isapi_shib
start end module name
00000001`80000000 00000001`8001b000 isapi_shib (export symbols) isapi_shib.dll
Loaded symbol image file: isapi_shib.dll
Image path: D:\opt\shibboleth-sp\lib\shibboleth\isapi_shib.dll
Image name: isapi_shib.dll
Timestamp: Tue Dec 01 12:23:50 2009 (4B155126)
CheckSum: 00023BC8
ImageSize: 0001B000
File version: 2.3.1.0
Product version: 2.3.1.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Internet2
ProductName: Shibboleth 2.3.1
InternalName: isapi_shib
OriginalFilename: isapi_shib.dll
ProductVersion: 2, 3, 1, 0
FileVersion: 2, 3, 1, 0
PrivateBuild: 2, 3, 1, 0
SpecialBuild: 2, 3, 1, 0
FileDescription: Shibboleth ISAPI Filter / Extension
LegalCopyright: Copyright © 2009 Internet2
LegalTrademarks: Copyright © 2009 Internet2
Comments: Copyright © 2009 Internet2

It looks like you are a few versions back on the Shibboleth component so my recommendation would be to upgrade to the latest stable build and to see if they may have already fixed the issue. If the newest version doesn’t resolve the issue then you will need to file a bug with the group who maintains the Shibboleth product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://shibboleth.net/pipermail/users/attachments/20111026/5d10ecef/attachment-0001.html