[JIRA] Created: (SSPCPP-401) IIS App Pool Crash

Fri Oct 28 17:54:26 BST 2011

IIS App Pool Crash
------------------

                 Key: SSPCPP-401
                 URL: https://issues.shibboleth.net/jira/browse/SSPCPP-401
             Project: Shibboleth SP - C++
          Issue Type: Bug
      Security Level: Standard (Standard bug, may impact functionality but does not represent a security vulnerability )
          Components: Error Handling
    Affects Versions: 2.4.3
         Environment: Windows 2008R2 (SP1 + current patches), IIS 7.5 and .NET 3.5 framework
            Reporter: jlaweave at idp.protectnetwork.org
            Assignee: Scott Cantor

Our IIS web service keeps crashing with the following error:
A process serving application pool 'app.Production' suffered a fatal communication error with the Windows Process Activation Service. The process id was '2576'. The data field contains the error number.

A debug of the crash shows an access violation that appears to be caused by Shibboleth as referenced from a support request we opened with Microsoft (below):

//From Microsoft PSS

I have finished analyzing the dumps you sent Friday and the crash is being caused by the Shibboleth ISAPI filter. In the crashing call stack below you can see that isapi_shib is calling into our ServerSupportFunction. When I look at the instruction that is causing the AV in our code, I can see that the Shibboleth component is passing in a bad pointer for the ul1 parameter, see http://msdn.microsoft.com/en-us/library/aa503395.aspx.

0:030> kL
Child-SP          RetAddr           Call Site
00000000`0b23b970 00000001`8000174e filter!W3_FILTER_CONTEXT::ServerSupportFunction+0x174
00000000`0b23bb80 00000001`8000c566 isapi_shib!TerminateFilter+0x45e
00000000`0b23bbc0 00000000`745e6f60 isapi_shib!GetFilterVersion+0x2896
00000000`0b23bc00 00000000`745b3b3c msvcr90!_CallSettingFrame+0x20
00000000`0b23bc30 00000000`77990c21 msvcr90!__CxxCallCatchBlock+0xfc
00000000`0b23bd00 00000001`80007ff2 ntdll!RcFrameConsolidation+0x3
00000000`0b23ea20 000007fe`f67e17e4 isapi_shib!HttpFilterProc+0x2d2
00000000`0b23eec0 000007fe`f67e1e01 filter!W3_FILTER_CONTEXT::NotifyFilters+0x178
00000000`0b23f0e0 000007fe`f8f6a185 filter!GlobalDoWork+0x351
00000000`0b23f310 000007fe`f8f6ab24 iiscore!W3_CONTEXT::SetupStateMachine+0x685
00000000`0b23f820 000007fe`fb4310d2 iiscore!W3_MAIN_CONTEXT::OnNewRequest+0x1b0
00000000`0b23f850 000007fe`fb43109c w3dt!UL_NATIVE_REQUEST::DoWork+0x126
00000000`0b23f8b0 000007fe`f8b01fba w3dt!OverlappedCompletionRoutine+0x1c
00000000`0b23f8e0 000007fe`f8b02024 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x7a
00000000`0b23f930 000007fe`f8b020a1 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x34
00000000`0b23f960 00000000`7783652d w3tp!THREAD_MANAGER::ThreadManagerThread+0x61
00000000`0b23f990 00000000`7796c521 kernel32!BaseThreadInitThunk+0xd
00000000`0b23f9c0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

As you can see from below the AV is caused by an attempted read from 8000e6f0. 

0:030> .exr -1
ExceptionAddress: 000007fef67ebc14 (filter!W3_FILTER_CONTEXT::ServerSupportFunction+0x0000000000000174)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000000
   Parameter[1]: 000000008000e6f0
Attempt to read from address 000000008000e6f0

The memory at the referenced address is free and is Page_Protect

0:030> !address 000000008000e6f0

Usage:                  Free
Base Address:           00000000`7fff0000
End Address:            00000000`ffb00000
Region Size:            00000000`7fb10000
Type:                   00000000    
State:                  00010000    MEM_FREE
Protect:                00000001    PAGE_NOACCESS

0:030> lmvm isapi_shib
start             end                 module name
00000001`80000000 00000001`80020000   isapi_shib   (export symbols)       isapi_shib.dll
    Loaded symbol image file: isapi_shib.dll
    Image path: D:\opt\shibboleth-sp\lib\shibboleth\isapi_shib.dll
    Image name: isapi_shib.dll
    Timestamp:        Sun Jul 03 17:00:27 2011 (4E10D86B)
    CheckSum:         00026390
    ImageSize:        00020000
    File version:     2.4.3.0
    Product version:  2.4.3.0
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      UCAID
    ProductName:      Shibboleth 2.4.3
    InternalName:     isapi_shib
    OriginalFilename: isapi_shib.dll
    ProductVersion:   2, 4, 3, 0
    FileVersion:      2, 4, 3, 0
    PrivateBuild:     2, 4, 3, 0
    SpecialBuild:     2, 4, 3, 0
    FileDescription:  Shibboleth ISAPI Filter / Extension
    LegalCopyright:   Copyright © 2011 UCAID
    LegalTrademarks:  Copyright © 2011 UCAID
    Comments:         Copyright © 2011 UCAID

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira