Shibboleth IdP Probe

Sun Aug 16 15:21:30 EDT 2015

Here's a wiki doc for a bash script that probes a sequence of
Shibboleth IdPs to determine which are based on the Shibboleth IdP V2
software:

https://wiki.shibboleth.net/confluence/x/rIBEAQ

The help file for the bash script is attached.

Federation operators, SP owners, and others can use this script to
track the installed base of Shibboleth IdP V2 over time. Currently the
percentage of V2 deployments remains quite high while the penetration
of V3, at least in production, is fairly small.

Tom
-------------- next part --------------
$ $SCRIPT_BIN/probe_shib_idps.sh -h
Shibboleth IdP Probe 0.8

Given a list of identifiers (usually entityIDs), determine which 
of those identifiers correspond to Shibboleth IdP V2 deployments. 
Non-IdPs are ignored. Non-Shibboleth IdPs are also ignored. This 
script probes Shibboleth IdP deployments only.

Usage: probe_shib_idps.sh [-hvq] [-t CONNECT_TIME -m MAX_TIME] [-f MD_FILE] [-d OUT_DIR] [ID ...]

The script optionally takes a sequence of identifiers on the command
line. If none are given, the script takes its input from stdin.

The script iterates over all input identifiers. For each identifier, 
if the corresponding entity is a Shibboleth IdP, the script sends a 
Shibboleth IdP V2 Status request to a well-known endpoint location at 
the IdP. If the HTTP response code is 200 and the response starts with 
"ok", then we know the IdP is based on the Shibboleth IdP V2 software. 
If, OTOH, the HTTP response code is 404, it is likely the IdP is based 
on the Shibboleth IdP V3 software. Other results are indeterminate.

Option -f takes an optional file argument (MD_FILE), the absolute 
path to a SAML metadata file. The script searches this file for an
entity descriptor as it processes each identifier. If this option 
is omitted, the script defaults to using a Metadata Query Server,
that is, a server that conforms to the Metadata Query Protocol.

Options:
   -h      Display this message
   -v      Write verbose messages to stdout
   -q      Run quietly (i.e., write no messages to stdout)
   -t      Time (in secs) to connect to the host
   -m      Maximum time (in secs) of a complete probe
   -f      Path to a metadata file
   -d      Path to an output directory

Option -h is mutually exclusive of all other options. Options
-q and -v are mutually exclusive of each other.

The value of the -t option is the TCP connect time, that is, the
maximum amount of time (in secs) allotted to the TCP connection.
Note that the TCP connect time includes the time it takes to do
a DNS name lookup. Since the latter is unconstrained, it may
consume all available TCP connect time, which is why the script
automatically adjusts and retries when this happens. Thus the
TCP connect time should be kept small (on the order of a few
seconds) since larger values will slow this script considerably.

The value of the -m option is the maximum total time (in secs)
allotted to each probe. A reasonable value is a few seconds
beyond the TCP connect time. Any value less than the TCP connect
time causes the script to immediately fail.

STDOUT

By default, the script outputs an abbreviated log to stdout (but 
this may be suppressed by use of the -q option). A line of 
standard output has the following space-delimited fields:

  1) code: a curl exit code
  2) output: a curl output string
  3) statusURL: the URL of the probed Status endpoint
  4) SHIBV: Shibboleth version indicator

  See the curl man page (http://linux.die.net/man/1/curl) for a
  brief description of possible exit codes. 

  The output string has the following format:

  response:999;dns:9.999;tcp:9.999;ssl:9.999;total:9.999

  The response in the output string is the HTTP response code of the
  probed web server. If the probe does not complete, the HTTP response
  will be 000. The remaining four values in the output string are times
  (in secs) computed by curl:

  dns is the elapsed time up to and including the DNS lookup
    (curl time_namelookup variable)
  tcp is the elapsed time up to and including the TCP connection
    (curl time_connect variable)
  ssl is the elapsed time up to and including the SSL exchange
    (curl time_appconnect variable) (only curl 7.19.0 and later)
  total is the total elapsed time of the probe
    (curl time_total variable)

  See the curl man page (curl --write-out option) for detailed 
  explanations of these timings.

  By definition, a probe succeeds if its exit code is 0. For our 
  purposes, a probe completely fails if its exit code is either 6 
  or 7. (Exit code 6 indicates a DNS lookup failure while code 7 
  means the host is unreachable on the network.) A probe that times
  out (exit code 28) is labeled as nonresponsive. All other exit codes 
  are regarded as indeterminate.

  The statusURL is computed from an HTTP endpoint location in metadata.
  See the log file for the actual statusURL probed by this script.

  The Shibboleth version indicator (SHIBV) takes on one of three 
  values: SHIB2, SHIB3, or SHIB?. These strings indicate Shibboleth
  IdP V2, Shibboleth IdP V3, or an unknown version of the Shibboleth
  IdP software, respectively.

  Note: This script detects Shibboleth IdP V2 deployments with high
  probability, that is, there is little or no chance of a false
  positive. However, some V2 deployments may evade this script (for
  various reasons) and thus be reported as "SHIB?". Similarly, the
  script detects Shibboleth IdP V3 deployments with reasonable
  likelihood, but be aware there is a significant chance of a false 
  positive in this case.

FILES

The script writes a number of output files. When the -d option is used, 
these output files are written to the given OUT_DIR, otherwise the files 
are written to the /tmp directory as shown below:

/tmp/out/idps-no-http-endpoint.txt

  A list of IdPs that do not expose an appropriate SAML2 HTTP endpoint 
  location in metadata. A line in the output file has the following 
  space-delimited fields:

  1) entityID: the entityID of the IdP
  2) registrarID: the registrar ID

  The entityID is the name of the IdP. An entityID is an arbitrary URI, 
  as given by the entityID XML attribute on the <md:EntityDescriptor> 
  element in SAML metadata.

  The registrarID is the name of the registrar that registered the IdP
  metadata in the first place. Note that this field may be blank in the
  log file. 

/tmp/out/idps-not-shibboleth.txt

  A list of non-Shibboleth IdPs, determined by inspecting an appropriate 
  SAML2 HTTP endpoint location in metadata. A line in the output file has 
  the following space-delimited fields:

  1) location: a SAML2 HTTP endpoint location
  2) entityID: the entityID of the IdP
  3) registrarID: the registrar ID

  The location field gives the HTTP endpoint location used to identify
  the IdP. If the location URL indicates the IdP is a Shibboleth IdP, 
  the statusURL is computed from the HTTP location on the fly.

  The entityID and the registrarID fields are the same as in the 
  previous output file. 

/tmp/out/idps-shibboleth-log.txt

  A log of each probe. Each line records the result of
  the probe of a single Shibboleth IdP. A line in the log file 
  has the following space-delimited fields:

  1) code: a curl exit code
  2) output: a curl output string
  3) statusURL: the URL of the probed Status endpoint
  4) location: a SAML2 HTTP endpoint location
  5) entityID: the entityID of the Shibboleth IdP
  6) registrarID: the registrar ID

  The code, output, and statusURL fields are the same as those printed
  to stdout.

  The location, entityID, and registrarID fields are the same as in the 
  previous output file.

/tmp/out/idps-shibboleth2-log.txt

  A log of each probe made to a Shibboleth IdP V2 deployment. If the HTTP 
  response code is 200 and the response body starts with "ok", then we 
  know the IdP is based on the Shibboleth IdP V2 software. 