Apache HttpClient and multiple A records

[Brent Putman]

I had an AI from Scott through the IC per-entity metadata working group
to look into how Apache HttpClient handles (or not) multiple A records,
such as is typically used in DNS round-robin type setups.  I think this
was a question b/c it's common in CDNs.

They have a DnsResolver plugin interface with an API like so:

   InetAddress[] resolve(String host) throws UnknownHostException;

which I thought was going to be necessary to customize for something
like this.  Note it returns an array of InetAddress.

Turns out that customization isn't necessary. The default out-of-the-box
behavior of HC, even in the older 4.3.6 version we're currently on, is
that it knows about all the A records for the requested domain, and if
there is a socket connect failure, tries them in turn.  I found an
example of this [1] as it relates to timeout configuration, and I tested
it out with a domain I know has multiple A records.  It did try them in
order as specified there:

DEBUG PoolingHttpClientConnectionManager - Connection request: [route:
{}->http://zoom.us:81][total kept alive: 0; route allocated: 0 of 2;
total allocated: 0 of 20]
DEBUG PoolingHttpClientConnectionManager - Connection leased: [id:
0][route: {}->http://zoom.us:81][total kept alive: 0; route allocated: 1
of 2; total allocated: 1 of 20]
DEBUG MainClientExec - Opening connection {}->http://zoom.us:81
DEBUG HttpClientConnectionOperator - Connecting to zoom.us/52.5.0.4:81
DEBUG HttpClientConnectionOperator - Connect to zoom.us/52.5.0.4:81
timed out. Connection will be retried using another IP address
DEBUG HttpClientConnectionOperator - Connecting to zoom.us/54.86.107.98:81
DEBUG DefaultManagedHttpClientConnection - http-outgoing-0: Shutdown
connection
DEBUG MainClientExec - Connection discarded
DEBUG DefaultManagedHttpClientConnection - http-outgoing-0: Close connection
DEBUG PoolingHttpClientConnectionManager - Connection released: [id:
0][route: {}->http://zoom.us:81][total kept alive: 0; route allocated: 0
of 2; total allocated: 0 of 20]

org.apache.http.conn.ConnectTimeoutException: Connect to zoom.us:81
[zoom.us/52.5.0.4, zoom.us/54.86.107.98] failed: connect timed out


I also tried with google.com.  They only have 1 IP v4 address in DNS,
but when that failed it then tried the IP v6 address (was running this
locally on a Mac):

DEBUG MainClientExec - Opening connection {}->http://www.google.com:81
DEBUG HttpClientConnectionOperator - Connecting to
www.google.com/216.58.218.4:81
DEBUG HttpClientConnectionOperator - Connect to
www.google.com/216.58.218.4:81 timed out. Connection will be retried
using another IP address
Error from HCDEBUG HttpClientConnectionOperator - Connecting to
www.google.com/2607:f8b0:4002:806:0:0:0:2004:81
DEBUG DefaultManagedHttpClientConnection - http-outgoing-0: Shutdown
connection


At this point I don't know about failure modes other than a connect
timeout, such as a socket timeout once it's connected and reading data. 
I wouldn't necessarily expect that to be retried, but I'm not sure if
that's the main use case here.

But this is at least encouraging and I think a little surprising.  I
think our impression was that HTTP client libraries in general probably
didn't do this very well, if at all, and happily it sounds like in this
case, that was incorrect.

--Brent

[1] http://www.baeldung.com/httpclient-timeout

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/dev/attachments/20161013/cefc66e6/attachment.html>