1. Introduction

The HTTP Requests library provides a clean and simplified interface to make HTTP requests.

The spirit of this library is to make it trivial to do easy things with HTTP requests. It can accomplish the most common use cases for an HTTP client, but does not aim to be a complete HTTP client implementation.

At its core, the library defines a framework for making requests. A built-in implementation of this framework based on Java’s built-in HTTP classes is included to make the library extremely lightweight, with its only external dependencies being groovy and SLF4J. There are also several implementation providers available that use popular HTTP client libraries, such as Jersey and Apache HTTP Components Client. When one of these providers is used, the library serves as an adapter, making the use of these popular libraries much easier and your code more concise.

1.1. Features

At a high level, the library provides the following features, and more.

  • Configure and submit HTTP requests without knowing how to use the underlying HTTP client implementation.

  • Supports common request features, including query parameters, headers, and request entities.

  • Supports the following HTTP methods:

    • GET

    • POST

    • PUT

    • DELETE

    • OPTIONS

    • TRACE

    • HEAD

  • Optionally disable SSL trust and hostname validation.

  • Log HTTP conversations.

  • Ability to create filters that affect the HTTP request during the various phases of its lifecycle.

  • Ability to create custom entity converters for custom object types.

  • Built-in entity converters for common object types.

  • HTTP status exceptions for non-successful response status codes available.

  • Requests can be built using a builder syntax or Groovy DSL closures.

2. Installing

The library is split into several module libraries, which can all be found on jCenter. The library contains a core module, which contains the bulk of the code and logic provided by the library. There are also implementation-specific libraries that provide adapters to popular HTTP client libraries, called providers. The table below lists out all of the modules that exist for the HTTP Requests library and what their purposes are.

Table 1. HTTP Requests Library Modules
Module Purpose

http-requests

A bill-of-materials POM for all libraries.

http-requests-core

Contains the core logic and interfaces that implementation modules will use, in addition to a built-in HTTP client implementation.

http-requests-jersey1

Provides a Jersey 1.x implementation of the HTTP requests library.

http-requests-jersey2

Provides a Jersey 2.x implementation of the HTTP requests library.

http-requests-httpcomponents-client

Provides an Apache HttpComponents Client implementation of the HTTP requests library.

http-requests-test

Contains the functional test cases that validate the functionality of an implementation library.

http-requests-grails

A Grails 3.x plugin that sets up and exposes an HttpClientFactory bean.

All modules are deployed under the group com.budjb.

As an example, using gradle and opting to use the Jersey 1.x implementation, a dependency should be added like below:

build.gradle
dependencies {
    compile "com.budjb:http-requests-jersey1:1.0.2"
}

Additionally, the bill-of-materials POM is available to keep dependencies in sync without the need to specify their version numbers.

3. Usage

Requests are made using the HttpClientFactory, HttpClient, and HttpRequest objects, regardless of what implementation module is being used. Requests return responses in the form of the HttpResponse type.

3.1. Configuring the Request

The HttpRequest object is used to configure an HTTP request. It contains many different properties relevant to an HTTP request, as well as configuration properties that alter the behavior of the HttpClient.

3.1.1. HttpRequest Properties

The HTTP Request configuration properties are enumerated in the following table.

Table 2. HttpRequest Properties
Property Type Default Value Description

uri

String

The URI of the request, including the protocol, port, host, and path.

accept

String

Content-Type the client will request from the host in its response.

contentType

String

application/octet-stream

Content-Type of the request. Applies when the request has an entity.

charSet

String

UTF-8

Character set of the request entity.

headers

Map

Request headers. Individual header names can contain multiple values.

queryParameters

Map

Query parameters of the request. Individual query parameters can contain multiple values.

connectionTimeout

Integer

0

The amount of time, in milliseconds, the request should wait for the connection to time out.

readTimeout

Integer

0

The amount of time, in milliseconds, the request should wait to time out while waiting for content.

sslValidated

Boolean

true

Whether SSL certification trust chains and host names should be validated.

followRedirects

Boolean

true

Whether the HTTP client should follow redirect responses received from the host.

bufferResponseEntity

Boolean

true

If true, the entity is read completely from the response and the response is closed. This allows the entity to be retrieved multiple times.

3.1.2. Building an HttpRequest

The HttpRequest object can be built in a couple different ways. All properties have getters and setters defined, and the setter methods all return the same object to support a builder-style syntax. Additionally, Groovy closures are supported.

Using the builder syntax, an HttpRequest object may be created like the following:

Builder Syntax
def request = new HttpRequest()
                      .setUri('http://example.com')
                      .setAccept('application/json')

The same request as above can be configured using a Groovy closure, such as:

Closure Syntax
def request = HttpRequest.build {
    uri = 'http://example.com'
    accept = 'application/json'
}

3.1.3. Behavior Modification

As mentioned before, some of the HttpRequest properties alter the behavior of the HTTP client. This section will identify those properties and their behavior.

followRedirects

By default, the HTTP client will follow redirect responses sent by the server by repeating the request against the new HTTP URI. If authors do not wish to use this behavior, it can be disabled by setting followRedirects to false. When this happens, the HttpResponse object will likely not contain much information other than the HTTP status code.

bufferResponseEntity

When this is enabled, the response entity (if there is one) is automatically read from the HTTP connection’s InputStream into a byte array stored in the HttpResponse object, and the stream is closed. This allows the entity to be retrieved multiple times. When disabled, the response InputStream can be read only once.

It is very important that the response is closed if this functionality is disabled, or the underlying system resources may not be freed up. This is true even when an HTTP status exception is thrown via the HttpStatusExceptionFilter, which includes the HttpResponse in the thrown exception.
sslValidated
Disabling this functionality makes HTTP requests inherently insecure and is not recommended in situations where host identity verification is important, such as production environments.

By default, all SSL and TLS connections go through the standard validation steps, and may throw an IOException if SSL validation fails. Sometimes, authors may wish to bypass this functionality when they know SSL validation will fail but they still trust the remote host. When this property is set to true, SSL validation will not occur and the request should succeed unless there’s a more serious SSL issue encountered.

Disabling SSL validation is a great tool for quickly deploying development environments with self-signed certificates that are not stored in Java keystores.

3.2. Creating the Client

An HttpClient is used to make requests, but HttpClient is an interface and can not be created directly. HTTP client provider modules provide implementations of HttpClient specific to the HTTP client being wrapped. In order to ensure that authors do not need to worry about what type of HttpClient needs to be created, an HttpClientFactory can be used to create the client instances using the createHttpClient() method.

The HttpClientFactory interface is implemented by provider modules and can be created directly. As an example, the Jersey 1.x module’s factory can be used to create a Jersey-specific HttpClient instance.

def factory = new JerseyHttpClientFactory()
def client = factory.createHttpClient()
The HttpClientFactory can be registered as a singleton Spring bean when applications utilize Spring Framework. The factory can then be injected as a dependency into other managed Spring beans, and the provider module can be swapped out transparently and without the need to modify dependent objects. The Grails 3.x plugin library does this work automatically for Grails 3.x applications.

Each provider module declares its own HttpClientFactory, which can be looked up in the following table.

Table 3. HttpClientFactory Classes By Provider
Provider HttpClientFactory Class

http-requests-core

com.budjb.httprequests.reference.ReferenceHttpClientFactory

http-requests-jersey1

com.budjb.httprequests.jersey1.JerseyHttpClientFactory

http-requests-jersey2

com.budjb.httprequests.jersey2.JerseyHttpClientFactory

http-requests-httpcomponents-client

com.budjb.httprequests.httpcomponents.client.HttpComponentsClientFactory

3.3. Using the Client

The HttpClient contains many methods used to initiate requests. Methods are named after the HTTP method to be used for the request.

Supported HTTP Methods
  • GET

  • POST

  • PUT

  • DELETE

  • TRACE

  • OPTIONS

  • HEAD

3.3.1. Client Methods

Every HTTP method is configured by an HttpRequest object. The request object can be constructed and passed to the client, or one of the convenience methods that take a Groovy closure can be used to create the request properties.

Using an HttpRequest Object
def client = new JerseyHttpClientFactory().createHttpClient()

def request = new HttpRequest('http://example.com/foo/bar')

def response = client.get(request)
Using a Groovy Closure
def client = new JerseyHttpClientFactory().createHttpClient()

def response = client.get {
    uri = 'http://example.com/foo/bar'
}

Several of the HTTP methods support request entities (or bodies). For those methods that do, applications can either pass an InputStream that will be sent directly with the HTTP request, or an object that has a valid entity converter registered with the HttpClient.

This documentation will not cover every single class method, but the API documentation can be referenced for a complete list of method signatures.

3.3.2. Client Examples

Basic HTTP GET Request

The HTTP GET method is one of the most used HTTP methods used. This method does not contain a request entity (or body).

Suppose that (for whatever reason) an application needed to retrieve the content’s of Google’s home page. The following code snippet would accomplish this.

Retrieve the google.com Page
def client = new JerseyHttpClientFactory().createHttpClient()
def response = client.get { uri = 'google.com' }
Posting an HTML Form

The FormData object contains a multi-valued map useful to simulate submitting an HTTP form.

Submitting a FormData object with the request automatically sets the Content-Type of the request.
Submit an HTML Form
def form = new FormData()
form.addField('foo', 'bar')
form.addField('hello', 'world')

def request = new HttpRequest().setUri('http://example.com/foo/bar')

def client = new JerseyHttpClientFactory().createHttpClient()
def response = client.post(request, form)
Submit an HTML Form with a Groovy Closure
def form = new FormData()
form.addField('foo', 'bar')
form.addField('hello', 'world')

def client = new JerseyHttpClientFactory().createHttpClient()
def response = client.post(form) { uri = 'http://example.com/foo/bar' }
Making a Request with an InputStream

Applications may need to submit a request with an InputStream, such as when the request is coming from the file system. The following example reads a a file from the disk and sends it in a request.

PUT a File from the Disk
def inputStream = new FileInputStream(new File('/path/to/file'))
def client = new JerseyHttpClientFactory().createHttpClient()
def response = client.put(inputStream) { uri = 'http://example.com/foo/bar' }
Sending a JSON Request

A common use case when working with RESTful APIs involves sending JSON encoded entities.

POST a JSON Entity
def client = new JerseyHttpClientFactory().createHttpClient()
def response = client.post([foo: 'bar', hello: 'world']) {
    uri = 'http://example.com/foo/bar'
    contentType = 'application/json'
}

3.4. Handling Responses

HttpClient request methods return HttpResponse objects that contain information about the response.

3.4.1. HttpResponse Properties

The set of response properties is smaller than that of the request, and provides basic information about the response

Table 4. HttpResponse Properties
Property Type Description

status

Integer

HTTP status code of the response.

contentType

String

Content-Type of the response, if the response contains an entity.

charSet

String

Character set of the response, if the response contains an entity.

headers

Map

Response headers.

allow

List

Contains a list of HttpMethod objects. Typically returned from HTTP OPTIONS requests.

3.4.2. Response Entities

The response entity can be retrieved by using the getEntity() method of the HttpResponse, which return an InputStream containing the entity. By default, the response entity, if available, is buffered in the HttpResponse and the response is closed. Subsequent calls to retrieve the response entity will reproduce the entity an unlimited amount of times. If the request properties disabled this functionality, the response’s entity can be read only once, but it does not require the memory space to buffer the entity in the HttpResponse object. Additionally, it is important to close the response when this functionality is disabled.

When bufferResponseEntity is false, it is very important that the HttpResponse is closed when it is no longer needed. If it is not closed, underlying system resources may not be freed up. The close() method can be called on the HttpResponse to properly clean up the request.

The HttpResponse supports entity conversion using the getEntity(Class<?> type) method. If an entity reader is registered that supports the requested type, it will be converted and returned.

3.4.3. HttpResponse Examples

Retrieving the Response as JSON

A very common use case is to make a request to an API service that responds with JSON. In this example, assume such an API exists and returns a Map structure.

def client = httpClientFactory.createHttpClient()
def response = client.get { uri = "https://example.com/api/foo_resource" }
Map json = response.getEntity(Map)
Using the Response InputStream

Sometimes it may be beneficial to prevent the entity from being buffered or stored in memory at all, especially when the entity is expected to be a large file. By disabling the buffering functionality and accessing the InputStream directly, the server response can be passed directly through streams without buffering, depending on the type of streams being used. The following example writes the response’s InputStream directly to a file.

try {
    def response = httpClientFactory.createHttpClient().get {
        uri = "https://example.com/large_file.tar.gz"
        bufferResponseEntity = false
    }

    try {
        byte[] buffer = new byte[8192]
        int read
        InputStream in = response.getEntity()
        OutputStream out = new FileOutputStream("/path/to/file")

        try {
            while (true) {
                read = in.read(buffer)

                if (read == -1) {
                    break
                }

                out.write(buffer, 0, read)
            }
        }
        finally {
            in.close()
            out.close()
        }
    }
    finally {
        response.close()
    }
}
catch (HttpStatusException e) {
    e.response.close()
}
There are libraries that simplify shoveling data from one stream to another, such as IOUtils from Apache commons.
Retrieving the Response Entity as a String

The simplest of use cases of an HTTP request is to use the response’s entity as a simple string. The following example simply grabs the contents of Google’s home page and prints it out to the system console.

def client = httpClientFactory.createHttpClient()
def response = client.get { uri = "http://google.com" }
println response.getEntity(String)
Handling Status Codes

Suppose an API service requires authentication, and the application must handle authentication failures. The following example watches for HTTP 401 response codes and takes an action based on that error case.

def retrieveData() {
    def client = httpClientFactory().createHttpClient()

    try {
        def response = client.get {
            uri = "https://example.com/api/example_resource"
            headers = ["X-Authentication": "some-authentication-token"]
        }
        return response.getEntity(Map)
    }
    catch (HttpUnauthorizedException e) {
        log.warn("call to retrieve data failed due to invalid authentication", e)
        return null
    }
}

4. Filters

The HttpClient supports the use of filter objects so that applications can react to certain events in the lifecycle of an HTTP request.

Filters can be added to the HttpClientFactory or to the HttpClient directly. Any filters registered with the HttpClientFactory will automatically be assigned to any HttpClient instance the factory creates. For both objects, the following methods are available to work with filters.

  • addFilter

  • removeFilter

  • clearFilters

  • getFilters

4.1. HttpClientRequestFilter

Applications may need to intercept an HTTP request directly before the request is actually made to modify the request parameters. Any request filter that is registered with the HttpClient will be called directly before the request is made.

A great example of a use of this filter is the built-in BasicAuthFilter

// given a client has been created...
client.addFilter(new BasicAuthFilter('username', 'password'))

The BasicAuthFilter is built with the credentials for the request, and the request is updated with that information just before the request is actually made.

4.2. HttpClientRequestEntityFilter

This filter allows applications to modify the OutputStream sent to the actual HTTP connection before it is sent. An example of a built-in filter that uses this functionality is the GZIPFilter, which wraps the output stream in a GZIPOutputStream.

4.3. HttpClientLifecycleFilter

This filter is actually a trait that provides several integration points related to the lifecycle of the request. The trait provides base implementations of each available method so that only the methods a filter cares about must be overridden. The methods available are as follows.

  • onRequest is available to inspect the request properties and the OutputStream, but it should not change anything. This filter method differs from the request filter in that the former is meant to modify the request, while this filter is meant to inspect it once all changes have been made.

  • onResponse is similar in purpose to onRequest, but is called once a response has been received.

  • onComplete is called once the HTTP request has completed and no further retries have been requested. This is the final filter opportunity before the HTTP request call returns to the caller.

4.4. HttpClientResponseFilter

The response filter is called after a response has been received from the HTTP request. This filter provides an opportunity to modify the HttpResponse.

4.5. HttpClientResponseEntityFilter

The response entity filter is called after a response has been received from the HTTP request. This filter provides an opportunity to modify the response’s entity InputStream.

4.6. HttpClientRetryFilter

The retry filter allows applications to determine if a request should be retried based on some criteria. If any one retry filter determines that the request should be retried, the request will be submitted again until no more retry filters request a retry.

A use case for this functionality is authentication schemes that provide an authentication token that can be reused until it expires. A retry filter might be created for this type of authentication scheme that inspects the response of a call to the service that requires the authentication token, and re-authenticates if the previous authentication token has expired. Once a new authentication token is obtained, the original request can be retried with the updated authentication token.

There is a built-in AuthenticationTokenHeaderFilter class that creates a framework for this type of authentication scheme.

4.7. Filter Chain

The graphic below illustrates the order in which the filters explained above are called during the process of making an HTTP request.

HTTP Request Filter Chain
Figure 1. HTTP Request Filter Chain

4.8. Built-In Filters

The library includes several built-in filters for common use cases.

4.8.1. BasicAuthFilter

The basic authentication filter provides simple basic auth functionality with a username and a password.

4.8.2. GZIPFilter

The GZIP filter wraps the output of the request with a GZIPOutputStream so that the entity’s contents are compressed with the GZIP algorithm. The appropriate Content-Encoding header is set.

4.8.3. DeflateFilter

The deflate filter wraps the output of the request with a DeflateOutputStream so that the entity’s contents are compressed with the deflate algorithm. The appropriate Content-Encoding header is set.

4.8.4. AuthenticationTokenHeaderFilter

This filter is abstract and provides a framework for HTTP requests that require token authentication and follow the flow:

  1. If no authentication token is known, authenticate.

  2. Make the HTTP request.

  3. If HTTP status 401 is received, and no retries have been attempted, re-authenticate and request a retry.

The filter provides integration points so that authentication logic and authentication token header names can be provided by concrete implementations of the filter.

4.8.5. LoggingFilter

The logging filter captures the contents of the request and the response and logs it. The LoggingFilter itself is an abstract class, and there are a couple bundled implementations.

ConsoleLoggingFilter

The ConsoleLoggingFilter is a concrete implementation of LoggingFilter that prints the contents of the log to the Java console.

Slf4jLoggingFilter

The Slf4jLoggingFilter is a concrete implementation of LoggingFilter that logs the contents of the log to an SLF4J logger. The filter can be configured with a logger name to use, otherwise the fully-qualified class name of the logging filter will be used. By default, the log is configured to log at the TRACE level. The filter can be configured to log at a different level if desired.

4.8.6. HttpStatusExceptionFilter

The HttpStatusExceptionFilter filter modifies how the response status is treated. When an HTTP status that is not considered successful (that is, not in the 200-299 range), an exception representing that status code is thrown. For example, if an HTTP 404 is received, an HttpNotFoundException will be thrown.

Use this functionality to create different error handling for expected error cases.

If no appropriate exception exists for the HTTP status code that is received, an HttpStatusException is thrown instead. This exception is actually the superclass of all HTTP status specific exceptions.

These exceptions contain the following information about the response:

Table 5. HttpStatusException Properties
Property Description

status

The HTTP status code of the response, as an Integer.

response

The HttpResponse object containing details about the received response.

5. Entity Converters

Entity converters are responsible for converting request and response entities to and from raw byte arrays. This prevents applications from having to make separate calls to helper methods or classes to do the conversion.

Much like filter, an entity converter can be registered with both the HttpClientFactory and HttpClient objects. Any entity converter registered with the factory will automatically be added to any clients it creates. Entity converters can be interacted with on both the factory and the client with the following methods.

  • addEntityConverter

  • removeEntityConveter

  • getEntityConverters

  • clearEntityConverters

5.1. Entity Readers

Entity readers are responsible for converting the raw byte array of a response entity into some other type of object. Applications can create a new entity reader for a custom object type that contains the logic to read relevant information from the byte array entity and create the custom object from it, populated with the data from the response.

Entity reader implementations should implement the EntityReader interface.

5.2. Entity Writers

Entity writers are responsible for converting an object into a raw byte array for use in a request. Much like the entity readers, entity writers provide applications a method to convert custom objects for use in a request without having to handle that logic separately from the request call.

Entity writer implementations should implement the EntityWriter interface.

Entity writers have the option to modify the request by adding the Content-Type of the entity it successfully converted, if a Content-Type makes sense. This only happens if a Content-Type had not been previously set on the request.

5.3. Built-In Converters

The library contains several built-in entity converters that are, by default, automatically registered with all client instances. Those built-ins include:

  • byte[] reader and writer.

  • String reader and writer.

  • GString writer.

  • Map reader and writer.

  • List reader and writer.

  • FormData reader and writer.

6. Grails Plugin

The Grails 3.x plugin makes using the http-requests library with Grails easy. To use the plugin, simply include the Grails plugin artifact (com.budjb:http-requests-grails:1.0.2) and a provider library.

During initialization, the Grails plugin will automatically locate the HttpClientFactory and register it as a Spring bean with the name httpClientFactory. This class can be injected into other managed Spring beans and used without needing to create the factory manually.

The plugin will only use the built-in HTTP client provider if no other provider is found on the classpath.

The plugin also makes it easy to create and use filters and converters. Filters should be created under the grails-app/http-filters directory, and any filter found under that directory will automatically be registered with the httpClientFactory bean. Likewise, any converter found in the grails-app/http-converters directory will automatically be registered with the httpClientFactory bean.

Filters and converters found in their appropriate directories are registered with the client factory, which will automatically include those filters and converters in any HttpClient that it creates.
Filters and converters are registered as managed Spring beans as singletons. Other managed Spring beans can be injected into those filters and converters, much like controllers and services.

There is a small selection of configuration values available to the library, as listed below. These properties can be assigned under the httprequests key in the application configuration.

Table 6. Configuration Properties
Name Type Default Description

autoLoadFactory

Boolean

true

Whether to automatically locate and register an HttpClientFactory from the classpath. If false, applications must manually register an HttpClientFactory with the name httpClientFactory, or the application will fail to start.

autoLoadConverters

Boolean

true

Whether to automatically register the built-in filters and converters to the httpClientFactory. This does not affect loading filters and converters found in the grails-app/http-filters and grails-app/http-converters directories.

scanPackages

List

Additional classpath packages to scan for HttpClientFactory instances. This is useful when custom providers are used.

The following example can be used to include the plugin. In this example, the bill-of-materials is used and the grails plugin and the Jersey 1.x provider is included.

build.gradle
dependencyManagement {
    imports {
        mavenBom "org.grails:grails-bom:$grailsVersion"
        mavenBom "com.budjb:http-requests:1.0.2"
    }
    applyMavenExclusions false
}

dependencies {
    // ...
    compile "com.budjb:http-requests-grails"
    compile "com.budjb:http-requests-jersey1"
}
Using the bill-of-materials allows the version of the http-requests libraries to not require a version.

7. Changelog

The changelog will be updated with release notes for each release of the library.

7.1. Version 1

Major version 1 of the library is the first revision of the interface of the http-requests library.

7.1.1. Version 1.0.2

  • Remove default character set from the HttpRequest.

  • Change default conversion character set from UTF-8 to ISO-8859-1 (per HTTP 1.1 spec).

7.1.2. Version 1.0.1

  • Add missing @DelegatesTo annotation to a method taking a closure.

  • If a retry filter is registered with the client, the request entity is automatically buffered so that it can be resent with subsequent requests.

  • Add a reference implementation of the client to avoid the need to include other provider implementation modules.

  • Made the Grails 3.x plugin compatible with JDK 1.7.

  • Modify Grails 3.x plugin so that the classpath scanner prefers the non-built-in HttpClientFactory if another provider is found.

7.1.3. Version 1.0.0

  • Initial non-beta release of the library.