Skip to main content

HTTP testing in R

7 Use httptest

In this chapter we aim at adding HTTP testing infrastructure to exemplighratia using httptest. For this, we start from the initial state of exemplighratia again. Back to square one!

Note that the httptest::with_mock_dir() function is only available in httptest version >= 4.0.0 (released on CRAN on 2021-02-01).

Corresponding pull request to exemplighratia Feel free to fork the repository to experiment yourself!

7.1 Setup

Before working on all this, we need to install httptest.

First, we need to run httptest::use_httptest() which has a few effects:

  • Adding httptest as a dependency to DESCRIPTION, under Suggests just like testthat.
  • Creating a setup file under tests/testthat/setup,

When testthat runs tests, files whose name starts with “setup” are always run first. The setup file added by httptest loads httptest.

We shall tweak it a bit to fool our package into believing there is an API token around in contexts where there is not. Since tests will use recorded responses when we are not recording, we do not need an actual API token when not recording, but we need gh_organizations() to not stop because Sys.getenv("GITHUB_PAT") returns nothing.

library(httptest)

# for contexts where the package needs to be fooled
# (CRAN, forks)
# this is ok because the package will used recorded responses
# so no need for a real secret
if (!nzchar(Sys.getenv("GITHUB_PAT"))) {
  Sys.setenv(GITHUB_PAT = "foobar")
}

So this was just setup, now on to adapting our tests!

7.2 Actual testing

The key function will be httptest::with_mock_dir("dir", {code-block}) which tells httptest to create mock files under tests/testthat/dir to store all API responses for API calls occurring in the code block. We are allowed to tweak the mock files by hand, and we will do that in some cases.

Let’s tweak the test file for gh_status_api, it becomes

with_mock_dir("gh_api_status", {
  test_that("gh_api_status() works", {
    testthat::expect_type(gh_api_status(), "character")
    testthat::expect_equal(gh_api_status(), "operational")
  })
})

We only had to wrap the whole test in httptest::with_mock_dir().

If we run this test (in RStudio clicking on “Run test”),

  • the first time, httptest creates a mock file under tests/testthat/gh_api_status/kctbh9vrtdwd.statuspage.io/api/v2/components.json.json where it stores the API response. We however dumbed it down by hand, to
{"components":[{"name":"API Requests","status":"operational"}]}
  • all the times after that, httptest simply uses the mock file instead of actually calling the API.

Let’s tweak our other test, of gh_organizations().

Here things get more exciting or complicated, as we also set out to adding a test of the error behavior. This inspired us to change error behavior a bit with a slightly more specific error message i.e. httr::stop_for_status(response) became httr::stop_for_status(response, task = "get data from the API, oops").

The test file tests/testthat/test-organizations.R is now:

with_mock_dir("gh_organizations", {
  test_that("gh_organizations works", {
    testthat::expect_type(gh_organizations(), "character")
  })
})

with_mock_dir("gh_organizations_error", {
  test_that("gh_organizations errors if the API doesn't behave", {
    testthat::expect_error(gh_organizations())
  })
},
simplify = FALSE)

The first test is similar to what we did for gh_api_status() except we didn’t touch the mock file this time, out of laziness. In the second test there is more to unpack: how do we get a mock file corresponding to an error?

  • We first run the test as is. It fails because there is no error, which we expected. Note the simplify = FALSE that means the mock file also contains headers for the response.
  • We replaced 200L with 502L and removed the body, to end up with a very simple mock file under tests/testthat/gh_organizations_error/api.github.com/organizations-5377e8.R 
structure(list(url = "https://api.github.com/organizations?since=1",
    status_code = 502L, headers = NULL), class = "response")
  • We re-run the tests. We got the expected error message.

Without the HTTP testing infrastructure, testing for behavior of the package in case of API errors would be more difficult.

Regarding our secret API token, since httptest doesn’t save the requests, and since the responses don’t contain the token, it is safe without our making any effort.

In this demo we used httptest::with_mock_dir() but there are other ways to use httptest, e.g. using httptest::with_mock_api() that does not require naming a directory (you’d still need to use a separate directory for mocking the error response).

Find out more in the main httptest vignette.

7.3 Also testing for real interactions

What if the API responses change? Hopefully we’d notice that thanks to following API news. However, sometimes web APIs change without any notice. Therefore it is important to run tests against the real web service once in a while.

As with vcr we setup a GitHub Actions workflow that runs once a week with tests against the real web service. The difference is what and where these tests are. As some tests with custom made mock files can be more specific (e.g. testing for actual values, whereas the latest responses from the API will have different values), instead of turning off mock files usage, we use our old original tests that we put in a folder called real-tests. Most of the time real-tests is .Rbuildignored but in the scheduled run, before checking the package we replace the content of tests with real-tests. An alternative would be to use testthat::test_dir() on that directory but in case of failures we would not get artifacts as we do with R CMD check (at least not without further effort).

Again, one could imagine other strategies, but in all cases it is important to keep checking the package against the real web service fairly regularly.

7.4 Summary

  • We set up httptest usage in our package exemplighratia by running use_httptest() and tweaking the setup file to fool our own package that needs an API token.
  • We wrapped test_that() into httptest::with_mock_dir() and ran the tests a first time to generate mock files that hold all information about the API responses. In some cases we modified these mock files to make them smaller or to make them correspond to an API error.

Now, how do we make sure this works?

  • Turn off wifi, run the tests again. It works! Turn on wifi again.
  • Open .Renviron (usethis::edit_r_environ()), edit “GITHUB_PAT” into “byeGITHUB_PAT”, re-start R, run the tests again. It works! Fix your “GITHUB_PAT” token in .Renviron.

So we now have tests that no longer rely on an internet connection nor on having API credentials.

We also added a continuous integration workflow for having a build using real interactions once every week, as it is important to regularly make sure the package still works against the latest API responses.

For the full list of changes applied to exemplighratia in this chapter, see the pull request diff on GitHub.

How do we get there with yet another package? We’ll try webfakes but first let’s compare vcr and httptest as they both use mocking.