21 Caching HTTP requests

Record HTTP calls and replay them

21.1 Package documentation

Check out https://docs.ropensci.org/vcr/ for documentation on vcr functions.

21.2 Terminology

  • vcr: the name comes from the idea that we want to record something and play it back later, like a vcr
  • cassette: A thing to record HTTP interactions to. Right now the only option is the file system (writing to files), but in the future could be other things, e.g. a key-value store like Redis
  • fixture: A fixture is something used to consistently test a piece of software. In this case, a cassette (just defined above) is a fixture - used in unit tests. If you use our setup function vcr_setup() the default directory created to hold cassettes is called fixtures/ as a signal as to what the folder contains.
  • Persisters: how to save requests - currently only option is the file system
  • serialize: translating data into a format that can be stored; here, translate HTTP request and response data into a representation on disk to read back later
  • Serializers: how to serialize the HTTP response - currently only option is YAML; other options in the future could include e.g. JSON
  • insert cassette: create a cassette (all HTTP interactions will be recorded to this cassette)
  • eject cassette: eject the cassette (no longer recording to that cassette)
  • replay: refers to using a cached result of an http request that was recorded earlier

21.3 Basic usage

21.3.1 In tests

In your tests, for whichever tests you want to use vcr, wrap them in a vcr::use_cassette() call like:

library(testthat)
vcr::use_cassette("rl_citation", {
  test_that("my test", {
    aa <- rl_citation()

    expect_is(aa, "character")
    expect_match(aa, "IUCN")
    expect_match(aa, "www.iucnredlist.org")
  })
})

OR put the vcr::use_cassette() block on the inside, but put testthat expectations outside of the vcr::use_cassette() block:

library(testthat)
test_that("my test", {
  vcr::use_cassette("rl_citation", {
    aa <- rl_citation()
  })

  expect_is(aa, "character")
  expect_match(aa, "IUCN")
  expect_match(aa, "www.iucnredlist.org")
})

Don’t wrap the use_cassette() block inside your test_that() block with testthat expectations inside the use_cassette() block, as you’ll only get the line number that the use_cassette() block starts on on failures.

The first time you run the tests, a “cassette” i.e. a file with recorded HTTP interactions, is created at tests/fixtures/rl_citation.yml. The times after that, the cassette will be used. If you change your code and more HTTP interactions are needed in the code wrapped by vcr::use_cassette("rl_citation", delete tests/fixtures/rl_citation.yml and run the tests again for re-recording the cassette.

21.3.2 Outside of tests

If you want to get a feel for how vcr works, although you don’t need too.

library(vcr)
library(crul)

cli <- crul::HttpClient$new(url = "https://eu.httpbin.org")
system.time(
  use_cassette(name = "helloworld", {
    cli$get("get")
  })
)

The request gets recorded, and all subsequent requests of the same form used the cached HTTP response, and so are much faster

system.time(
  use_cassette(name = "helloworld", {
    cli$get("get")
  })
)

Importantly, your unit test deals with the same inputs and the same outputs - but behind the scenes you use a cached HTTP response - thus, your tests run faster.

The cached response looks something like (condensed for brevity):

http_interactions:
- request:
    method: get
    uri: https://eu.httpbin.org/get
    body:
      encoding: ''
      string: ''
    headers:
      User-Agent: libcurl/7.54.0 r-curl/3.2 crul/0.5.2
  response:
    status:
      status_code: '200'
      message: OK
      explanation: Request fulfilled, document follows
    headers:
      status: HTTP/1.1 200 OK
      connection: keep-alive
    body:
      encoding: UTF-8
      string: "{\n  \"args\": {}, \n  \"headers\": {\n    \"Accept\": \"application/json,
        text/xml, application/xml, */*\", \n    \"Accept-Encoding\": \"gzip, deflate\",
        \n    \"Connection\": \"close\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\":
        \"libcurl/7.54.0 r-curl/3.2 crul/0.5.2\"\n  }, \n  \"origin\": \"111.222.333.444\",
        \n  \"url\": \"https://eu.httpbin.org/get\"\n}\n"
  recorded_at: 2018-04-03 22:55:02 GMT
  recorded_with: vcr/0.1.0, webmockr/0.2.4, crul/0.5.2

All components of both the request and response are preserved, so that the HTTP client (in this case crul) can reconstruct its own response just as it would if it wasn’t using vcr.

21.3.3 Less basic usage

For tweaking things to your needs, make sure to read the docs about configuration (e.g., where are the fixtures saved? can they be re-recorded automatically regulary?) and request matching (how does vcr match a request to a recorded interaction?)

All components of both the request and response are preserved, so that the HTTP client (in this case crul) can reconstruct its own response just as it would if it wasn’t using vcr.

21.4 vcr enabled testing

21.4.1 check vs. test

TLDR: Run devtools::test() before running devtools::check() for recording your cassettes.

When running tests or checks of your whole package, note that you’ll get different results with devtools::check() (check button of RStudio build pane) vs. devtools::test() (test button of RStudio build pane). This arises because devtools::check() runs in a temporary directory and files created (vcr cassettes) are only in that temporary directory and thus don’t persist after devtools::check() exits.

However, devtools::test() does not run in a temporary directory, so files created (vcr cassettes) are in whatever directory you’re running it in.

Alternatively, you can run devtools::test_file() (or the “Run test” button in RStudio) to create your vcr cassettes one test file at a time.

21.4.2 CI sites: GitHub Actions, Appveyor, etc.

Refer to the security chapter.