11 vcr (& webmockr), httptest, webfakes
We’re now at a nice stage where we have made a demo of usage for each of the HTTP testing packages, in our exemplighratia package. Of course, the choice of strategy in the demo is a bit subjective, but we hope it showed the best of each tool.
A first message that’s important to us: if you’re learning about HTTP testing and using it in a branch of your own package sounds daunting, create a minimal package for playing!
11.1 What HTTP client can you use (curl, httr, crul)
- httptest only works with httr (the most popular HTTP R client);
- vcr (& webmockr) works with both httr and crul (the two “high-level” HTTP R clients);
- webfakes works with any R HTTP client, even base R if you wish.
11.2 Sustainability of the packages
All packages (vcr, webmockr, httptest, webfakes) are actively maintained. During the writing of this book, issues and pull requests were tackled rather quickly, and always in a very nice way.
11.3 Test writing experience
In all cases having HTTP tests, i.e. tests that work independently from any internet connection, depends on
- setup, which is mainly adding a dependency on the HTTP testing packages in DESCRIPTION, and a setup or helper file;
- providing responses from the API.
The difference between packages, the test writing experience depends on how you can provide responses from the API, both real ones and fake ones.
With vcr and httptest for tests testing normal behavior, after set up (for which there is a helper function), testing is just a function away (vcr::use_cassette(), httptest::with_mock_dir(), httptest::with_mock_api()).
Recording happens automatically during the first run of tests.
You might also provide fake recorded response or dumb down the existing ones.
For creating API errors, and API sequence of responses (e.g. 502 then 200), you end up either using webmockr, or amending mock files, see vcr and httptest related docs.5
With webfakes you need to create an app. There could be one per test, per test file or for the whole test suite. It might seem like more overhead code but being able to share an app between different tests reduces this effort. You can test for an API sequence of responses (e.g. 502 then 200) by following an how-to. The one thing that’s not supported in webfakes yet is a smooth workflow for recording responses, so at the time of writing you might need to write your own workflow for recording responses.
In general setup&test writing might be easier for packages with mocking (vcr and httptest) but you might be able to replicate more complex behavior with webfakes (such as an OAuth dance).
11.3.1 The special case of secrets
With webfakes as no authentication is needed at any point, you have less chance of exposing a secret.
With httptest only the body of responses is saved, so unless it contains secrets, no further effort is needed. If you need to redact mock files, see the corresponding vignette.
With vcr as all HTTP interactions, including request URLs and headers, are saved to disk, you will most often have to use the filter_sensitive_data, filter_request_header and/or filter_response_header arguments of vcr::vcr_configure().
11.3.2 How about making real requests
In all three cases, switching back to real requests might be an environment variable away (turning vcr off, setting the URL of the real web service as URL to be connected to instead of a webfakes fake web service). However, your tests using fixed/fake responses / a fake web service might not work with real requests as you can’t trigger an API error, and as you might test for specific values in your tests using mock files whereas the API returns something different every day. Therefore, and it’s a challenge common to all three packages, you might need to choose to have distinct tests as integration tests/contract tests. See also our chapter about making real requests.
11.4 Test debugging experience
Sadly sometimes one needs to run code from the tests in an interactive session, either to debug tests after making a code change, or to learn how to write HTTP tests.
With webfakes, debugging works this way: load the helper or test file where
- the app is created,
- the environment variable connecting your package code to the fake web service is changed.
Then run your code. To debug webfakes apps, follow the guidance.
With vcr, refer to the debugging vignette: you’ll have to load the helper file or source the setup file after making sure the paths use in it work both from tests/testthat/ and the package root (see ?vcr::vcr_test_path), and then use vcr::inject_cassette(); don’t forget to run vcr::eject_cassette() afterwards.
With webmockr debugging is quite natural, run the code that’s in the test, in particular webmockr::enable() and webmockr::disable().
With httptest, the process is similar as with vcr except the key functions are
11.5 Conclusion
In this chapter we compared the three R packages that make HTTP testing easier. If you are still unsure which one to pick, first try packages out without commitment, in branches or so, but then choose one and commit to your lock-in.
“Every piece of code written in a given language or framework is a step away from any other language, and five more minutes you’ll have to spend migrating it to something else. That’s fine. You just have to decide what you’re willing to be locked into.
(…)
Code these days becomes obsolete so quickly, regardless of what’s chosen. By the time your needs change, by the time the latest framework is obsolete, all of the code will be rotten anyway
(…)
The most dangerous feature about these articles examining cloud lock-in is that they introduce a kind of paralysis into teams that result in applications never being completely fleshed out or finished.”
Vicki Boykis, “Commit to your lock-in”.