Fooling Your Own Tests

Fast, isolated/independent, repeatable, self-validating and thorough/timely. Those are the F.I.R.S.T princliples of proper software testing. But, those criterias might not be achievable because of the nature of the tested code. What could we do to achieve those principles? Fool the tests.

Let’s say that we’re developing what I was in charge of, an API wrapper. Its purpose is to communicate with external API through HTTP requests, parse the response, and present in the programming language’s native data structure. Now, we’re presented with some obstacle in testing our code:

  • Speed. Doing HTTP requests is often the opposite of fast, thus violating the first F.I.R.S.T principleshehe. And it will only get worse if the API endpoints suddenly become unreachable on testing.
  • Dependency. Creating an API call means that the test depends on external factor, such as the availability of the API host, the network connection, network speed, etc.
  • Consistency. What if the data returned by the API changed? That could break assertion statements on the tests.

Introducing: VCR and Webmock

Those violations boils down to one source of problem: HTTP requests. If we could sever the dependency to HTTP requests, our test would have no problem. Fortunately, there is a solution to this problem: mocking. No, we’re not insulting the code, we’re going to intercept the HTTP requests and mock (as in fake) the response with our own predetermined data. It’s just like fooling your own test, but in a good way. Kinda.

In ruby, HTTP request mocking can be achieved with VCR and webmock. Webmock is responsible for intercepting HTTP requests and redirect it somewhere else, while VCR will receive those redirected requests and returns proper HTTP responses using predetermined data. Those two modules can be used in conjunction with rspec, the testing suite we use.

To use those two modules, all we need to do is add these lines to the spec_helper.rb file.

require 'webmock/rspec'
require 'vcr'

VCR.configure do |c|
  c.cassette_library_dir = 'spec/fixtures/rbcaw_cassettes'
  c.hook_into :webmock

Capturing Real HTTP Traffic

Additional problem comes if we mock the data: how do we determine the predetermined datahehe. Inputing the data by hand would be troublesome if the tested endpoints are numerous. VCR is helpful in this case as it allows us to record real HTTP request, save it in a file (called cassettes), and replay it later (hence the name VRC).

On the sample code earlier, the cassettes directory can be configured by changing the c.cassette_library_dir value. The directory is relative to the rails’ root directory.

As a warning, the captured traffics will be stored as YAML file as plaintext. This might be troublesome if the network traffics are sensitive information. But, we haven’t found any solution that is more secure and compatible with our current git setup.

Playing the Cassettes

This is a sample of our test case:

context 'with proper connection to the host' do
  before do
    VCR.insert_cassette 'sprint_info', :record => :new_episodes, :match_requests_on => [:method, :uri, :body]

    @task_sprint_info =
    @project_sprint_info = conduit.sprint.get_project_start_end_date(test_project.phid)

  it 'should retrieve the task\'s sprint info properly' do
  	# This function do HTTP request, which is mocked by VCR+webmock
    expect(@task_sprint_info).to eq(test_task.sprint_info)

  it 'should retrieve the project\'s sprint info properly' do
  	# This function do HTTP request, which is mocked by VCR+webmock
    expect(@project_sprint_info).to eq(test_project.sprint_info)

  after do

First, we’ll need to tell the RSpec context to utilize VCR (and thus, webmock). This can be achieved by using the before block and using the VCR.insert_cassette statement. The statement’s first parameter would be the cassettes’s name (without the .yml extension). The :record => :new_episodes parameter tells VCR to record new requests that weren’t recorded on the cassette. The :match_requests_on parameter tells VCR to match requests based on the :method, :uri, :body of the original HTTP request. The match_on’s argument are important as POST API calls with same method and uri might be different if their parameters — which are stored on the body — are not considered for matching.

The HTTP requests done after the before block will not use real network connection (unless the HTTP request is not recorded beforehand) but will be redirected to VCR and it will return the pre-recorded data. Because the requests don’t use real HTTP connection, it also will be much faster. Neat stuff.

Lastly we’ll need to eject the cassette, as other test context might use different cassettes. This can be done with VCR.eject_cassette

Mocking a Network Failure

Another nice feature of webmock is its ability to simulate network timeout. This will be useful to test our code ability to handle network-related errors. This can be done with this block:

before do
  stub_request(:any, /

The function stub_request().to_timeout will make request made to its specified URL returns a Timeout::Error. The URL is expressed in regular expression, hence the lack of quotes and the use of slashes.

Fooling Your Tests is Not That Bad After All

With this method we could easily test our API wrapper’s code while still abiding the F.I.R.S.T principle, which is nice. But, we might need to regularly update the stored cassettes (which should be added to git, if not then our whole effort will be for naught), which can be done simply by deleting the cassette files and re-run the test, which will repopulate the cassettes with latest API call responses.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s