Roses are Red, I Catch Flu; How I Succeed on Testing My Own Code (Part Two)

In the last post, I talked about how RSpec is insufficient in testing codes that run in thread different than the one RSpec runs in. So I decided to try using MiniTest, and it kind of works. Yay!

So, to recap: RSpec can’t mock or stub functions for codes that run in thread different than the one where the mock/stub procedure was called. This was not ideal for the caching module, Phrefetch, which runs concurrently and write to the DB on that concurrent thread. That results in real DB insert operation, thus slowing down the test.

This GitHub gist states that all testing library failed on thread-safety test, except for MiniTest. So I decided to give it a try.

We already used RSpec to test every Ruby code, and porting all of them (and tell the other team members to switch to MiniTest) would be troublesome. So I decided to just use both RSpec and MiniTest, and then combine the coverage report.


Stubbing The Caching Logic

Rather than stubbing the DB operations, I decided to simply stub the caching logic, as it is already tested on another test(s). The stubbed caching logic will simply returns after called, so the caching will appear to be executed successfully almost instantly.

This can be done easily by using the gem minitest-hook, which supplies the helper block around. In the following code, the ‘should finish caching’ it will be run inside the Phrefetch::Cacher.stub block, thus stubbing the Phrefetch::Cacher logic code.

around do |&block|
  Phrefetch::Cacher.stub :cache, 'OK' do
    super(&block)
  end
end

before(:all) do
  VCR.insert_cassette 'phrefetch', :record => :new_episodes, :match_requests_on => [:method, :uri, :body]
end

before do
  Phrefetch::Phrefetch.instance.stop
  @observer = PhrefetchSpecHelper::PhrefetchTestObserver.new
end

it 'should finish caching' do
  Phrefetch::Phrefetch.instance.start(conduit: PhrefetchSpecHelper.conduit,
                                      observers: [@observer],
                                      use_logger: true)

  count = @observer.attempt_count

  Timeout::timeout(PHREFETCH['caching_timeout_secs']) do
    sleep 1 while @observer.attempt_count == count
  end

  @observer.is_success.must_equal true
  @observer.is_timeout.must_equal false
  @observer.is_error.must_equal false
end

Running Both RSpec Specs and MiniTest Specs

Problems immediately appear because both RSpec and MiniTest specs use the same default file naming convention: *_spec.rb. So, one of them should change the naming. I decided to change MiniTest’s specs to *_spec_minitest.rb and its helpers to *_spec_minitest_helper.rb so that RSpec won’t execute those specs, as MiniTest and RSpec specs are obviously incompatible.

That can be done by creating a new rake task lib/tasks/mini_test.rake:

require 'rake/testtask'

Rake::TestTask.new do |t|
  t.pattern = 'spec/**/*_spec_minitest.rb'
  t.warning = false
end

That task will search for MiniTest specs on the spec/ directory and its subdirectories with file name matching with *_spec_minitest.rb.

Testing can be done with the command:

rake test

Merging RSpec and MiniTest Coverage using SimpleCov

As with RSpec, we will need to configure SimpleCov in main helper of MiniTest (which should be manually require‘d on each MiniTest spec):

ENV['RAILS_ENV'] ||= 'test'
require File.expand_path('../../config/environment', __FILE__)

require 'simplecov'
SimpleCov.start
Rails.application.eager_load!

require 'simplecov-lcov'
require 'minitest/autorun'
require 'minitest/pride'
require 'minitest/hooks/default'

SimpleCov::Formatter::LcovFormatter.report_with_single_file = true
SimpleCov.formatter = SimpleCov::Formatter::LcovFormatter

Running rake test will now produce a lcov file on coverage directory. This file can be merged with other lcov files generated from RSpec or Jest with the node module lcov-result-merger.

The problem is, if a file is not accessed at all by SimpleCov, all lines will be marked as zeroes even if said lines are simply empty lines or meta keywords such as begin or end.

Selection_002.png
The zeroes of doom

And said files was accessed on other lcov file, the merge result would be ridiculous:

Selection_003.png
Empty lines such as 6, 8, and 13; meta keyword such as 21; and multi-line command such as 29-34 should not be counted

To alleviate this, each testing library (in this case, RSpec and MiniTest) would need to either:

  1. Access all files by using require
  2. Don’t generate coverage for files without test

Solution #2 would be stupid as catching untested code is the target of TDD. As I suspect that MiniTest will be used only on Phrefetch, I made a compromise:

  1. Phrefetch will have a dummy test on RSpec that simply require it, so the Phrefetch code will be initialized properly (empty lines and meta keywords won’t be counted)
  2. MiniTest will only cover Phrefetch (and classes it depends on, mainly Rails model and RbCAW), and not other files to prevent it from creating the awful zeroes of doom

The result is pretty nice, our test that was previously ran for ~18 seconds on my PC reduced to 4.2 seconds (RSpec) + 3.1 seconds (MiniTest) = 7.3 seconds, with the code coverage not impacted.

sattvika@moebox: ~-Workspace-Kuliah-PPL-PPLD2_004.png
Nice
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s