So, to recap: RSpec can’t mock or stub functions for codes that run in thread different than the one where the mock/stub procedure was called. This was not ideal for the caching module, Phrefetch, which runs concurrently and write to the DB on that concurrent thread. That results in real DB insert operation, thus slowing down the test.
This GitHub gist states that all testing library failed on thread-safety test, except for MiniTest. So I decided to give it a try.
We already used RSpec to test every Ruby code, and porting all of them (and tell the other team members to switch to MiniTest) would be troublesome. So I decided to just use both RSpec and MiniTest, and then combine the coverage report.
Stubbing The Caching Logic
Rather than stubbing the DB operations, I decided to simply stub the caching logic, as it is already tested on another test(s). The stubbed caching logic will simply returns after called, so the caching will appear to be executed successfully almost instantly.
This can be done easily by using the gem
minitest-hook, which supplies the helper block
around. In the following code, the ‘should finish caching’
it will be run inside the
Phrefetch::Cacher.stub block, thus stubbing the
Phrefetch::Cacher logic code.
around do |&amp;block| Phrefetch::Cacher.stub :cache, 'OK' do super(&amp;block) end end before(:all) do VCR.insert_cassette 'phrefetch', :record =&gt; :new_episodes, :match_requests_on =&gt; [:method, :uri, :body] end before do Phrefetch::Phrefetch.instance.stop @observer = PhrefetchSpecHelper::PhrefetchTestObserver.new end it 'should finish caching' do Phrefetch::Phrefetch.instance.start(conduit: PhrefetchSpecHelper.conduit, observers: [@observer], use_logger: true) count = @observer.attempt_count Timeout::timeout(PHREFETCH['caching_timeout_secs']) do sleep 1 while @observer.attempt_count == count end @observer.is_success.must_equal true @observer.is_timeout.must_equal false @observer.is_error.must_equal false end
Running Both RSpec Specs and MiniTest Specs
Problems immediately appear because both RSpec and MiniTest specs use the same default file naming convention:
*_spec.rb. So, one of them should change the naming. I decided to change MiniTest’s specs to
*_spec_minitest.rb and its helpers to
*_spec_minitest_helper.rb so that RSpec won’t execute those specs, as MiniTest and RSpec specs are obviously incompatible.
That can be done by creating a new rake task
require 'rake/testtask' Rake::TestTask.new do |t| t.pattern = 'spec/**/*_spec_minitest.rb' t.warning = false end
That task will search for MiniTest specs on the
spec/ directory and its subdirectories with file name matching with
Testing can be done with the command:
Merging RSpec and MiniTest Coverage using SimpleCov
As with RSpec, we will need to configure SimpleCov in main helper of MiniTest (which should be manually
require‘d on each MiniTest spec):
ENV['RAILS_ENV'] ||= 'test' require File.expand_path('../../config/environment', __FILE__) require 'simplecov' SimpleCov.start Rails.application.eager_load! require 'simplecov-lcov' require 'minitest/autorun' require 'minitest/pride' require 'minitest/hooks/default' SimpleCov::Formatter::LcovFormatter.report_with_single_file = true SimpleCov.formatter = SimpleCov::Formatter::LcovFormatter
rake test will now produce a
lcov file on
coverage directory. This file can be merged with other
lcov files generated from RSpec or Jest with the node module
The problem is, if a file is not accessed at all by SimpleCov, all lines will be marked as zeroes even if said lines are simply empty lines or meta keywords such as
And said files was accessed on other
lcov file, the merge result would be ridiculous:
To alleviate this, each testing library (in this case, RSpec and MiniTest) would need to either:
- Access all files by using
- Don’t generate coverage for files without test
Solution #2 would be stupid as catching untested code is the target of TDD. As I suspect that MiniTest will be used only on Phrefetch, I made a compromise:
- Phrefetch will have a dummy test on RSpec that simply
requireit, so the Phrefetch code will be initialized properly (empty lines and meta keywords won’t be counted)
- MiniTest will only cover Phrefetch (and classes it depends on, mainly Rails model and RbCAW), and not other files to prevent it from creating the awful zeroes of doom
The result is pretty nice, our test that was previously ran for ~18 seconds on my PC reduced to 4.2 seconds (RSpec) + 3.1 seconds (MiniTest) = 7.3 seconds, with the code coverage not impacted.