What is Code Coverage?
It is well understood that writing tests for your projects is a must, but these tests are only effective if you know what lines are covered.
Code coverage is a measurement used to express which lines of code were executed by a test suite. Intuitively we can say that a source code with high code coverage contains less software bugs.
By using code coverage, we can ensure that our test suite covers enough to aid the confidence in changing the codes.
code coverage = hits / (hits + partial + miss)
- hits: codes executed by test suite
- partial: codes not fully executed by test suite
- miss: codes not executed by test suite
Example: A code base with 5 lines executed by tests out of 12 total lines will receive a coverage ratio of 41%
Using code coverage
By using tools used to measure the code coverage, we can not only learn about how much of our codes are covered by tests, but we can also know what exact parts of the codes are not covered well enough. This is made possible because the tools usually map the running execution process back to the source codes.
There are many tools or libraries available, for examples, free ones supporting ruby are:
- simplecov https://github.com/colszowka/simplecov
- codeclimate https://codeclimate.com/
- coveralls https://coveralls.io/
- codecov https://codecov.io/
Actually we are trying to use coveralls at first, but after signing up in the service, we found out that they are not supporting repos hosted in GitLab. (our repo is in GitLab!).
We also tried and explored code climate, just to find out that the free version only support GitHub 😦
Next, we try to use codecov and find out that it is actually good enough, and they support GitLab. yay…
codecov : Code Coverage done right.
Codecov aims to provide an easy access for developers to keep record of coverage reports, share results and graph progress.
Codecov claims that they “incentivize developers to write tests and increase coverage” and they “delivers coverage metrics directly into the modern workflow to promote more code coverage, especially in pull requests where new features and bug fixes commonly occur.”
You #shipit we #coverit @codecov
Basically, Codecov is a static analysis tool. Codecov handles merging reports automatically from all uploads and languages.
Furthermore, simplecov is using Ruby’s built-in Coverage library to gather code coverage data.
Thing to note is:
Codecov does not run your test suite. That is the job of the CI Provider. Codecov gathers coverage reports and other key data for static analysis.
Intermezzo: What is a reasonable code coverage percentage?
Testivus On Test Coverage
Early one morning, a programmer asked the great master:
“I am ready to write some unit tests. What code coverage should I aim for?”
The great master replied:
“Don’t worry about coverage, just write some good tests.”
The programmer smiled, bowed, and left.
Later that day, a second programmer asked the same question.
The great master pointed at a pot of boiling water and said:
“How many grains of rice should I put in that pot?”
The programmer, looking puzzled, replied:
“How can I possibly tell you? It depends on how many people you need to feed, how hungry they are, what other food you are serving, how much rice you have available, and so on.”
“Exactly,” said the great master.
The second programmer smiled, bowed, and left.
Toward the end of the day, a third programmer came and asked the same question about code coverage.
“Eighty percent and no less!” Replied the master in a stern voice, pounding his fist on the table.
The third programmer smiled, bowed, and left.
After this last reply, a young apprentice approached the great master:
“Great master, today I overheard you answer the same question about code coverage with three different answers. Why?”
The great master stood up from his chair:
“Come get some fresh tea with me and let’s talk about it.”
After they filled their cups with smoking hot green tea, the great master began to answer:
“The first programmer is new and just getting started with testing. Right now he has a lot of code and no tests. He has a long way to go; focusing on code coverage at this time would be depressing and quite useless. He’s better off just getting used to writing and running some tests. He can worry about coverage later.”
“The second programmer, on the other hand, is quite experience both at programming and testing. When I replied by asking her how many grains of rice I should put in a pot, I helped her realize that the amount of testing necessary depends on a number of factors, and she knows those factors better than I do – it’s her code after all. There is no single, simple, answer, and she’s smart enough to handle the truth and work with that.”
“I see,” said the young apprentice, “but if there is no single simple answer, then why did you answer the third programmer ‘Eighty percent and no less’?”
The great master laughed so hard and loud that his belly, evidence that he drank more than just green tea, flopped up and down.
“The third programmer wants only simple answers – even when there are no simple answers … and then does not follow them anyway.”
The young apprentice and the grizzled great master finished drinking their tea in contemplative silence.
The point of the story above is that we should not focus on how many code coverage we should aim for, but instead focus on designing tests with better logic and functionality.
That’s all, cheers!