Universal code coverage renderer in Emacs

Context

I frequently use code coverage to identify untested code and evaluate the risk of making a code change.

Since I use Emacs for basically everything, I tried to use existing code coverage packages for Emacs. Unfortunately, every code coverage package seems to only be compatiable with a specific code coverage reporting format. The industry has yet to converge on one code coverage format to rule them all.

This means that to get code coverage for Python, Ruby and Go (the languages I use the most at the moment), you need to use more than one tool. I wanted to overcome that limitation and build a unified way to display code coverage across languages. This article walks you through the process of building this new Emacs package.

NOTE: I took inspiration from pycoverage, which has the benefit of being easy to understand.

Rendering coverage data

I started by defining two variables, for the covered and uncovered lines of the current file:

(defvar-local coverage-data-not-covered nil)
(defvar-local coverage-data-covered nil)

A function to render the code coverage indicator:

(defun coverage-line-fringe (linenum)
  "Given LINUM a line number return a propertized fringe for it"
  (let ((color (cond
                ((member linenum coverage-data-not-covered) "red")
                ((and (not (member linenum coverage-data-not-covered))
                      (member linenum coverage-data-covered)) "green")
                ((and (not (member linenum coverage-data-not-covered))
                      (not (member linenum coverage-data-covered))) ""))))
    (propertize " " 'face `(:background ,color :foreground " "))))

What is interesting to note here is that a given line can be in one of three cases: covered, uncovered or not relevant for code coverage.

We can then write a minor mode to set up code coverage on the fringe:

(defun coverage-set-data () (interactive) (message "I do nothing"))

(define-minor-mode coverage-mode
  "Allow annotating the file with coverage information"
  :lighter coverage-mode-text
  (if coverage-mode
      (progn
        (linum-mode t)
        (setf linum-format 'coverage-line-format)
        (coverage-get-data))
    (setf linum-format 'dynamic)
    (linum-delete-overlays)
    (linum-mode -1)))

You can test that it displays correctly by pushing numbers to coverage-data-not-covered and coverage-data-covered, and it should show my lines as green or red:

Note that it took about 20 lines of code to get this example working, which is a prime example of Emacs extensibility!

Fetching real coverage data

Now that we can render the fringe, all that’s left to do is fetching the coverage data in Emacs. I took the shortest path here and built a python script that understands all the code coverage format I use and outputs the coverage data for a given file on stdout. I omitted the ruby and python coverage extractor for the sake of brevity:

#!/usr/local/bin/python3
# Universal code coverage transformer
import sys
import collections
import os
import re
import json
import xmltodict
from typing import List, Optional, Dict, Any


class Coverage:
    """Represents code coverage for one file as two lists: covered
    and uncovered lines"""
    covered: List[str]
    uncovered: List[str]

    def __init__(self, covered: List[str], uncovered: List[str]) -> None:
        self.covered = covered
        self.uncovered = uncovered

    def __str__(self) -> str:
        return "|".join(self.covered)+"\n"+"|".join(self.uncovered)


def extract_go_coverage(fname: str) -> Optional[Coverage]:
    """extract_go_coverage extract coverage from current dir and given file for go projects
    returns None if no go coverage exist"""
    go_coverage_file = "cover.out"
    if not os.path.exists(go_coverage_file):
        return None
    # Get the coverage info
    with open(go_coverage_file) as f:
        coverage_info = [k for k in set(f.readlines()) if fname in k]

    # Mapping of line to covered / not covered
    res: Dict[int, bool] = collections.defaultdict(bool)

    for line in coverage_info:
        match = re.search(
            r"(\w+):(\d+)\.(\d+),(\d+).(\d+) (\d+) (\d+)",
            line
        )
        if not match:
            continue
        _, lowv, _, highv, _, _, count = match.groups()
        low, high = int(lowv), int(highv) + 1
        is_covered = int(count) > 0
        for u in range(low, high):
            res[u] = res[u] or is_covered

    sorted_keys = sorted(res.keys())
    covered: List[str] = [str(k) for k in sorted_keys if res[k]]
    uncovered: List[str] = [str(k) for k in sorted_keys if not res[k]]
    return Coverage(covered, uncovered)


def extract_ruby_coverage(fname: str) -> Optional[Coverage]:
    """extract_ruby_coverage extract coverage from current dir and given file for ruby projects
    returns None if no ruby coverage exist"""
    ...


def extract_python_coverage(fname: str) -> Optional[Coverage]:
    """extract_python_coverage extract coverage from current dir and given file for python projects
    returns None if no python coverage exist"""
    ...


if __name__ == '__main__':
    fname = sys.argv[1]
    # Try to parse the different kind of coverage that we know about
    fn = [
        extract_go_coverage,
        extract_ruby_coverage,
        extract_python_coverage,
    ]
    for f in fn:
        res = f(fname)
        # If we find one that makes sense, print it and exit
        if res is not None:
            print(res)
            sys.exit(0)
    print("Coverage not found!")
    sys.exit(1)

You can test the script by running it directly from the root of a repo with coverage data available and it should display two lines containing covered and uncovered lines “|” separated.

Now we can use the script in Emacs. We can start by building a function to set the coverage data for a given file in the current project; we invoke the external tool that we build and parse its output:

(defun coverage-set-data-for-project-file (fname)
  "Process coverage data for FNAME and set the globals use to display coverage"
  (let* (;; Function to parse a line from the coverage program
         (parse-line 
          (lambda (x) (-map string-to-number (split-string x "|"))))
         ;; Root of the project
         (root
          (projectile-project-root)
         ;; Relative fname of the requested file 
         (relative-fname
          (replace-regexp-in-string root "" fname))
         ;; Command to run to get coverage, using external script
         (cmd
          (format "cd '%s' &&  extract-coverage '%s'" root relative-fname))
         ;; Raw coverage information returned by external script
         (coverage-lines
          (split-string (shell-command-to-string cmd) "\n"))
      (setq coverage-data-covered (parse-line (car coverage-lines))
            coverage-data-uncovered (parse-line (cadr coverage-lines))))))

Finally, we define a high-level function to fetch the code coverage data and re-display it:

(defun set-coverage-data ()
  "Display coverage data for the current buffer"
  (interactive)
  (coverage-set-data-for-project-file (buffer-file-name))
  (linum-mode -1)
  (linum-mode t))

Now calling (coverage-mode) displays the code coverage report using the fringe: