Javascript is required
·
11 min read

Property Based Testing With Typescript

Property Based Testing With Typescript Image

In my current project my colleague Michael Seifert introduced property based testing in our Python codebase. It was the first time I heard about it and it sounded fascinating, so I wanted to also implement it in our frontend code based on Vue.js with Jest as testing framework and TypeScript as programming language.

In this article I want to give you an introduction to property based testing and show you how you can use it in the most used TypeScript-based testing frameworks like Jest, Karma, and Mocha.

Example Based Testing

TL;DR: A single concrete example of expected behavior of the system under test.

Let me first describe how most of us developers usually write their unit tests.

Let's assume we want to test this simple TypeScript function:

1/**
2 * Returns the position of the first occurrence of `pattern` in `text`
3 */
4export const indexOf = (text: string, pattern: string): number => {
5  return text.indexOf(pattern)
6}

Typical unit tests for this method using Jest or Mocha would be:

1describe('Example based tests', () => {
2  it('should return -1 if text does not contain the given pattern', () => {
3    expect(indexOf('abc123', 'zzz')).toBe(-1)
4  })
5
6  it('should return 0 if text contains the given pattern', () => {
7    expect(indexOf('123abc', '123')).toBe(0)
8  })
9
10  it('should return 0 if empty strings are compared', () => {
11    expect(indexOf('', '')).toBe(0)
12  })
13})

So basically we define a set of certain inputs, and the expected result of our function under test if it executes with this given input. If the set of examples is well-chosen the tests can provide high confidence that the function behaves as expected.

As you can imagine, there can be many permutations and mutations of possible inputs and that's exactly the use case where property based testing might be useful for your application.

What is Property Based Testing?

TL;DR: Another way to test programs with generated random (but constrained) inputs instead of relying on hard-coded inputs and outputs.

Property based testing has been introduced by the QuickCheck framework in Haskell and since then it has become quite famous especially in functional programming.

It provides another approach to example based testing and can cover tests as unit, integration and even E2E (end-to-end) tests (which I will cover later in this article).

As the name suggests, property based testing relies on properties. You can think of a property as a trait you expect to see in your output by your given inputs. The expected result does not have to be itself and most of the time it will not be.

An exemplary property :

for all (x, y, ...)

such as precondition(x, y, ...) holds

property(x, y, ...) is true

Using properties, we could state that:

for any strings a, b and c

b is a substring of a + b + c

The testing framework will take this information, generate multiple random entries and runs checks on them. If the test fails, it will provide the used seed and a counterexample. The suggested counterexample is the minimal failing counterexample.

For this substring example: whenever the tested string contains a . in itself, the above check fails and the minimal counterexample would be {a: '.', b: '', c: ''} and not something like {a: 'y837dD!d.', b: 'acxSAD4', c: '!y,wqe2"'}.

As a result, our code is tested more thoroughly and we might find unexpected bugs while running our tests.

Benefits

  • Coverage: Theoretically, all possible inputs are generated without any restrictions which can cover the whole range of integers, string or whatever type you need for your test. This can help to discover unexplored code paths in your program.
  • Reproducible: A seed is produced each time a property test runs. Using this seed it is possible to re-run a test with the same set of data. If the test run fails, the seed and the failing test will be printed to the command line so that it is fully reproducible.
  • Shrink: After a failing test, the framework tries to reduce the input to a smaller input. An example: If your test fails due to a certain character in a string the framework will run the test again with a string that only contains this certain character.

It is also important to note that it does not — by any means — replace unit testing. It only provides an additional layer of tests that might prove very efficient to reduce some boilerplate tests.

Property based testing with TypeScript

Available Libraries

There exist two popular libraries for property based testing with TypeScript (and JavaScript): JSVerify and fast-check

I prefer fast-check for the following reasons:

  • It is more actively maintained.
  • It has strong and up-to-date built-in types thanks to TypeScript (the library itself is also written in TypeScript).

Writing a first fast-check test

To install fast-check you need to run this command in your terminal:

bash
npm i fast-check -D

Then you are already ready to use the library in your existing test framework, like in Jest or Mocha as shown in the following example:

1import * as fc from 'fast-check'
2
3describe('Property based tests', () => {
4  it('should always contain itself', () => {
5    fc.assert(fc.property(fc.string(), (text) => indexOf(text, text) !== -1))
6  })
7
8  it('should always contain its substrings', () => {
9    fc.assert(
10      fc.property(fc.string(), fc.string(), fc.string(), (a, b, c) => {
11        // Alternatively: no return statement and direct usage of expect or assert
12        return indexOf(b, a + b + c) !== -1
13      })
14    )
15  })
16})

Let's take a quick look at the anatomy of our fast-check tests:

  • fc.assert runs the property
  • fc.property defines the property
  • fc.string() defines the inputs the framework has to generate
  • text => { ... } checks the output against the generated value

If we run this tests, we can see that we receive an error:

1Error: Property failed after 1 tests
2{ seed: -481667763, path: "0:0:0:1", endOnFailure: true }
3Counterexample: ["",""," "]
4Shrunk 3 time(s)
5Got error: Property failed by returning false

The error message is correct, and we found an edge-case for our indexOf method under test which we most probably would not have discovered with example based testing.

With these simple steps you can easily introduce property based testing to projects which use Jest or Mocha as test framework independent of the web framework you are using. The code for this demo is available at GitHub.

Angular & Karma Demo

In the following demo, I want to show you how you can integrate property based testing in an Angular application (which per default uses Karma) as test runner. Additionally, I also want to demonstrate the usage of property based testing for end-to-end (E2E) tests using Protractor. The code for this demos is available at GitHub.

First Karma property based unit test

As a base we use an Angular project created with the Angular CLI.

Next step is to install fast-check we, therefore, need to run this command in the terminal:

bash
npm i fast-check -D

For a first test, we add our indexOf test method to app.component.ts:

1@Component({
2  selector: 'app-root',
3  templateUrl: './app.component.html',
4  styleUrls: ['./app.component.sass'],
5})
6export class AppComponent {
7  title = 'angular-demo'
8
9  /**
10   * Returns the position of the first occurrence of `pattern` in `text`
11   */
12  indexOf(text: string, pattern: string): number {
13    return text.indexOf(pattern)
14