Observing the size of generated test data

Mar 21, 2015

The sample function defined in QuickCheck

QuickCheck has a function called sample which, given a generator, prints some examples values to the standard output (stdout).

It’s signature is:

sample :: Show a => Gen a -> IO ()

Example

Using the choose generator we can use sample to generate random integers in the inclusive range [0, 9] and print them:

λ sample $ choose (0, 9 :: Int)

Output:

1
5
0
2
3
4
6
5
5
4
1

The sample function defined in FsCheck

FsCheck also defines a sample function, though it behaves in a slightly different way, as it takes two additional arguments size and n:

val sample : size:int -> n:int -> gn:Gen<'a> -> 'a list

// `size` is the size of generated test data
// `n` is the number of samples (defaults to 11 in QuickCheck)

Let’s pause here and run sample by supplying different values for size.

Using 0 for size

open FsCheck.Gen

let sample'  = sample 0 11 // Size = 0, Number of samples = 11
let sample'' = sample' <| choose (0, 9)

Output:

Length = 11
    [0]: 4
    [1]: 0
    [2]: 4
    [3]: 0
    [4]: 3
    [5]: 9
    [6]: 3
    [7]: 9
    [8]: 2
    [9]: 8
    [10]: 2

However, since size is 0 the result should be:

Length = 11
    [0]: 0
    [1]: 0
    [2]: 0
    [3]: 0
    [4]: 0
    [5]: 0
    [6]: 0
    [7]: 0
    [8]: 0
    [9]: 0
    [10]: 0

Using 1000 for size

open FsCheck.Gen

let sample'  = sample 1000 11 // Size = 1000, Number of samples = 11
let sample'' = sample' <| choose (0, 9)

Output:

Length = 11
    [0]: 8
    [1]: 6
    [2]: 2
    [3]: 3
    [4]: 7
    [5]: 3
    [6]: 1
    [7]: 2
    [8]: 6
    [9]: 2
    [10]: 0

The above result looks incorrect, in fact it looks the same as the first result, where size was 0.

Since size is 1000 the result should look like:

Length = 11
    [0]: -78
    [1]: 347
    [2]: 242
    [3]: 707
    [4]: 414
    [5]: 49
    [6]: 551
    [7]: 45
    [8]: 897
    [9]: -99
    [10]: 77

At this point, we observed that if a generator ignores size (e.g. as choose does) then sample ignores it’s size parameter.


Just sample with a different generator, then.

Instead of the choose generator, the following example uses generate<int>.

Using 0 for size

open FsCheck.Arb
open FsCheck.Gen

let sample'  = sample 0 11 // Size = 0, Number of samples = 11
let sample'' = sample' <| generate<int>

The results are now looking good:

Length = 11
    [0]: 0
    [1]: 0
    [2]: 0
    [3]: 0
    [4]: 0
    [5]: 0
    [6]: 0
    [7]: 0
    [8]: 0
    [9]: 0
    [10]: 0

That’s because generate<int> takes size into account.

Using 1000 for size

open FsCheck.Arb
open FsCheck.Gen

let sample'  = sample 1000 11 // Size = 1000, Number of samples = 11
let sample'' = sample' <| generate<int>

The results are now looking good:

Length = 11
    [0]: -98
    [1]: 307
    [2]: 142
    [3]: 507
    [4]: 414
    [5]: 39
    [6]: 501
    [7]: 4
    [8]: 807
    [9]: -81
    [10]: 31

How to always control the size of test data, then?

That seems to be the purpose of the sized function.

For the sake of completeness, here’s the original example with choose rewritten to use sized:

sample and choose, using 0 for size

open FsCheck.Gen

let sample'  = sample 0 11 // Size = 0, Number of samples = 11
let sample'' = sample' <| (sized <| fun s -> choose (0, s))

Results to the following, which is now correct:

Length = 11
    [0]: 0
    [1]: 0
    [2]: 0
    [3]: 0
    [4]: 0
    [5]: 0
    [6]: 0
    [7]: 0
    [8]: 0
    [9]: 0
    [10]: 0

sample and choose, using 1000 for size

open FsCheck.Gen

let sample'  = sample 1000 11 // Size = 1000, Number of samples = 11
let sample'' = sample' <| (sized <| fun s -> choose (-s, s))

Results to the following, which now looks correct:

Length = 11
    [0]: 105
    [1]: 682
    [2]: -239
    [3]: 658
    [4]: -6
    [5]: 9
    [6]: 203
    [7]: -43
    [8]: 304
    [9]: -125
    [10]: -423

References

Appendix

The above F# examples use the backward pipe <| operator to look similar to the Haskell examples. In F#, it’s idiomatic to use the forward pipe operator instead.</p>

Using the forward pipe operator, the previous example would be:

let result =
    (fun s -> Gen.choose (-s, s))
    |> Gen.sized
    |> Gen.sample 1000 11

Generators and the 'choose' function

Feb 21, 2015

Test data is produced, and distributed, by test data generators.

Both QuickCheck and FsCheck provide default generators for primitive types, but it’s also possible to build custom generators for custom types.

Generators have types of the form Gen a, where a is the type of the test data to be produced.

QuickCheck

Generators are built from the function choose, which makes a random choice of a value from a range, with a uniform distribution in the closed interval [a, a].

choose :: Random a => (a, a) -> Gen a

The type Gen is an instance of Haskell’s Monad. This involves implementing its minimal complete definition:

return :: a -> Gen a
(>>=) :: Gen a -> (a -> Gen b) -> Gen b

Example: Take a random element from a list

Using do notation:

import Test.QuickCheck

takeFromList :: [a] -> Gen a
takeFromList xs =
    do i <- choose (0, length xs - 1)
       return $ xs !! i

A possible translation into vanilla monadic code:

import Test.QuickCheck

takeFromList :: [a] -> Gen a
takeFromList xs =
    choose (0, length xs - 1) >>= \i -> return $ xs !! i

FsCheck

Similarly, FsCheck defines the type gen as a computation expression.

The choose function is non-generic; instead it generates a 32-bit signed integer, with a uniform distribution in the closed interval [l, h].

val choose : l:int * h:int -> Gen<int>

Example: Take a random element from a list

open FsCheck
open FsCheck.Gen

let takeFromList xs =
    gen { let! i = choose (0, List.length xs - 1) 
          return xs |> Seq.nth i }

Notice that takeFromList has equivalent signature with the one in Haskell:

val takeFromList :  xs:'a list -> Gen<'a> // F#
    takeFromList :: [a]        -> Gen a   // Haskell

References

FsCheck setup in F#

Feb 8, 2015

In the previous post I used Cabal to bootstrap QuickCheck with Hspec. In Haskell, Cabal is a build system which also doubles as a package manager.

This post attempts to do the same in F#; bootstrap FsCheck with xUnit.net, but instead of MSBuild and XML, I’m going to use Paket, Fake, and F#.

Install Paket, if not already installed:

// samples-fscheck.fsx

open System
open System.IO

Environment.CurrentDirectory <- __SOURCE_DIRECTORY__

if not (File.Exists "paket.exe") then
    let url = "https://github.com/fsprojects/Paket/releases/download/0.26.3/paket.exe"
    use wc = new Net.WebClient()
    let tmp = Path.GetTempFileName()
    wc.DownloadFile(url, tmp)
    File.Move(tmp, Path.GetFileName url)

Reference Paket and specify the required packages:

// samples-fscheck.fsx

#r "paket.exe"

Paket.Dependencies.Install """
    source https://nuget.org/api/v2
    nuget FsCheck.Xunit
    nuget FAKE
    nuget xunit.runners
""";;

Reference Fake and specify the build parameters:

// samples-fscheck.fsx

#r "packages/FAKE/tools/FakeLib.dll"

open Fake
open Fake.FscHelper

let outputPath = Path.Combine(Environment.CurrentDirectory, "bin")
let outputFile = @"bin\GettingStarted.dll";
let references =
    [ "packages/xunit/lib/net20/xunit.dll";
      "packages/FsCheck/lib/net45/FsCheck.dll";
      "packages/FsCheck.Xunit/lib/net45/FsCheck.Xunit.dll" ]

Use Fake’s DSL to define the build tasks:

// samples-fscheck.fsx

Target "CreateOutputPath" (fun _ -> 
    CreateDir outputPath)

Target "CleanOutputPath" (fun _ ->
    CleanDir outputPath)

Target "CompileFiles" (fun _ ->
    [ "tests/GettingStarted.fs" ]
    |> Fsc (fun options ->
        { options with
            Output = outputFile
            FscTarget = Library
            References = references }))

Target "CopyAssemblyReferences" (fun _ ->
    CopyFiles outputPath references)

Target "RunTests" (fun _ ->
    !! outputFile
    |> xUnit (fun options -> options))

Compose the build tasks into a build pipeline:

// samples-fscheck.fsx

"CreateOutputPath"
    ==> "CleanOutputPath"
    ==> "CompileFiles"
    ==> "CopyAssemblyReferences"
    ==> "RunTests"

RunTargetOrDefault "RunTests"

Icebreaker:

Add a simple (icebreaker) test, because there’s always a bit of work involved in getting everything up and running:

// tests/GettingStarted.fs

module GettingStarted

open System
open FsCheck.Xunit

let add x y = x + y

[<Property>]
let ``icebreaker`` x y =
    add x y = add y x

Run a build:

Run the following command in a terminal:

fsi samples-fscheck.fsx

See the tests running, and passing, via xUnit.net’s built-in test runner:

fsi samples-fscheck

found: <Path>\samples-fscheck\paket.dependencies
Resolving packages:
    - exploring FAKE 3.14.9
    - exploring xunit 1.9.2
    - exploring xunit.runners 1.9.2
    - exploring FsCheck 1.0.4
    - exploring FsCheck.Xunit 1.0.4
Locked version resolutions written to <Path>\samples-fscheck
\paket.lock
Building project with version: LocalBuild
Shortened DependencyGraph for Target RunTests:
<== RunTests
<== CopyAssemblyReferences
<== CompileFiles
<== CleanOutputPath
<== CreateOutputPath

The resulting target order is:
 - CreateOutputPath
 - CleanOutputPath
 - CompileFiles
 - CopyAssemblyReferences
 - RunTests
Starting Target: CreateOutputPath
<Path>\samples-fscheck\bin already exists.
Finished Target: CreateOutputPath
Starting Target: CleanOutputPath (==> CreateOutputPath)
Deleting contents of <Path>\samples-fscheck\bin
Finished Target: CleanOutputPath
Starting Target: CompileFiles (==> CleanOutputPath)
FSC with args:[|"-o"; "bin\GettingStarted.dll"; "-a"; "--platform:anycpu";
  "--reference:packages/xunit/lib/net20/xunit.dll";
  "--reference:packages/FsCheck/lib/net45/FsCheck.dll";
  "--reference:packages/FsCheck.Xunit/lib/net45/FsCheck.Xunit.dll";
  "tests/GettingStarted.fs"|]
Finished Target: CompileFiles
Starting Target: CopyAssemblyReferences (==> CompileFiles)
Finished Target: CopyAssemblyReferences
Starting Target: RunTests (==> CopyAssemblyReferences)
<Path>\samples-fscheck\packages\xunit.runners\tools\xunit.co
nsole.clr4.exe "<Path>\samples-fscheck\bin\GettingStarted.dl
l"
xUnit.net console test runner (64-bit .NET 4.0.30319.34209)
Copyright (C) 2013 Outercurve Foundation.

xunit.dll:     Version 1.9.2.1705
Test assembly: <Path>\samples-fscheck\bin\GettingStarted.dll


1 total, 0 failed, 0 skipped, took 0.558 seconds
Finished Target: RunTests

---------------------------------------------------------------------
Build Time Report
---------------------------------------------------------------------
Target                   Duration
------                   --------
CreateOutputPath         00:00:00.0018707
CleanOutputPath          00:00:00.0046358
CompileFiles             00:00:03.4156059
CopyAssemblyReferences   00:00:00.0058707
RunTests                 00:00:01.9816487
Total:                   00:00:05.4737549
Status:                  Ok
---------------------------------------------------------------------

See the test failing, as a form of Double Entry Bookkeeping:

// Change:
add x y = add y x

// To:
add x y = add 0 1

Re-run the command in a terminal:

fsi samples-fscheck

xUnit.net console test runner (64-bit .NET 4.0.30319.34209)
Copyright (C) 2013 Outercurve Foundation.

xunit.dll:     Version 1.9.2.1705
Test assembly: <Path>\samples-fscheck\bin\GettingStarted.dll


GettingStarted.icebreaker [FAIL]

   Falsifiable, after 1 test (2 shrinks) (StdGen (1751373120,295969456)):
   (0, 0)
   Stack Trace:
   sorry no stacktrace

1 total, 1 failed, 0 skipped, took 0.679 seconds
Running build failed.

For convenience, all the above are also available on GitHub.

References

QuickCheck setup in Haskell

Jan 30, 2015

Cabal is the build system for Haskell, it also doubles as a package manager. Create a .cabal file:

-- samples-quickcheck.cabal

Name:                   samples-quickcheck
Version:                0.0.0
Cabal-Version:          >= 1.10
Build-Type:             Simple

Library
  Default-Language:     Haskell2010
  HS-Source-Dirs:       tests
  GHC-Options:          -Wall
  Build-Depends:        base
                      , hspec
                      , QuickCheck

Test-Suite spec
  Type:                 exitcode-stdio-1.0
  Default-Language:     Haskell2010
  Hs-Source-Dirs:       tests
  Ghc-Options:          -Wall
  Main-Is:              Spec.hs
  Build-Depends:        base
                      , hspec
                      , QuickCheck

Enable automatic Hspec discovery, so that each module ending with Spec can be automatically considered as a test module:

-- tests/Spec.hs
{-# OPTIONS_GHC -F -pgmF hspec-discover #-}

Add a simple (icebreaker) test, because there’s always a bit of work involved in getting everything up and running:

-- tests/GettingStartedSpec.hs
module GettingStartedSpec (spec) where

import Test.Hspec
import Test.Hspec.QuickCheck

spec :: Spec
spec = do
    describe "icebreaker" $ do
        prop "icebreaker" $ \x y ->
            add x y == add y x

add :: Int -> Int -> Int
add x y = x + y

Create a Cabal sandbox, install dependencies, build the package, and enable the tests, by running the following commands in a terminal:

cabal update                      # Download the most recent list of packages
cabal install cabal-install       # Optional, can be prompted by cabal update

cabal sandbox init                # Initialise the sandbox
cabal install --only-dependencies # Install dependencies into the sandbox
cabal build                       # Build your package inside the sandbox

cabal configure --enable-tests    # Enable the test suites

Run the tests via Cabal’s built-in test runner:

cabal test

Building samples-quickcheck-0.0.0...
Preprocessing library samples-quickcheck-0.0.0...
In-place registering samples-quickcheck-0.0.0...
Preprocessing test suite 'spec' for samples-quickcheck-0.0.0...
[1 of 2] Compiling GettingStartedSpec (tests\GettingStartedSpec.hs, dist\buil...
[2 of 2] Compiling Main (tests\Spec.hs, dist\build\spec\spec-tmp\Main.o)...
Linking dist\build\spec\spec.exe ...
Running 1 test suites...
Test suite spec: RUNNING...
Test suite spec: PASS
Test suite logged to: dist\test\samples-quickcheck-0.0.0-spec.log
1 of 1 test suites (1 of 1 test cases) passed.

See the test failing, as a form of Double Entry Bookkeeping:

-- Change:
add x y == add y x

-- To:
add x y == add 0 1
cabal test

Building samples-quickcheck-0.0.0...
Preprocessing library samples-quickcheck-0.0.0...
In-place registering samples-quickcheck-0.0.0...
Preprocessing test suite 'spec' for samples-quickcheck-0.0.0...
[1 of 2] Compiling GettingStartedSpec (tests\GettingStartedSpec.hs,...
[2 of 2] Compiling Main (tests\Spec.hs, dist\build\spec\spec-tmp\Main.o)...
Linking dist\build\spec\spec.exe ...
Running 1 test suites...
Test suite spec: RUNNING...

GettingStarted
  icebreaker
    icebreaker FAILED [1]

Failures:

  1) GettingStarted.icebreaker icebreaker
       Falsifiable (after 1 test and 1 shrink):
       0
       0

Randomized with seed 1531351472

Finished in 0.0000 seconds
1 example, 1 failure
Test suite spec: FAIL
Test suite logged to: dist\test\samples-quickcheck-0.0.0-spec.log
0 of 1 test suites (0 of 1 test cases) passed.

To use QuickCheck interactively, start a REPL by doing:

cabal repl spec

In the above command, spec is just the name of the test-suite, as declared in the .cabal file.

For convenience, all the above are also available on GitHub.

References

Why QuickCheck can be useful

Jan 29, 2015

(And so FsCheck.)

One of the many challenges when doing Test-Driven Development is to come up with good enough random input values, in such a way that the system under test (SUT) can be exercised every time though the same code path.

These random input values, that exercise the SUT every time through the same code path, belong into a category, or set, or class, named Equivalence Class, after the xUnit Test Patterns book terminology.

QuickCheck provides a DSL, through its various combinators, that allows you to create and consume such Equivalence Classes.

A practical use of Equivalence Classes can be found at the Equivalence Classes, xUnit.net, FsCheck, Property-Based Testing NDC talk by Mark Seemann.

QuickCheck also provides a sophisticated mechanism for executing test cases multiple times, and doing analytics through the concepts of classifying and shrinking, when reporting back the test results.

In-depth introduction for QuickCheck can be found in the original paper, and for FsCheck can be found in this article by Scott Wlaschin.

QuickCheck is created by Koen Claessen and John Hughes. FsCheck is created by Kurt Schelfthout as a port of QuickCheck.

subscribe via RSS