Less fragmented properties with Hedgehog

Apr 10, 2017

In this post, I’ll be porting a property-based test from QuickCheck 2 and from FsCheck 2 to Hedgehog. See this post for a brief introduction to Hedgehog.

Setup phase (de)fragmentation

When writing a property, essentially we test a unit in isolation and so some of the practices for traditional unit-tests can also apply to property-based tests. One common practice is to structure each test in three distinct parts executed in sequence

  1. Setup phase
  2. Exercise phase
  3. Verification phase

In classic TDD these phases are called Arrange, Act, Assert, while in BDD they’re called Given, When, Then.

Examples in Haskell

Often the setup phase can become fragmented, as shown in the following example (source) usually because of the ‘noise’ that’s generated because of the Arbitrary required by forAll:

Notice how the setup phase becomes less fragmented with Hedgehog, as shown in the following example (source):

It can be worth pointing out also the fine-grained control we can have over the scope of generated values with the Range combinators1.

Examples in F#

FsCheck is similar to QuickCheck, so its forAll needs an Arbitrary as well, as shown below (source):

Here’s the same example written with Hedgehog (source):

Notice how the setup phase becomes less fragmented with Hedgehog, and how the property Computation Expression makes it easier to separate each distinct phase of the property-based test.


  1. The Range combinators aren’t available yet in the F# version. They are tracked though by this GitHub item. 

Property-based testing becomes easier with Hedgehog

Apr 9, 2017

Hedgehog is a brand new property-based testing system.

In mainstream scenarios it

  • makes properties less fragmented
  • doesn’t require to write shrinkers; it does it for free
  • provides fine-grained control over the scope and shrinking of generated values

It is available for

  • Haskell
  • F#
  • Scala (plannned)
  • PureScript (planned)

It can be used from

Jacob Stanley, the creator of Hedgehog, made yesterday an announcement on Haskell Reddit and overall received interesting feedback, including one from Koen Claessen1.


  1. Koen Claessen created QuickCheck together with John Hughes. 

Write you some QuickCheck - Properties that hold under certain conditions

Jun 13, 2016

This post is part of a series of posts on implementing a minimal version of QuickCheck from scratch. The source code is available on GitHub.

In the previous post, we’ve seen how an assertion can be turned into a property and sampled, using only the Gen and Property modules:

fun x -> if x > 3 then true else false
|> Property.forAll g
|> Property.evaluate
|> Gen.sample;;

val it : Result list =
  [ { Status = Some false
      Args   = ["0"] }

    { Status = Some false
      Args   = ["-1"] }

    { Status = Some false
      Args   = ["-1"] }

    { Status = Some false
      Args   = ["3"] }

    { Status = Some false
      Args   = ["1"] }

    { Status = Some false
      Args   = ["-5"] }

    { Status = Some true
      Args   = ["9"] }

    { Status = Some true
      Args   = ["12"] }

    { Status = Some true
      Args   = ["5"] }

    { Status = Some false
      Args   = ["-2"] }

    { Status = Some true
      Args   = ["6"] }
  ]
>

Notice that in this particular case, some tests passed:

val it : Result list =
  [ ...

    { Status = Some true
      Args   = ["9"] }

    { Status = Some true
      Args   = ["12"] }

    { Status = Some true
      Args   = ["5"] }

    ...

    { Status = Some true
      Args   = ["6"] }
  ]
>

while the rest of them failed:

val it : Result list =
  [ { Status = Some false
      Args   = ["0"] }

    { Status = Some false
      Args   = ["-1"] }

    { Status = Some false
      Args   = ["-1"] }

    { Status = Some false
      Args   = ["3"] }

    { Status = Some false
      Args   = ["1"] }

    { Status = Some false
      Args   = ["-5"] }

    ...

    { Status = Some false
      Args   = ["-2"] }

    ...
  ]
>

In the assertion:

fun x -> if x > 3 then true else false

when we feed x with an integer that is lower then 3, the test fails.

In order to discard test cases where x is lower then 3, we use the ==> operator:

fun x -> x > 3 ==> if x > 3 then true else false

The ==> operator takes a predicate and a function, and returns a property that holds when the predicate returns true:

(* Returns a property that holds under certain conditions. Laws which are
   simple equations are conveniently represented by boolean functions but
   sometimes laws hold only under certain conditions. So this implication
   combinator represents such conditional laws. *)
let implies b x =
    if b then x |> convert
    else     () |> convert

let (==>) b x = implies b x

let private convert candidate =
    match box candidate with
    | :? Property   as p -> p
    | :? bool       as b -> boolProperty b
    | :? Lazy<bool> as b -> boolProperty b.Value
(* ... *) 
    | _                  -> unitProperty

Sample output

fun x -> x > 3 ==> if x > 3 then true else false
|> Property.forAll g
|> Property.evaluate
|> Gen.sample;;

val it : Property.Result list =
  [ { Status = null (* Discarded *)
      Args   = ["0"] }

    { Status = null (* Discarded *)
      Args   = ["2"] }

    { Status = null (* Discarded *)
      Args   = ["-3"] }

    { Status = null (* Discarded *)
      Args   = ["-4"] }

   { Status = null (* Discarded *)
     Args   = ["0"] }

   { Status = Some true
     Args   = ["5"] }

   { Status = Some true
     Args   = ["4"] }

   { Status = null (* Discarded *)
     Args   = ["1"] }

   { Status = Some true
     Args   = ["6"] }

   { Status = Some true
     Args   = ["16"] }

   { Status = null (* Discarded *)
     Args   = ["-2"] }
  ]
>

Tip: Using Lazy Computations

By default, F# expressions are evaluated exactly once when defined, however sometimes at most once semantics are needed, as shown below:

fun x -> x = x <> 0 ==> (1/x = 1/x)
|> Property.forAll g
|> Property.evaluate
|> Gen.sample

(* System.DivideByZeroException : Attempted to divide by zero. *)

In the above example, x can get the value 0, and although the test case is discared, the right part of the condition is eagerly evaluated. — In such cases, a lazy computation is required:

fun x -> x = x <> 0 ==> lazy (1/x = 1/x)
|> Property.forAll g
|> Property.evaluate
|> Gen.sample

(* Does not throw an exception. *)

val it : Property.Result list =
  [ { Status = null (* Discarded *)
      Args   = ["0"] }

   { Status = Some true
     Args   = ["1"] }

    { Status = Some true
      Args   = ["-3"] }

    { Status = Some true
      Args   = ["-7"] }

   { Status = Some true
     Args   = ["4"] }

   { Status = Some true
     Args   = ["9"] }

   { Status = Some true
     Args   = ["13"] }

   { Status = Some true
     Args   = ["-22"] }

   { Status = Some true
     Args   = ["16"] }

   { Status = Some true
     Args   = ["-37"] }
  ]
>

Write you some QuickCheck - Properties

May 27, 2016

This post is part of a series of posts on implementing a minimal version of QuickCheck from scratch. The source code is available on GitHub.

Prelude

In QuickCheck the programmer writes assertions about logical properties that a function should fulfill.

Take List.rev as an example, which is a function that returns a new list with the elements in reverse order:

List.rev [1; 2; 3];;

val it : int list = [3; 2; 1]

One logical property of List.rev is:

  • Calling List.rev twice must return the elements in the original order.

Here’s an example assertion:

List.rev (List.rev [1; 2; 3]) = [1; 2; 3];;

val it : bool = true

A generic assertion

In the previous example List.rev was tested against an example value [1; 2; 3]. To make the assertion generic, the example value can be parameterized as any list:

fun xs -> List.rev (List.rev xs) = xs;;

val it : xs:'a list -> bool when 'a : equality = <fun:clo@33>

QuickCheck will then attempt to generate a test case that falsifies the assertion. In order to do that, it needs to know which generator to use, to feed xs with random values.

A generator for lists of integers

Values for xs need to be generated by a generator. The following one is for lists of type integer:

let g = Gen.list Gen.int;;

val g : Gen<int list>

QuickCheck Properties

Every possible value generated by the g generator must now be supplied to the assertion, as shown below:

fun xs -> List.rev (List.rev xs) = xs
|> Property.forAll g;;

val it : Property

But what is forAll? This comes from predicate logic and essentially means that the assertion holds for all possible values generated by g:

let forAll (g : Gen<'a>) (f : ('a -> 'b)) : Property = 
    Prop(gen { 
             let!      x = g           (* Get 'a from the Gen<'a>. *)
             let! actual = f x         (* Apply the value 'a to f. *)
                           |> convert  (* Turn it into a Property. *)
                           |> evaluate (* Get the results from it. *)
             return { actual with Arguments = x.ToString() :: result.Arguments }
         })

In its simplest form, Property is a generator of Result, (Gen<Result>)1, which means we can easily try out a Property just as we would do with a generator.

type Property =
    private
    | Prop of Gen<Result>

and Result =
    { Ok : option<bool>
      Arguments : list<string> }

Try out (see it pass)

Using only the Gen and Property modules, the assertion can be turned into a property and sampled:

fun xs -> xs |> List.rev |> List.rev = xs
|> Property.forAll g (* For each value generated by g, the property must hold. *)
|> Property.evaluate
|> Gen.sample;;

val it : Result list =
  [ { Status = Some true
      Args   = ["[]"] }

    { Status = Some true
      Args   = ["[0]"] }

    { Status = Some true
      Args = ["[-4; 2]"] }

    { Status = Some true
      Args   = ["[-5; -2; -5; ... ]"] }

    { Status = Some true
      Args   = ["[3]"] }

    { Status = Some true
      Args   = ["[4; 9; -5; ... ]"] }

    { Status = Some true
      Args   = ["[-1; 8; 9; ... ]"] }

    { Status = Some true
      Args   = ["[9; 0; -15; ... ]"] }

    { Status = Some true
      Args   = ["[-11; 18; -9; ... ]"] }
  ]
>

The assertion holds for all cases, which means the ‘test’ passes.

Try out (see it fail)

To see the ‘test’ fail, the assertion can be changed from

fun xs -> xs |> List.rev |> List.rev = xs

to

fun xs -> xs |> List.rev = xs

and run again:

fun xs -> xs |> List.rev = xs
|> Property.forAll g
|> Property.evaluate
|> Gen.sample;;

val it : Result list =
  [ { Status = Some true
      Args   = ["[]"] }
 
    { Status = Some false
      Args   = ["[1; -1]"] }

    { Status = Some false
      Args   = ["[-2; 2]"] }
    
    { Status = Some false
      Args   = ["[3; 3; -3; ... ]"] }

    { Status = Some false
      Args   = ["[-8; 9]"] }

    { Status = Some false
      Args   = ["[1; -6; 11; ... ]"] }

    { Status = Some false
      Args   = ["[-6; 16; -5; ... ]"] }

    { Status = Some false
      Args   = ["[-1; 8; 6; ... ]"] }
 
    { Status = Some false
      Args   = ["[4; 8; -8; ... ]"] }
  ]
> 

The test now fails. — But, there’s some noise in the results; the counter-example isn’t obvious.

It’d be wise if we can clean the noise by trying to report back the minimal counter-example. This process is called shrinking and will be shown over the next post(s).


  1. To support shrinking, a Property needs to hold more information. As an example, see the Property in QuickCheck 2

Atom + Stack = no globally installed GHC/packages

May 19, 2016

When doing Haskell programming

Stack

The output should look like this:

$ stack setup
Downloaded lts-5.17 build plan.
Caching build plan
Fetching package index ...remote: Counting objects: 213332, done.
remote: Compressing objects: 100% (171913/171913), done.
Receiving objects: 100% (213332/213332), 49.84 MiB | 1.79 MiB/s, done.-reused 0

Resolving deltas: 100% (56842/56842), completed with 1 local objects.
From https://github.com/commercialhaskell/all-cabal-hashes
 * [new tag]         current-hackage -> current-hackage
Fetched package index.
Populated index cache.
Preparing to install GHC to an isolated location.
This will not interfere with any system-level installation.
ghc-7.10.3:   15.89 MiB / 128.40 MiB ( 12.37%) downloaded...

GHC installed to [..]\AppData\Local\Programs\stack\x86_64-windows\ghc-7.10.3\
stack will use a locally installed GHC
For more information on paths, see 'stack path' and 'stack exec env'
To use this GHC and packages outside of a project, consider using:
stack ghc, stack ghci, stack runghc, or stack exec
  • In the same directory as Stack.yaml execute stack build ghc-mod hlint stylish-haskell

atom-haskell

  • Install Atom
  • Install atom-haskell via apm: apm install language-haskell haskell-ghc-mod ide-haskell autocomplete-haskell ide-haskell-cabal
  • Start Atom via Stack from the command-line stack exec atom.cmd

That’s it – Happy Haskell programming!

subscribe via RSS