Copyright | (c) 2009-2014 Bryan O'Sullivan |
---|---|
License | BSD-style |
Maintainer | bos@serpentine.com |
Stability | experimental |
Portability | GHC |
Safe Haskell | Trustworthy |
Language | Haskell2010 |
Criterion.Main
Contents
Description
Wrappers for compiling and running benchmarks quickly and easily.
See defaultMain
below for an example.
All of the IO
-returning functions in this module initialize the timer
before measuring time (refer to the documentation for initializeTime
for more details).
Synopsis
- data Benchmarkable
- data Benchmark
- env :: NFData env => IO env -> (env -> Benchmark) -> Benchmark
- envWithCleanup :: NFData env => IO env -> (env -> IO a) -> (env -> Benchmark) -> Benchmark
- perBatchEnv :: (NFData env, NFData b) => (Int64 -> IO env) -> (env -> IO b) -> Benchmarkable
- perBatchEnvWithCleanup :: (NFData env, NFData b) => (Int64 -> IO env) -> (Int64 -> env -> IO ()) -> (env -> IO b) -> Benchmarkable
- perRunEnv :: (NFData env, NFData b) => IO env -> (env -> IO b) -> Benchmarkable
- perRunEnvWithCleanup :: (NFData env, NFData b) => IO env -> (env -> IO ()) -> (env -> IO b) -> Benchmarkable
- toBenchmarkable :: (Int64 -> IO ()) -> Benchmarkable
- bench :: String -> Benchmarkable -> Benchmark
- bgroup :: String -> [Benchmark] -> Benchmark
- nf :: NFData b => (a -> b) -> a -> Benchmarkable
- whnf :: (a -> b) -> a -> Benchmarkable
- nfIO :: NFData a => IO a -> Benchmarkable
- whnfIO :: IO a -> Benchmarkable
- nfAppIO :: NFData b => (a -> IO b) -> a -> Benchmarkable
- whnfAppIO :: (a -> IO b) -> a -> Benchmarkable
- defaultMain :: [Benchmark] -> IO ()
- defaultMainWith :: Config -> [Benchmark] -> IO ()
- defaultConfig :: Config
- makeMatcher :: MatchType -> [String] -> Either String (String -> Bool)
- runMode :: Mode -> [Benchmark] -> IO ()
How to write benchmarks
The Benchmarkable
type is a container for code that can be
benchmarked. The value inside must run a benchmark the given
number of times. We are most interested in benchmarking two
things:
IO
actions. MostIO
actions can be benchmarked directly.- Pure functions. GHC optimises aggressively when compiling with
-O
, so it is easy to write innocent-looking benchmark code that doesn't measure the performance of a pure function at all. We work around this by benchmarking both a function and its final argument together.
Benchmarking IO actions
Most IO
actions can be benchmarked easily using one of the following
two functions:
nfIO
::NFData
a =>IO
a ->Benchmarkable
whnfIO
::IO
a ->Benchmarkable
In certain corner cases, you may find it useful to use the following variants, which take the input as a separate argument:
nfAppIO
::NFData
b => (a ->IO
b) -> a ->Benchmarkable
whnfAppIO
:: (a ->IO
b) -> a ->Benchmarkable
This is useful when the bulk of the work performed by the function is not bound by IO, but rather by pure computations that may optimize away if the argument is known statically, as in 'nfIO'/'whnfIO'.
Benchmarking pure code
Because GHC optimises aggressively when compiling with -O
, it is
potentially easy to write innocent-looking benchmark code that will
only be evaluated once, for which all but the first iteration of
the timing loop will be timing the cost of doing nothing.
To work around this, we provide two functions for benchmarking pure code.
The first will cause results to be fully evaluated to normal form (NF):
nf
::NFData
b => (a -> b) -> a ->Benchmarkable
The second will cause results to be evaluated to weak head normal form (the Haskell default):
whnf
:: (a -> b) -> a ->Benchmarkable
As both of these types suggest, when you want to benchmark a function, you must supply two values:
- The first element is the function, saturated with all but its last argument.
- The second element is the last argument to the function.
Here is an example that makes the use of these functions clearer. Suppose we want to benchmark the following function:
firstN :: Int -> [Int] firstN k = take k [(0::Int)..]
So in the easy case, we construct a benchmark as follows:
nf
firstN 1000
Fully evaluating a result
The whnf
harness for evaluating a pure function only evaluates
the result to weak head normal form (WHNF). If you need the result
evaluated all the way to normal form, use the nf
function to
force its complete evaluation.
Using the firstN
example from earlier, to naive eyes it might
appear that the following code ought to benchmark the production
of the first 1000 list elements:
whnf
firstN 1000
Since we are using whnf
, in this case the result will only be
forced until it reaches WHNF, so what this would actually
benchmark is merely how long it takes to produce the first list
element!
Types
data Benchmarkable #
Creating a benchmark suite
perBatchEnv :: (NFData env, NFData b) => (Int64 -> IO env) -> (env -> IO b) -> Benchmarkable #
perBatchEnvWithCleanup :: (NFData env, NFData b) => (Int64 -> IO env) -> (Int64 -> env -> IO ()) -> (env -> IO b) -> Benchmarkable #
perRunEnvWithCleanup :: (NFData env, NFData b) => IO env -> (env -> IO ()) -> (env -> IO b) -> Benchmarkable #
toBenchmarkable :: (Int64 -> IO ()) -> Benchmarkable #
bench :: String -> Benchmarkable -> Benchmark #
Running a benchmark
nf :: NFData b => (a -> b) -> a -> Benchmarkable #
whnf :: (a -> b) -> a -> Benchmarkable #
nfIO :: NFData a => IO a -> Benchmarkable #
whnfIO :: IO a -> Benchmarkable #
nfAppIO :: NFData b => (a -> IO b) -> a -> Benchmarkable #
whnfAppIO :: (a -> IO b) -> a -> Benchmarkable #
Turning a suite of benchmarks into a program
defaultMain :: [Benchmark] -> IO () Source #
An entry point that can be used as a main
function.
import Criterion.Main fib :: Int -> Int fib 0 = 0 fib 1 = 1 fib n = fib (n-1) + fib (n-2) main = defaultMain [ bgroup "fib" [ bench "10" $ whnf fib 10 , bench "35" $ whnf fib 35 , bench "37" $ whnf fib 37 ] ]
defaultMainWith :: Config -> [Benchmark] -> IO () Source #
An entry point that can be used as a main
function, with
configurable defaults.
Example:
import Criterion.Main.Options import Criterion.Main myConfig = defaultConfig { -- Resample 10 times for bootstrapping resamples = 10 } main = defaultMainWith myConfig [ bench "fib 30" $ whnf fib 30 ]
If you save the above example as "Fib.hs"
, you should be able
to compile it as follows:
ghc -O --make Fib
Run "Fib --help"
on the command line to get a list of command
line options.
defaultConfig :: Config Source #
Default benchmarking configuration.
Other useful code
Create a function that can tell if a name given on the command line matches a benchmark.