packcheck

TL; DR

• Just copy .travis.yml and appveyor.yml to your package repo and your package is CI ready.
• Copy packcheck.sh to your local machine and run it from your package directory (works on Linux/OSX/Windows) to perform all the same tests that are done by CI, locally.

What is it?

packcheck is a minimal yet complete "hello world" Haskell package with model travis and appveyor config files that can be used unmodified in any Haskell package. The CI configs can be modified declaratively to adapt to any kind of build scenario you can imagine.

The package includes packcheck.sh, a high level universal super build script to uniformly, consistently build and comprehensively sanity test a Haskell package across build tools (stack/cabal) and across all platforms (Linux/MacOS/Windows). You do not need to be familiar with any of the build tools to use it.

This is also a minimal yet complete model package (with tests, benchmarks, Linux/MacOS/Windows CI already working) that can be used as a starting point to develop a new package. Beginners can use it to learn about haskell package metadata structure.

What all does it do?

In a single invocation it performs a whole battery of tests: build source, build benchmarks, build docs, run tests, create source distribution, build from source distribution, test install after build, perform distribution checks, generate coverage report, optionally send coverage report to coveralls.io. Everything can be controlled by the user.

Where can I use it?

Everywhere. The same build steps can be consistently performed on or using:

• Linux/OSX/Windows
• Travis/Appveyor/Local Host
• stack/cabal

How do I use it?

To use it for CI, simply copy the travis, appveyor config files from this package to your package and that's it. It should work without modification, of course you can edit them to customize. For use on local host, just copy over the packcheck.sh script and put it in your PATH. Run the script from the package directory of the package you want to build.

$ packcheck.sh stack
$ packcheck.sh cabal

Salient Features

• Comprehensive battery of tests for CI or pre-release to hackage
• works for all build tools, all platforms, all CIs
• tests the source distribution of the package so you can rest assured that what you release on hackage is exactly what you tested. Also performs distribution checks and whether the package installs successfully.
• reproduce a failed CI on local machine. You can just cut and paste the same command on your local machine and run it there for easy debugging.
• can send coverage information to coveralls.io with a simple option.
• installs all the required tools automatically (including stack) or lets you know what it needs so you can install/use your own. It never overwrites an existing tool during install.

Usage Examples

You can run these commands on your local machine as well as inside a CI script.

Make sure you are in the package directory. You can try these commands in this package itself:

$ cd packcheck

Stack build (installs stack automatically if not found, creates a stack.yaml if not found):

$ ./packcheck.sh stack RESOLVER=lts-6

Set pvp-bounds before the test:

$ ./packcheck.sh stack RESOLVER=lts-6 SDIST_OPTIONS="--pvp-bounds both"

Stack build with system installed GHC, when GHCVER is specified it looks for the specified GHC version in PATH:

$ ./packcheck.sh stack GHCVER=7.10.3

Stack build with a specific stack.yaml config file and specified build flags, and requiring a specific cabal version:

$ ./packcheck.sh stack RESOLVER=lts-7.24 STACK_YAML=stack-8.0.yaml STACK_BUILD_OPTIONS="--flag streamly:examples-sdl" CABALVER=1.24

Cabal build using stack installed ghc:

$ stack exec ./packcheck.sh cabal RESOLVER=lts-6

Cabal build using system installed ghc and cabal on PATH:

$ ./packcheck.sh cabal GHCVER=7.10.3 CABALVER=1.22

Run hlint commands on the directories src and test:

$ ./packcheck.sh stack HLINT_COMMANDS="hlint lint src; hlint lint test"

Send coverage info of the testsuites named test1 and test2 to coveralls.io using hpc-coveralls. Note that this currently works only with a cabal build:

$ ./packcheck.sh cabal GHCVER=8.0.2 COVERALLS_OPTIONS="test1 test2"

Diagnostics

There may be issues due to some environment variables unknowingly set or some command line parameters or env variables being misspelled and therefore silently ignored. To avoid any such issues the cleanest way to invoke the script is to use a clean environment using env -i and CHECK_ENV=y parameter. When this parameter is set unwanted/misspelled variables are detected and reported.

$ env -i CHECK_ENV=y ./packcheck.sh stack

For performance diagnostics the script prints the time elapsed from the beginning at each build step performed.

Full Reference

Options marked DESTRUCTIVE! are fine in a CI environment. But on a local machine sometimes it may not be desirable as it will change the state of your global cabal config, so consider that before using these options.

By default cabal builds are done using sandboxes. It creates any temporary files or build artifacts inside .packcheck directory. See the clean and cleanall commands to release the temporary space.

stack is automatically installed and can be used to do cabal builds as well. If you specify BUILD=cabal and RESOLVER at the same time then the cabal build uses stack installed cabal and ghc, both are installed automatically when needed.

For pure cabal builds i.e. when BUILD=cabal and RESOLVER is not specified, cabal and ghc must be pre-installed on the system before building.

$ packcheck.sh --help

--------------------------------------------------
Usage
--------------------------------------------------
./packcheck.sh COMMAND [PARAMETER=VALUE ...]

For example:
./packcheck.sh stack RESOLVER=lts-10.0 GHC_OPTIONS="-O0 -Werror"

Control parameters can either be passed on command line or exported
as environment variables. Parameters marked DESTRUCTIVE may modify
your global user config or state.

--------------------------------------------------
Commands
--------------------------------------------------
stack                   : build using stack
cabal                   : build using cabal
clean                   : remove the .packcheck directory
cleanall                : remove .packcheck, .stack-work, .cabal-sandbox directories
help                    : show this help message

--------------------------------------------------
Commonly used parameters or env variables
--------------------------------------------------
RESOLVER                : Stack resolver to use for stack or cabal builds
GHCVER                  : [a.b.c] GHC version prefix (may not be enforced when using stack)
CABALVER                : [a.b.c.d] Cabal version (prefix) to use
STACKVER                : [a.b.c.d] Stack version (prefix) to use
GHC_OPTIONS             : Specify GHC options to use
SDIST_OPTIONS           : Arguments to stack/cabal sdist command (e.g. --pvp-bounds)
DISABLE_SDIST_BUILD     : [y] Do not build from source distribution
DISABLE_BENCH           : [y] Do not build benchmarks, default is to build but not run
PATH                    : [path] Set PATH explicitly for predictable builds
TEST_INSTALL            : [y] DESTRUCTIVE! Install the package after building (force install with cabal)

--------------------------------------------------
Advanced stack build parameters or env variables
--------------------------------------------------
STACK_YAML              : Alternative stack config, cannot be a path, just the file name
STACK_OPTIONS           : ADDITIONAL stack global options (e.g. -v) to append
STACK_BUILD_OPTIONS     : ADDITIONAL stack build command options to append
STACK_UPGRADE           : [y] DESTRUCTIVE! Upgrades stack to latest version

--------------------------------------------------
Advanced cabal build parameters or env variables
--------------------------------------------------
CABAL_USE_STACK_SDIST   : [y] Use stack sdist (to use --pvp-bounds)
CABAL_CONFIGURE_OPTIONS : ADDITIONAL default cabal configure options to append
CABAL_CHECK_RELAX       : [y] Do not fail if cabal check fails on the package.
CABAL_NO_SANDBOX        : [y] DESTRUCTIVE! Clobber (force install) global cabal ghc package db
CABAL_HACKAGE_MIRROR    : [y] DESTRUCTIVE! Specify an alternative mirror, will modify the cabal user config file.
CABAL_REINIT_CONFIG     : [y] DESTRUCTIVE! Remove old cabal config to avoid any config incompatibility issues

--------------------------------------------------
Coverage related parameters or env variables
--------------------------------------------------
COVERALLS_OPTIONS       : hpc-coveralls args and options, usually just test suite names
COVERAGE                : [y] Just generate coverage information

--------------------------------------------------
hlint related parameters or env variables
--------------------------------------------------
HLINT_COMMANDS          : hlint commands e.g.'hlint lint src; hlint lint test'

--------------------------------------------------
Diagnostics parameters or env variables
--------------------------------------------------
CHECK_ENV               : [y] Treat unknown env variables as error, used with env -i
BASE_TIME               : System time to be used as base for timeline reporting