Running Julia tests

Running all tests with Pkg.test()

Before running tests for a new package, make sure the test environment is already set up. If the package does not have tests yet, follow creating-julia-test-env first. Tests that use using Test must have Test available from the package or test project.

To run all of your package's tests, use the following command:

$ julia --project -e 'using Pkg; Pkg.test()'

Running specific test sets only

To run only particular test sets (for example, those defined as @testset "label" begin ... end), use the ~/.julia/bin/testrunner tool.

If testrunner is already installed, this command prints its path:

$ command -v testrunner

If you do not have the testrunner command, install it with:

$ julia -e 'using Pkg; Pkg.activate(); Pkg.Apps.add(url="https://github.com/aviatesk/TestRunner.jl")'

Basic Usage

$ testrunner demo.jl "basic tests"                   # Run the testset named "basic tests"

$ testrunner demo.jl "basic tests" "struct tests"    # Run multiple named testsets

$ testrunner demo.jl '(:(@test startswith(inner_func2(), "inner")))'  # Run a standalone test case

See the guide to learn more.

Caveat: There are cases where name or regex filters do not work as expected

Sometimes, even after specifying a testset name or a regex, all tests may run instead of just the intended ones (this seems to be due to a pattern matching behavior in TestRunner.jl). Example:

# Expected: Only "calculate: basic operations" should run (4 tests), but actually all 20 tests run
$ testrunner --project=. test/runtests.jl "calculate: basic operations"
$ testrunner --project=. test/runtests.jl 'r"^calculate: basic operations$"'
$ testrunner --project=. test/runtests.jl L6:11   # Even if you include the @testset declaration line, all tests are executed

As a workaround, specify the line range including only the @test lines (do not include the @testset ... begin declaration line). This way, you can ensure that only the intended testset is executed:

# In the case where the target @test lines are in lines 7-10 of runtests.jl
$ testrunner --project=. test/runtests.jl L7:10

Always confirm whether your filter worked as intended by checking the Total count in the Test Summary or the n_passed field in the --json output.

Keeping sessions warm with WarmTestRunner.jl

WarmTestRunner.jl is a test runner that accelerates local test iterations while developing Julia packages.

Whereas standard Pkg.test() runs your tests in a fresh Julia process every time, WarmTestRunner.jl reuses a daemon process and a pool of warm workers, spreading the loading and compilation cost across multiple test runs.

As a general guideline, use it as follows:

• For everyday edit/test cycles: using WarmTestRunner; runtests()
• For final checks before merges, releases, or CI-style runs: Pkg.test()

To install WarmTestRunner.jl:

$ julia -e 'using Pkg; Pkg.activate(); Pkg.develop(url="https://github.com/terasakisatoshi/WarmTestRunner.jl")'

Basic Usage

$ cd path/to/target/package
$ julia --project -e 'using WarmTestRunner; runtests()'

To run testsets in parallel, start the WarmTestRunner daemon with multiple jobs in one terminal, then run tests with split_testsets = true from another terminal:

# Terminal 1
$ cd path/to/target/package
$ julia --project -e 'using WarmTestRunner; serve(jobs=4)'

# Terminal 2
$ cd path/to/target/package
$ julia --project -e 'using WarmTestRunner; runtests(split_testsets = true)'

This splits testsets and executes them across the warm worker pool.

When runtests() starts the daemon automatically, a daemon process runs in the background. To manually stop any WarmTestRunner daemon, run the following:

$ cd path/to/target/package
$ julia --project -e 'using WarmTestRunner; stop()'

If you run Pkg.build() for the target package, stop the daemon (stop()) first and rerun runtests() afterwards.

This is especially important when developing a Julia package with a C interface. In that workflow, Pkg.build() is often used to rebuild or replace a shared library, and WarmTestRunner workers may still have the old shared library loaded. Stop the daemon before rebuilding so the next test run starts from fresh workers:

$ cd path/to/target/package
$ julia --project -e 'using WarmTestRunner; stop()'
$ julia --project -e 'using Pkg; Pkg.build()'
$ julia --project -e 'using WarmTestRunner; runtests()'
# update files in ./src or ./test/
$ julia --project -e 'using WarmTestRunner; runtests()'
# update files in ./src or ./test/ ... repeat the cycle

Caveat: When the workspace structure changes, run stop() first

The daemon caches the path and contents of Project.toml into its worker pool at startup, so runtests() may fail to start with an error after any of the following workspace structure changes:

• Deleting or regenerating Project.toml / Manifest.toml
• Moving or renaming the package directory
• Editing the [workspace] / [deps] / [sources] sections
• Removing files the worker is still referencing from outside the daemon

Example errors (recorded in the controller log ~/.julia/warmtestrunner/logs/controller-*.err.log):

nested task error: ".../path/to/Project.toml": No such file
ERROR: timed out waiting for server record for ...

The recovery procedure is the same as for Pkg.build(): first stop(), then run runtests() again:

$ julia --project -e 'using WarmTestRunner; stop()'
$ julia --project -e 'using WarmTestRunner; runtests()'