FATE Automated Testing Environment

Table of Contents

1. Introduction

FATE provides a regression testsuite embedded within the Libav build system. It can be run locally and optionally configured to send reports to a web aggregator and viewer http://fate.libav.org.

It is advised to run FATE before submitting patches to the current codebase and provide new tests when submitting patches to add additional features.

2. Running FATE

2.1 Samples and References

In order to run, FATE needs a large amount of data (samples and references) that is provided separately from the actual source distribution.

To inform the build system about the testsuite location, pass ‘--samples=<path to the samples>’ to configure or set the SAMPLES Make variable or the FATE_SAMPLES environment variable to a suitable value.

The dataset is available through rsync, is possible to fetch the current sample using the straight rsync command or through a specific Makefile target.

# rsync -aL rsync://fate-suite.libav.org/fate-suite/ fate-suite

# make fate-rsync SAMPLES=fate-suite

3. Manual Run

FATE regression test can be run through make. Specific Makefile targets and Makefile variables are available:

3.1 FATE Makefile targets

‘fate-list’

List all fate/regression test targets.

‘fate-rsync’

Shortcut to download the fate test samples to the specified testsuite location.

‘fate’

Run the FATE test suite (requires the fate-suite dataset).

3.2 Fate Makefile variables

‘V’

Verbosity level, can be set to 0, 1 or 2.

‘0’

show just the test arguments

‘1’

show just the command used in the test

‘2’

show everything

‘SAMPLES’

Specify or override the path to the FATE samples at make time, it has a meaning only while running the regression tests.

‘THREADS’

Specify how many threads to use while running regression tests, it is quite useful to detect thread-related regressions.

    make V=1 SAMPLES=/var/fate/samples THREADS=2 fate

4. Automated Tests

In order to automatically testing specific configurations, e.g. multiple compilers, tests/fate.sh is provided.

This shell script builds Libav, runs the regression tests and prepares a report that can be sent to fate.libav.org or directly examined locally.

4.1 Testing Profiles

The configuration file passed to fate.sh is shell scripts as well.

It must provide at least a slot identifier, the repo from which fetch the sources, the samples directory, a workdir with enough space to build and run all the tests. Optional submit command fate_recv and a comment to describe the testing profile are available.

Additional optional parameter to tune the Libav building and reporting process can be passed.

slot=                                   # some unique identifier
repo=git://git.libav.org/libav.git      # the source repository
samples=/path/to/fate/samples
workdir=                                # directory in which to do all the work
fate_recv="ssh -T fate@fate.libav.org"  # command to submit report
comment=                                # optional description

# the following are optional and map to configure options
arch=
cpu=
cross_prefix=
cc=
target_os=
sysroot=
target_exec=
target_path=
extra_cflags=
extra_ldflags=
extra_libs=
extra_conf=     # extra configure options not covered above

#make=          # name of GNU make if not 'make'
makeopts=       # extra options passed to 'make'
#tar=           # command to create a tar archive from its arguments on
                # stdout, defaults to 'tar c'

4.2 Submitting Reports

In order to send reports you need to create an ssh key and send it to root@libav.org. The current server fingerprint is a4:99:d7:d3:1c:92:0d:56:d6:d5:61:be:01:ae:7d:e6