test

Requirements

To start testing a core library / a plugin, you need the following requirements:

  • Make sure that the "Testing" subdirectory exists in your library / plugin directory (if not, create it)
  • The CMakeLists under the Testing directory is in place (if not, copy and adapt the one from medGui/Testing)
  • The library / plugin CMakeLists contains the following code:

if (BUILD_TESTING) add_subdirectory (Testing) endif()

Setting up a new test

To create a new test, just run dtkTestGenerator:

DtkTestGenerator1.png

Type in the class you want to test (or test name), and choose the "Testing" directory of the targeted library / plugin (the one containing the class to test):

DtkTestGenerator2.png

Click on File -> Generate. The generator creates the file:

  • medViewContainersTest.h ([TestName]Test.h)
  • medViewContainersTest.cpp ([TestName]Test.cpp)

under the "Testing" directory you provided.

Finally, add the test in the CMakeLists under the Testing directory of your target lib / plugin:

set(${PROJECT_NAME}_HEADERS_MOC
  medViewContainersTest.h)

set(${PROJECT_NAME}_SOURCES
  medViewContainersTest.cpp)

set(${PROJECT_NAME}_MAIN_SOURCES
  medViewContainersTest.cpp)

Configure, make and run ctest (or make tests). The test runs and fails (of course, it has not been defined yet)!

Anatomy of a test

Let's have a look at the generated files.

medViewContainersTest.h:

#include <dtkCore/dtkTest.h>

class medViewContainersTestObject : public QObject
{
    Q_OBJECT

public:
             medViewContainersTestObject(void);
    virtual ~medViewContainersTestObject(void);


private slots:
    /*
     * initTestCase() is called before the
     * execution of all the test cases.
     * If it fails, no test is executed.
     */
    void initTestCase();

    /*
     * init() is called before each test.
     * If it fails, the following
     * test is not executed.
     */
    void init();

    /*
     * cleanup() is called after each test.
     */
    void cleanup();

    /*
     * cleanupTestCase() is called
     * after all test have been executed.
     */
    void cleanupTestCase();

private slots:

    // every function here is a test, unless they end with "_data"
    // in this case "testFoo_data()" will prepare some data
    // to run "testFoo()" different times
    // with different input values
    // void testFoo_data();
    void testFoo();
};

As you see, it is a regular Qt object (containing the Q_OBJECT macro). Note the inclusion of the dtkCore.h header:

#include <dtkCore/dtkTest.h>

Now, go directly at the end of the header:

private slots:

    // every function here is a test, unless they end with "_data"
    // in this case "testFoo_data()" will prepare some data
    // to run "testFoo()" different times
    // with different input values
    // void testFoo_data();
    void testFoo();

This is the most important part: declare here your test functions. Qt will automatically understand that those functions are those containing the tests - furthermore, they will be exposed to the command line to be able to choose only one function to be tested. Here testFoo() is a test function but you can rename it and add as many functions as you want. We will discuss later what the testFoo_data() is.


The other private slots are here for Qt testing mechanism:

  • initTestCase() is called prior to running all test functions. It is called only once to give a chance to initialize some global properties. This can be used to initialize the plugin path for instance.
  • init() is called before running every test functions. It is run as many times as there is test functions. It can be used to reset a global variable that has been changed in a test function to start the next test function with the same initial conditions (example: reset a view).
  • cleanup() and cleanupTestCase() follow the same principle and are called after each test, and at the end of the test execution.

medViewContainers.cpp

#include "medViewContainersTest.h"

medViewContainersTestObject::medViewContainersTestObject(void)
{
}

medViewContainersTestObject::~medViewContainersTestObject(void)
{
}

void medViewContainersTestObject::initTestCase()
{
}

void medViewContainersTestObject::init()
{
}

void medViewContainersTestObject::cleanup()
{
}

void medViewContainersTestObject::cleanupTestCase()
{
}

*
void medViewContainersTestObject::testFoo_data()
{
}
*/

void medViewContainersTestObject::testFoo()
{
    QVERIFY(true==false);
}

/**
   DTKTEST_NOGUI_MAIN will create the entry point without running
   a window manager (such as X on linux). If you need one, change
   it to DTKTEST_MAIN().
 **/
DTKTEST_NOGUI_MAIN(medViewContainersTest,medViewContainersTestObject)


Let's focus on 2 things. The first is this part:

void medViewContainersTestObject::testFoo()
{
    QVERIFY(true==false);
}

This is the default implementation of the test function testFoo(). QVERIFY is Qt macro...which verifies if the condition is true. Of course, QVERIFY(true==false) will always return false: the default test is consequently failing!


The second part is this one:

/**
   DTKTEST_NOGUI_MAIN will create the entry point without running
   a window manager (such as X on linux). If you need one, change
   it to DTKTEST_MAIN().
 **/
DTKTEST_NOGUI_MAIN(medViewContainersTest,medViewContainersTestObject)

You see that there are 2 possible macros to declare a test:

  • DTKTEST_MAIN creates an entry point with a instance of a QApplication. The event loop is running, allowing GUI to be tested, signal/slot connections to be functional, etc.
  • DTKTEST_NOGUI_MAIN does not have an event loop. Useful to have lightweight tests. But do not use it if you want to test UI or if you need the event loop.

You will also notice that the test declaration with the above macros need 2 parameters: medViewContainersTest and medViewContainersTestObject. This is for the CTest / QTest compatibility. Always use this syntax:

DTKTEST_(NOGUI)_MAIN (TestName, TestNameObject)


Now, let's implement an actual test for the medViewContainers.

Testing a core class (not a plugin)

Let's test the medViewContainerSingle class. To do so, we need:

  • dummy views to feed the container with
  • nothing else

Indeed, view containers are placeholders for...views, so we need some. Since we are testing a core feature, we MUST NOT create a dependency on plugins. But nothing prevents us from creating DUMMY views, whose purpose will only be to test the view containers.


Here are my dummy views:

class testView : public medAbstractView
{
public:
     testView();
    ~testView();

    QString identifier(void) const;

public:
    QWidget *widget(void);

public slots:
    void close(void);

private:
    QWidget *m_widget;
};

testView::testView()
{
    m_widget = new QWidget;
    QLabel *label = new QLabel ("testView", m_widget);
    QHBoxLayout * layout = new QHBoxLayout (m_widget);
    layout->addWidget(label);
}

testView::~testView()
{
}

QString testView::identifier() const
{
    return "testView";
}

QWidget *testView::widget()
{
    return m_widget;
}

void testView::close()
{
    m_widget->close();
    emit closed();
}


They are compliant with medAbstractView's API. Their widget is a simple QWidget showing the text "testView".


Now the test writing. We start by renaming testFoo() into testSingle() for more clarity. Here is the implementation of the test function testSingle():

void medViewContainersTestObject::testSingle()
{
    // create a view container single
    medViewContainerSingle *container = new medViewContainerSingle;
    container->setFixedSize(500, 500);
    container->show();

    // setup signal spies
    QSignalSpy spy1 (container, SIGNAL(viewAdded(dtkAbstractView*)));
    QSignalSpy spy2 (container, SIGNAL(viewRemoved(dtkAbstractView*)));
    QSignalSpy spy3 (container, SIGNAL(focused(dtkAbstractView*)));

    // create dummy view
    testView *view1 = new testView;

    // test setView:
    // - view should become visible
    // - container->view() should return view1
    // - syp1.count() should return 1
    container->setView(view1);

    QVERIFY(view1->widget()->isVisible());
    QCOMPARE(container->view(), view1);
    QCOMPARE(spy1.count(), 1);

    // test focus:
    // - container->current() should be itself
    // - spy3.count() should be 1
    container->setFocus (Qt::MouseFocusReason);

    QVERIFY (container->current()==container);
    QCOMPARE (spy3.count(), 1);

    // test null view
    container->setView ((dtkAbstractView*)NULL);
    QVERIFY(!view1->widget()->isVisible());
    QVERIFY(container->view()==NULL);

    // restore view1
    container->setView(view1);

    // create 2nd dummy view
    testView *view2 = new testView;

    // test view replacement:
    // - view1 should be hidden
    // - view2 should become visible
    // - spy1.count() should be 3
    // - spy2.count() should be 2
    // - container->view() should be view2
    container->setView(view2);

    QVERIFY(!view1->widget()->isVisible());
    QVERIFY(view2->widget()->isVisible());
    QCOMPARE(spy1.count(), 3);
    QCOMPARE(spy2.count(), 2);
    QCOMPARE(container->view(), view2);

    // test closing:
    // - view2 should be hidden
    // - spy2.count() should be 3
    // - container->view() should be null
    QMetaObject::invokeMethod(view2, "closing", Qt::DirectConnection);

    QVERIFY(!view2->widget()->isVisible());
    QCOMPARE(spy2.count(), 3);
    QCOMPARE(container->view(), (dtkAbstractView*)NULL);

    // cleanup before exiting test function
    delete container;
    delete view1;
    delete view2;
}


The first part:

medViewContainerSingle *container = new medViewContainerSingle;
    container->setFixedSize(500, 500);
    container->show();

creates a new medViewContainerSingle, set its size and display it.

Now, we setup signal spies that will count how many times a specific signal has been emitted during the test function execution. It will allow us to monitor if the number of signal emission is correct.

QSignalSpy spy1 (container, SIGNAL(viewAdded(dtkAbstractView*)));
    QSignalSpy spy2 (container, SIGNAL(viewRemoved(dtkAbstractView*)));
    QSignalSpy spy3 (container, SIGNAL(focused(dtkAbstractView*)));

Then, we just create a dummy view:

testView *view1 = new testView;

Now, we run a test:

container->setView(view1);

    QVERIFY(view1->widget()->isVisible());
    QCOMPARE(container->view(), view1);
    QCOMPARE(spy1.count(), 1);

We test setView() and checks that:

  • the view's widget is indeed visible
  • the container's view is the dummy view
  • the signal viewAdded() has been emitted once


The rest of the test is self-explanatory. Note the use of the macros:

  • QCOMPARE
  • QVERIFY

that, if the conditions are not met, will fail and output the line in the test where the fail occured. QCOMPARE has the advantage of outputing the actual and expected values.


Running the test

The test is launched by any of the following:

  • executing the binary 'medGuiTest.exe'
  • typing 'ctest' at the root of the build tree
  • type 'make tests' at the root of the build tree

'ctest' and 'make tests' will run all test suites of your project. Note that you can execute those commands directly from the build tree of the library / plugin to run only the test suite of that library / plugin.


One advantage of executing 'medGuiTest.exe' directly is that you can access the Qt command line. Just type in:

medGuiTest.exe medViewContainersTest -help

and you'll get:

 Usage: medViewContainersTest [options] [testfunction[:testdata]]...
    By default, all testfunctions will be run.

 options:
 -functions : Returns a list of current testfunctions
 -datatags  : Returns a list of current data tags.
              A global data tag is preceded by ' __global__ '.
 -xunitxml  : Outputs results as XML XUnit document
 -xml       : Outputs results as XML document
 -lightxml  : Outputs results as stream of XML tags
 -flush     : Flushes the results
 -o filename: Writes all output into a file
 -silent    : Only outputs warnings and failures
 -v1        : Print enter messages for each testfunction
 -v2        : Also print out each QVERIFY/QCOMPARE/QTEST
 -vs        : Print every signal emitted
 -random    : Run testcases within each test in random order
 -seed n    : Positive integer to be used as seed for -random. If not specified,
              the current time will be used as seed.
 -eventdelay ms    : Set default delay for mouse and keyboard simulation to ms milliseconds
 -keydelay ms      : Set default delay for keyboard simulation to ms milliseconds
 -mousedelay ms    : Set default delay for mouse simulation to ms milliseconds
 -keyevent-verbose : Turn on verbose messages for keyboard simulation
 -maxwarnings n    : Sets the maximum amount of messages to output.
                     0 means unlimited, default: 2000
 -nocrashhandler   : Disables the crash handler

 Benchmark related options:
 -callgrind      : Use callgrind to time benchmarks
 -tickcounter    : Use CPU tick counters to time benchmarks
 -eventcounter   : Counts events received during benchmarks
 -minimumvalue n : Sets the minimum acceptable measurement value
 -iterations  n  : Sets the number of accumulation iterations.
 -median  n      : Sets the number of median iterations.
 -vb             : Print out verbose benchmarking information.

 -help      : This help

Some useful functions:

  • '-maxwarnings 0': display all warning messages
  • '-vs': display all emitted signals

Example:

medGuiTests medViewContainersTest -vs -maxwarnings 0 testSingle

The above command will run the test function 'testSingle()' only, display all emitted signals and all debug messages.

Testing a plugin

Testing a plugin, or I should say a type (being a data, process, view, toolbox, etc.) in a plugin is not very different from testing a core class. The basics are the same, though there are a few specificities:

  • the plugin manager must be initialized with a correct plugin path. Indeed, plugins are discovered at runtime, and we need this.
  • some types like process or views can be fed with different data. It is desired to be able to reuse the same test code segment with different input data. Take the example of the image readers - when problematic images comes up, one finds a bug and fixes the reader. How can we be sure that previously supported images are still compatible with the updated reader? Qt has an answer for that, it's the testFoo_data() method.


Initializing the plugin manager

First of all, the environment variable MEDINRIA_PLUGIN_PATH should be defined and pointing to your actual plugin location. Having it defined in the medinria settings is not enough, it must be an environment variable. The reason is that tests do not inherit medinria's settings.


Second, we provide helper functions for tests in the medTest library. Include the medTest.h header and use:

medTest::initializePlugins();

in the initTestCase() method of your test. You are all set!


Testing with data

Qt documentation is well done for this. See:

http://qt-project.org/doc/qt-4.8/qtestlib-tutorial2.html


Example with the DICOM reader:

void medImageDataReaderTestObject::testRead_data()
{
    QTest::addColumn<QString>("filePath");
    QTest::addColumn<QString>("pixelType");
    QTest::addColumn<int>("xDimension");
    QTest::addColumn<int>("yDimension");
    QTest::addColumn<int>("zDimension");
    QTest::addColumn<qreal>("xSpacing");
    QTest::addColumn<qreal>("ySpacing");
    QTest::addColumn<qreal>("zSpacing");
    QTest::addColumn<QVector4D>("origin");
    QTest::addColumn<QMatrix4x4>("cosines");
    QTest::addColumn<int>("minRange");
    QTest::addColumn<int>("maxRange");

    QString file;
    medTest::envString ("MEDINRIA_DICOM_TEST_DATA_ROOT", file);

    QTest::newRow("chu_nice/1/1/1") << file + QDir::separator() + "chu_nice" + QDir::separator() + "1" + QDir::separator() + "1" + QDir::separator() + "1"
        << "short"
        << 512 << 512 << 331
        << 0.550781 << 0.550781 << 0.625
        << QVector4D (-141.0, -193.8, -111.25, 0.0)
        << QMatrix4x4()
        << -3024 << 3071;
}

void medImageDataReaderTestObject::testRead()
{
    QFETCH(QString, filePath);
    QFETCH(QString, pixelType);
    QFETCH(int, xDimension);
    QFETCH(int, yDimension);
    QFETCH(int, zDimension);
    QFETCH(qreal, xSpacing);
    QFETCH(qreal, ySpacing);
    QFETCH(qreal, zSpacing);
    QFETCH(QVector4D, origin);
    QFETCH(QMatrix4x4, cosines);
    QFETCH(int, minRange);
    QFETCH(int, maxRange);

    QDir dir(filePath);
    dir.setFilter(QDir::Files);

    QStringList fileList;
    if (dir.exists()) {
        QDirIterator directory_walker(filePath, QDir::Files, QDirIterator::Subdirectories);
        while (directory_walker.hasNext()) {
           fileList << directory_walker.next();
        }
    }

    QVERIFY(!fileList.isEmpty());

    dtkSmartPointer <dtkAbstractDataReader> reader;

    QList<QString> readers = dtkAbstractDataFactory::instance()->readers();
    for (int i=0; i<readers.size(); i++) {
        dtkSmartPointer <dtkAbstractDataReader> dataReader (dtkAbstractDataFactory::instance()->readerSmartPointer(readers[i]));
        dataReader->enableDeferredDeletion(false);
        if (dataReader->canRead( fileList )) {
            reader = dataReader;
            break;
        }
    }

    QVERIFY(!reader.isNull());
    QVERIFY(reader->read(fileList));

    dtkSmartPointer<dtkAbstractData> data = reader->data();
    data->enableDeferredDeletion(false);

    QVERIFY(!data.isNull());

    medAbstractDataImage *imData = qobject_cast<medAbstractDataImage*>(data);

    QCOMPARE (this->typeAsString(imData->pixelType()), pixelType);
    QCOMPARE (imData->xDimension(), xDimension);
    QCOMPARE (imData->yDimension(), yDimension);
    QCOMPARE (imData->zDimension(), zDimension);
    DTKCOMPARE (imData->xSpacing(), xSpacing, 3);
    DTKCOMPARE (imData->ySpacing(), ySpacing, 3);
    DTKCOMPARE (imData->zSpacing(), zSpacing, 3);
    QVERIFY (qFuzzyCompare(imData->origin(), origin)); // fuzzy compare is needed
    QVERIFY (norm(imData->directionCosines() - cosines)<1e-6); // more robust than QCOMPARE or fuzzy compare
    QCOMPARE (imData->minRangeValue(), minRange);
    QCOMPARE (imData->maxRangeValue(), maxRange);
}