Dienstag, 5. Juni 2012

symfony2 Test Database Best Pratice

When you build a reliable testsuite, you get to a point where it gets really slow. Unittests may be fast, but then the functional tests get executed and it takes time to set up clean databases and a clean environment before a test can be executed. With over 900 tests, one of our projects took way over 10 minutes to build on our Jenkins server. You probably know that XP teaches to have a 10 minute build. So what to do?

In this post, I'll first explain our setup before the change. It was pretty good and might be usefull to many people. Afterwards I'll cover the changes we did and what happened.

Using sqlite for tests

In my most popular blog post to date I showed how to set up the test database for every test class. This was the first step to have tests which not depend on external stuff like a database which is up to date and filled with correct data. With only this in place, you still have two problems:
  • It takes a huge amount of time to delete all tables, create new ones and then execute the fixtures. If you have a big application with many tables and many fixtures, the time really gets an issue
  • You need to make sure the database is set up and the credentials are correct, so you are not completly independend.
Let's get the second one right first. You might be using sqlite already as the symfony2 profiler uses it. So why not use sqlite instead of your normal database when executing the tests (I took this idea from this blog post)? As it's only one file without credentials and can be set up on-the-fly, you only have to make sure sqlite is installed on the machine. You can use sqlite by adding this to your config_test.yml:

doctrine
    dbal: 

        driver: pdo_sqlite 
        path: %kernel.cache_dir%/test.db 
        charset: UTF8

Now when you run the tests, a file named test.db is created inside your cache directory and you can use it. This shouldn't break any of your tests if you setup your database prior to your tests. If you clear your cache for any reason while the tests gets executed, you can also store the file in some temporary folder.

This was what we did. When the execution of the test suite took to long, we searched for a way to make it faster. As mentioned above, the drop, create and fixtures:load take a huge amount of time, so getting rid of it will have a huge impact on the testsuite run time.

A copy might be faster!

I remember reading about an approach where you create a sqlite file, copy it somewhere and than overrides it every time you want to "reset" the database. This is clever, as dropping, creating and adding the fixtures is way slower than just copying one file of a few hundred kb.

To do this, you need your own bootstrap, as you wanna create and backup the database whatever test comes first and then just replace the database. So I wrote a very simple bootstrap.php:

<?php

require_once __DIR__ . '/bootstrap.php.cache';

require_once __DIR__ . '/AppKernel.php';

use Symfony\Bundle\FrameworkBundle\Console\Application;
use Symfony\Component\Console\Input\ArrayInput;

$kernel = new AppKernel('test', true); // create a "test" kernel
$kernel->boot();

$application = new Application($kernel);
$application->setAutoExit(false);

deleteDatabase();
executeCommand($application, "doctrine:schema:create");
executeCommand($application, "doctrine:fixtures:load");

backupDatabase();

function executeCommand($application, $command, Array $options = array()) {
    $options["--env"] = "test";
    $options["--quiet"] = true;
    $options = array_merge($options, array('command' => $command));

    $application->run(new ArrayInput($options));
}

function deleteDatabase() {
    $folder = __DIR__ . '/cache/test/';
    foreach(array('test.db','test.db.bk') AS $file){
        if(file_exists($folder . $file)){
            unlink($folder . $file);
        }
    }
}

function backupDatabase() {
    copy(__DIR__ . '/cache/test/test.db', __DIR__ . '/cache/test/test.db.bk');
}

function restoreDatabase() {
    copy(__DIR__ . '/cache/test/test.db.bk', __DIR__ . '/cache/test/test.db');


Let's make it quick: Use the old bootstrap.php.cache, but also boot the kernel, delete the old databases (by deleting the sqlite files) and create it again including all fixtures. In your phpunit.xml, you need to change the bootstrap parameter to the filename you choose for this file (mine is bootstrap.php).

In your test class (maybe the abstract class described by me?) instead of running all the commands, you simply have to call restoreDatabase(). That's it, you're database is fresh as new!

Results, please

Our goal was to make the tests faster. On my virtual machine as well as on Jenkins, the build is much faster. It takes 9 minutes now on Jenkins including all the other stuff going on (pdepend, phpci and so on). On my virtual machine, the tests take 7 minutes.

If I only run unit tests which don't need a database, the execution is way slower, as it generates the database nontheless. On the other hand, reseting the database once more now is very cheap (only one file copy), so adding new test classes does not add much time.

Another plus is that our testclasses are not clutteret with code to set up databases. The global function restoreDatabase simply does what is needed and the tests don't care anymore.

I guess there are more ways to tune your tests. But this has to wait till the suite grows and executes slower than 10 minutes.

Keine Kommentare:

Kommentar veröffentlichen