Behat Testing on Staging & Vagrant

Behat Testing on Staging & Vagrant
  • Behat configuration on staging
  • What is behat.yml
  • How to write a feature
  • How to write a context
  • How to run Behat tests (Manually & Jenkins)
  • Test step execution results
  • Behat Selenium Driver

Behat configuration on staging


First of all, check both php and java versions by the following commands.

[chunxinxu@stage-02 ~]$ php --version
PHP 5.3.3 (cli) (built: Feb  9 2016 10:36:17) 
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies


[chunxinxu@stage-02 ~]$ java -version
java version "1.6.0_38"
OpenJDK Runtime Environment (IcedTea6 1.13.10) (rhel-
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

Then make sure we have downloaded and run Selenium Server. Remember to download the selenium jar file and update its version in the command below.

[chunxinxu@stage-02 ~]$ java -jar selenium-server-standalone-2.44.0.jar 
09:16:23.445 INFO - Launching a standalone server

Install Composer

Composer is a PHP dependency manager that will make sure all the pieces you need get installed.

curl -sS | php

Create a folder /opt/drupalextension to place all the behat config files

Create a file called composer.json and include the following:

  "require": {
    "drupal/drupal-extension": "~3.0",
    "phpunit/phpunit": "4.8.*"
  "config": {
    "bin-dir": "bin/"

Install behat and its extensions by composer. And make both composer and behat system-wide.

> mv composer.phar /usr/local/bin/composer
> composer install
> ln -s /opt/drupalextension/bin/behat /usr/local/bin/behat

Test that your install worked by typing the following. If you were successful, you’ll see the help output.

bin/behat --help 

Now we have behat and its extensions installed. Next step is to write up some features and their Context functions. We leave behat core and its extensions in /opt/drupalextension folder which is outside Phoenix folder. However, it's better to keep behat features inside Phoenix so that they can share the same git repo. You may find all *.feature files in /var/www/phoenix/custom/behat directory. The structure can be found the image below.

What is behat.yml

behat.yml is a configuration file for behat tests. Wherever you make your test folder (in our staging, the folder is /var/www/phoenix/custom/behat), inside it create the behat.yml file. You can find 2 variations in /custom/behat folder: behat-vagrant.yml and behat-staging.yml. Because we have different environments on local vagrant and staging. 2 separated yml files can solve the multiple test environments issue. We just need point out which yml to use when run behat command, like "behat --config behat-staging.yml".


    default:   # default is the suite name. We can create whatever name we want.
      contexts:   # List contexts we need for this test suite.
        - FeatureContext
        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MinkContext
        - Drupal\DrupalExtension\Context\MessageContext
        - Drupal\DrupalExtension\Context\DrushContext
  extensions:   # Define extensions
      goutte: ~
      selenium2: ~
      base_url:   # Tell behat which website we want to test.
      blackbox: ~
      api_driver: 'drush'   # Tell behat which api driver we want to use. 2 options: drupal & drush
        alias: 'local'   # If drush alias is set, define it here.
        root: '/var/www/phoenix'   # Tell behat where to run drush commands.
        drupal_root: '/var/www/phoenix'   # Define root directory for Drupal.

How to write a feature

Behat is a tool to test the behavior of your application, described in a special language called Gherkin. Gherkin serves as your project’s documentation as well as your project’s automated tests. Like YAML and Python, Gherkin is a whitespace-oriented language that uses indentation to define structure. Most lines in Gherkin start with a special keyword.

  1. Feature: Some terse yet descriptive text of what is desired. 
  2. Scenario: Some determinable business situation. 
  3. Scenario step definition starts with keywords: Given, And, When, But and Then.

Every *.feature file conventionally consists of a single feature. Lines starting with the keyword "Feature:". A feature usually contains a list of scenarios. You can write whatever you want up until the first scenario, which starts with "Scenario:". You can use tags, like "@atd" to group features and scenarios together, independent of your file and directory structure.

Feature: Content Management
  In order to check the permissions on ATD new order screen
  As a staff user or an anonymous user
  I should / should not create a new order

  Scenario: I login as a developer I have permission to see
    Given I am logged in as test staff behat2
    When I go to "/phx/orders/new"
    Then I should see "No actions required for this order"

How to write a context

From the feature above, we can see a step definition "Given I am logged in as test staff behat2". In fact, the format of this step is "Given I am logged in as test staff :name". Developers need to tell Behat what to do towards this step. So we need to write a callback function for this. In behat, it recognises callback function through its comments, like below. We need write the comments exactly like "@Given I am logged in as test staff :name" so that the step definition will point to the function assertLoggedInByTestStaffName().  We can put all these functions in FeatureContext.php or create a new CustomContext.php for the tests.

   * @Given I am logged in as test staff :name
  public function assertLoggedInByTestStaffName($name) {
    if (!isset($this->users[$name])) {
      throw new \Exception(sprintf('No staff user with %s name is registered with the driver.', $name));
    // Change internal current user.
    $this->user = $this->users[$name];
    // Login.

How to run Behat tests

Basically, we run Behat tests automatically by Jenkins. We may set it run period or after git pull. We can also run it manually on staging by the following:

cd /var/www/phoenix/custom/behat
behat --config behat-staging.yml

As mentioned above, we can group scenarios by tags. So we can run some tests with specific tags.

All tests with a certain tag (using @ symbol)

behat --tags @tagname

All tests except a certain tag (omitting @ symbol)

behat --tags ~tagname

We can also run with multiple tags:
# Run tests tagged with @tagname1 OR @tagname2

behat --tags "@tagname1,@tagname2"

# Run tests tagged with both @tagname1 AND tagname2 (showing you can also use the equals sign after "--tags" if you prefer)

behat --tags="@tagname1&&@tagname2"

# Run tests tagged with @tagname1 AND @tagname2 AND NOT @tagname3

behat --tags "@tagname1&&@tagname2&&~@tagname3"

Test step execution results

How can we tell what exactly “failed” or “passed” when executing a step? And how does Behat actually check that a step executed properly? For that, we have step execution types. Behat differentiates between seven types of step execution results: “Successful Steps”, “Undefined Steps”, “Pending Steps”, “Failed Steps”, “Skipped Steps”, “Ambiguous Steps” and “Redundant Step Definitions”.

Successful Steps: If the definition method does not throw any Exception, the step is marked as successful (green).

Undefined Steps: When Behat cannot find a matching definition, the step is marked as undefined, and all subsequent steps in the scenarios are skipped.

Pending Steps: When a definition method throws aBehat\Behat\Tester\Exception\PendingException exception, the step is marked as pending, reminding you that you have work to do.

Failed Steps: When a definition method throws any Exception (exceptPendingException) during execution, the step is marked as failed.

Skipped Steps: Steps that follow undefinedpending or failed steps are never executed, even if there is a matching definition.

Ambiguous Steps: When Behat finds two or more definitions that match a single step, this step is marked as ambiguous.

Redundant Step Definitions: Behat will not let you define a step expression’s corresponding pattern more than once.

Behat Selenium Driver

Selenium2Driver provides a bridge for the Selenium2 tool. The Selenium2Driver has been installed on both vagrant and staging. But if you want to install it manually, do it with composer like below.

$ composer require behat/mink-selenium2-driver

In order to talk with selenium server, you should install and configure it first:

  1. Download the Selenium Server from the project website.
  2. Run the server with the following command (update the version number to the one you downloaded):
$ java -jar selenium-server-standalone-2.53.0.jar

You may have some problems to switch on selenium server if your java version is too low. Like it on staging, the JAVA version used to be 1.5 and it needs to be upgraded to 1.7. There are plenty of reference on how to upgrade JAVA online, so no repeating here.

Now you get selenium server on. When you run your behat feature tests, you may still have various problems dependent on your own system. Here is some clues to check:

  1. Check Firefox installed with latest version: >sudo yum install firefox
  2. Make correct DISPLAY variable is used: >export DISPLAY=localhost:0.0
  3. Make sure xvfb running: >Xvfb &
  4. Make dbus library run correctly by bash command: >dbus-uuidgen>/var/lib/dbus/machine-id




blog tag: