Story-Based BDD - An alternative approach testing with Ember • BDD Testing

In this post we want to introduce you to a way, to test your Ember application: story-based BDD. BDD stand for Behaviour Driven Development. It combines the ideas of TDD (Test Driven Development) with DDD (Domain Driven Design).

At its core is an ubiquitous, domain-specific language (DSL) using natural language constructs, that serves to improve understanding and collaboration between non-technical people with the domain knowledge (business analysts, product owners etc.) and developers. This language is used to describe the intended behavior of the software as a series of examples (“specification by example”).

To give you a grasp on how that could look like, here is a real world example from one of our apps, a specification of how the cookie disclaimer we all learned to love should behave:

  Scenario: Show a cookie disclaimer
    When I go to the authentication page
    Then I should see a cookie disclaimer
    When I click the accept cookies button
    Then I should not see a cookie disclaimer

  Scenario: Hide accepted disclaimer
    Given I have accepted the cookie disclaimer
    When I go to the authentication page
    Then I should not see a cookie disclaimer

At the same time, by being a semi-formal parseable language, often formalized using theGherkin syntax as shown above, it serves as the foundation for implementing your (acceptance) tests: it provides the test specification, that describes what to test. Obviously that won’t be executable by itself, so the developer still has to provide the test implementation, which defines how to test it.

OK, NICE, BUT NOW SHOW ME SOME CODE

If that maybe sounded a bit too broad to you, and you wonder how that actually translates into executable tests, I can understand you. So let’s dive into some code. Of course we don’t want to start from scratch, implementing our tests based on this paradigm. But luckily that are a number of solid test frameworks for story-based BDD for various languages: there is Cucumber for various languages, including Ruby, Java and JavaScript, Behave for Python or Behat for PHP, to just name a few. And guess what, there is also an Ember addon ember-cli-yadda, that builds on top the JavaScript BDD library Yadda, a JavaScript implementation similar to Cucumber. Its responsibility is to integrate the parsing of Gherkin-based feature files, its execution based on the accompanying step files, and the integration of all that into the command you are all too familiar with: ember test.

Important to note here is that it will not replace all your testing stack with something completely new. To the contrary, it will run on top of either QUnit or Mocha (whatever you already use), and for the test implementation you continue to have all the testing tools from Ember and its ecosystem at your hands: async/await with the beautiful test helpers from @meber/test-helpers, ember-test-selectors, ember-cli-page-object, ember-cli-mirage, all covered…

Also you can write some tests the BDD-way, and others using the traditional approach using QUnit or Mocha directly. In fact, as BDD’s natural language capability is quite suitable for describing user stories, I think it is quite common to write acceptance tests in that way, and continue to use the more lower level test frameworks for the lower level test types, like unit or component integration tests. At least that’s what we commonly do, so I will focus on acceptance tests for the remainder of this blog post. But technically there is nothing that would prevent you from using this addon for any type of test.

But now, let’s finally get our hands dirty! As you can imagine, it all starts with this command:

ember install ember-cli-yadda

This will automatically add a few files to our project, some I will come back to a bit later.

MY FIRST FEATURE

For the purpose of this tutorial, we will implement some basic tests that will cover the error handling of our app. So let’s create our first BDD feature:

ember g feature error

This will generate the feature file tests/acceptance/error.feature, which holds our natural language test specification, along with its implementation in tests/acceptance/steps/error-steps.js.

Let’s replace the dummy feature file with something more meaningful. For testing that our handling of page not found errors works correctly, we could specify that as follows:

@setupApplicationTest
Feature: Error page

  Scenario: Error shown when visiting /foo

    Given I am an unauthenticated user
    When I visit /foo
    Then I should see the error message "Page not found."

So let’s quickly go through this line by line:

Line 1: @setupApplicationTest is a Yadda annotation, that already has a default implementation in tests/helpers/yadda-annotations.js, which will call the function with the same name, which you probably already know from your traditional tests, to setup all tests in this file as acceptance tests. There are a few other annotations pre-defined, like @setupRenderingTest, @setupTest and @ignore, but you are free to extend the existing implementations or introduce new annotations.
Line 2: This introduces our new feature, our error page
Line 4: A scenario is basically a test, which is written as an example what a user would typically do
Lines 6 – 8: these are the steps of this scenario, written in a natural language. The only formal requirement is that they start with one of the keywords Given, When or Then (or And and But, which just repeats the previous line)

You might wonder what happens when we run our tests now. Obviously this cannot really work, as our computer, dumb as it is, cannot know what all this actually means. And as such, when running your tests (either by ember t -s or ember s and opening http://localhost:4200/tests), you will see an Undefined step error, for all three of the above steps.

Obviously the framework is not capable of natural language processing, and has no machine learning algorithms implemented, although that would probably boost its marketing success considerably, so unfortunately we still have to implement all of this by ourselves. The frameworks “just” provides a mechanism to map these natural language constructs to actual concrete implementations.

So let’s implement our steps now. The first step is actually just a noop. In a real app, you would probably also have to implement a I am an authenticated user step, using the authentication layer of your app, e.g. ember-simple-auth. But for the sake of simplicity, we will ignore this for our tutorial here. So our tests/acceptance/steps/error-steps.jscould look like this now:

import steps from './steps';
import { expect } from 'chai';

export default function() {
  return steps()
    .given('I am an unauthenticated user', function() {
      // nothing to do here...
    })
  ;
}

We have added a new .given() definition to our steps dictionary. When you look at your test runner now, you will see that the first step does not show the Undefined step error anymore, as we have a matching step implementation for it now.

Note that I have been using Mocha here, along with Chai for its more expressive set of assertions, which allow you to also write regular tests in a more readable BDD style. But this is not a requirement at all, you can use QUnit just as well!

Let’s add the second step now:

import steps from './steps';
import { expect } from 'chai';
import { visit } from '@ember/test-helpers';

export default function() {
  return steps()
    .given('I am an unauthenticated user', function() {
      // nothing to do here...
    })
    .when('I visit /foo', async function() {
      await visit('/foo');
    })
  ;
}

MAKE THINGS REUSABLE

Technically these two steps are fine, but they fail at one very important point: they are not really reusable. But this is one of the benefits of BDD I especially appreciate: if done right, it not only encourages, but even enforces reusability of test implementation. So what’s wrong with our steps so far?

The first one, even as it literally does nothing, is not reusable, as it is defined in error-steps.js, which is only taken into account for the error.feature. But it’s easy to see that this step might be used in many different feature definitions as well. Luckily the solution is easy: as you can see we import another steps file steps.js, which is used exactly for this purpose: to hold all steps implementations that are reusable across features. Simply by moving the .given() step from error-steps.js to steps.js, we can reuse that step in any other feature.

The same can be said for the second step, but it suffers from another drawback: it has the URL to visit hardcoded. In a real app you would have dozens or even hundreds of different URLs you want to visit in tests, and you certainly don’t want to duplicate the code for each of them.

But we have you covered on this as well, by using so called parameterized steps: any occurrence of a word starting with $ is treated as a parameter. In fact you can even use complex regular expressions with matching groups to extract parameters. So our formerly hardcoded URL becomes a variable parameter like this, making this step reusable for any other feature or scenario:

    .when('I visit $url', async function(url) {
      await visit(url);
    })

Now we just have to implement our third step, which in this case is actually specific to the error feature, so we again add it to our error-steps.js:

import steps from './steps';
import { expect } from 'chai';
import { find } from '@ember/test-helpers';

export default function() {
  return steps()
    .then('I should see the $type message "$message"', function(messageType, messageText) {
      let selector = `[data-test-${messageType}-message]`;
      expect(find(selector)).to.have.trimmed.text(messageText);
    })
  ;
}

This will cover our last step I should see the error message "Page not found.", and again by using parameters it will be reusable for different messages and also different message types (besides errors). In case you are wondering about the trimmed.text()assertion: this is available by using the chai-dom Chai plugin. Again you can use QUnit as well, in this case I would recommend using qunit-dom for your DOM-based assertions. Also this example uses a data-test attribute selector to find the element that will hold the error message, powered by ember-test-selectors.

At this point, we should have all test steps implemented, so no more Undefined steperrors, and given our app is working correctly, the test should pass now! 🎉

ADD SUPPORT FOR EMBER-CLI-MIRAGE

Going forward with our “Page not found” example, we might want to add tests for cases where the route itself is known, but the dynamic model is not available, i.e. the API called by the route’s model hook return a 404 response.

This leads us to tests based around loading data with ember-data and ember-cli-mirage, which provide some good examples where the reusable character of yadda’s steps is again very helpful. So for most acceptance tests based around dynamic data you would have to define some seed data. Mirage’s factories provide a powerful concept for this, and together with the parameterized steps we just learned about, this allows to create some easy to use, generic and reusable steps. Here is a simple one:

    .given('there (?:are|is) ?([0-9]*) ([a-z-]+)(?: models)? in my database', function(count, model) {
      server.createList(singularize(model), parseInt(count) || 10);
    })

With this simple step implementation, all of the following Given steps would be available in any new feature specification, without requiring any further implementation:

    Given there are users in my database
    Given there is a blog-post in my database
    Given there are 5 products in my database

Given some advanced features like yadda converters and multiline steps with csv-based tabular data, even generic steps supporting overriding Mirage’s factories for certain properties are possible:

    Given there are the following products in my database:
    ---
    id | name  | price | vat-rate |
    1  | Foo   | 99    | 19       |
    2  | Bar   | 199   | 7        |
    ---

The implementation of this is skipped for now, an eventual follow-up blog post might go into more detail!

SETUP MIRAGE

What remains to be done here is to actually setup Mirage for all our acceptance tests. Mirage provides the setupMirage function for this, that should be called in our test’s beforeEach() hook. But how to do this, as ember-cli-yadda handles the tests setup and teardown logic (again based on either QUnit or Mocha) on its own. The solution are the already mentioned annotations. We declared our feature to use the setupApplicationTest() function (provided by either ember-qunit or ember-mocha) by using the @setupApplicationTest annotation. What this actually does is specified in your project’s tests/helpers/yadda-annotations.js file. ember-cli-yaddaautomatically installs a default implementation of these annotation, but as this file is under your own control as part of your repo, you can customize it at your will. Here we could introduce a new annotation like @setupMirage to call the setup function, or just extend the existing handling of the @setupApplicationTest annotation to call setupMirage() for every application test in our test suite. We choose the latter option now: the setupYaddaTest() hook gets called for every scenario to return a function that’s responsible to setup the scenario’s test. So instead of the default implementation, that just delegates to setupApplicationTest()…

function setupYaddaTest(annotations) {
  if (annotations.setupapplicationtest) {
    return setupApplicationTest;
  }
  // ...
}

… we add the setupMirage() call:

import setupMirage from 'ember-cli-mirage/test-support/setup-mirage';
// ...
function setupYaddaTest(annotations) {  
  if (annotations.setupapplicationtest) {
    return function() {
      let hooks = setupApplicationTest();
      setupMirage(hooks);
    }
  }
  // ...
}

SCENARIO OUTLINES

Having laid out the basics to test with Mirage, let’s get back to the previous example of testing our error page. A test specification for a not-found error due to a non-existing model could look like this:

  Scenario: Error shown when visiting /blog/2

    Given I am an unauthenticated user
    When I visit /blog/123
    Then I should see the error message "Page not found."

As we already have the necessary steps implemented, and we don’t have a step that creates the blog-post model (here with ID 123), this should already work and correctly test the error handling of our application. What is annoying however is that you would have to duplicate this scenario for every URL you want to test, that is supposed to render an error page. In fact the only difference compared to the first error scenario mentioned at the beginning of this blog post is the URL in the visit step (here /blog/123 instead of /foo).

But there is another powerful solution available: kind-of similar to parameterized steps, we can parameterize whole scenarios with variadic data, using so called example tables:

  Scenario: Error shown when visiting [url]

    Given I am an unauthenticated user
    When I visit [url]
    Then I should see the error message "Page not found."

  Where:

  | url       |
  | /foo      |
  | /blog/123 |

We just unified both separate but almost identical scenarios into one scenario outline, where the difference is given by the example table following the Where keyword. The first line specifies the parameter’s name, while all following lines provide the actual data, which generates a new scenario variant with the given data. Any reference to a parameter in brackets (like [url] here) will get replaced with the actual data.

This is useful to write similar tests with differing data, for example to cover different edge cases, without requiring any duplication. In our case we can use this to add coverage for even more aspects of our error handling:

  Scenario: Error shown when visiting [url]

    Given I am an [authstatus] user
    When I visit [url]
    Then I should see the error message "[message]"

  Where:

  | authstatus      | url          | message         |
  | unauthenticated | /foo         | Page not found. |
  | unauthenticated | /blog/123    | Page not found. |
  | authenticated   | /private/foo | Page not found. |
  | unauthenticated | /private/foo | Access denied.  |
  | unauthenticated | /private     | Access denied.  |

CONCLUSIONS

The idea that product owners or other non-technical people can write usable BDD feature specifications sounds appealing, but honestly never worked out for us. Maybe we didn’t try hard enough, I don’t know. Would love to hear from someone where that worked out.
Nevertheless as a tech guy I still see a benefit in having a much better readable test specification, e.g. to easily grasp at a code review what kind of functionality a given PR is supposed to deliver.
Even when working on a feature myself, I find it useful to first write the acceptance test specification in prose. Sometimes I just find it difficult to do a thorough test-first approach for a more complex new feature. In theory you would implement the test first, and then work on the actual functionality until all tests are green. For tests with a limited scope (component or unit tests), that often works fine. But for more complex acceptance tests this is not always the case, at least for me. Sometimes you aren’t yet sure how a given feature should be implemented in detail, so it’s difficult to write the test for it before that. And you can’t really be sure the test implementation actually remotely works. Say you introduce a new route, the test will immediately fail on the first await visit() call as the route is not yet known. So whether all the following test code actually works remains unclear. With story-based BDD you can at least write the test specification first, which helps you to get a better understanding of what needs to be done, what edge cases to cover etc. The implementation of that test can be done later, in sync with the actual implementation or even afterwards.
As shown in this post, it is very easy to produce reusable test code. In fact BDD does not only support that, in a sense it does this automatically for you, by having shared and parameterized test steps. Often enough you don’t even have to implement anything for a new test, when all the steps are implemented already.
A word of caution though: by leaving the mainstream, or the “safety of the herd” you are taking a risk that you should be aware of, which is that you might have to invest into the maintenance of those more exotic libraries (like ember-cli-yadda in this case), when others are not able to do this in time. At least this was the case for me, when the awesome new Ember testing APIs (RFC268 in particular) were implemented in the default ember-qunit, but were not yet supported in ember-mocha and ember-cli-yadda. So I had to invest some time and contribute a bunch of PRs (if you are interested: ember-mocha: #190, #192, #199, #200, ember-cli-yadda: #65, #70, #71 ). But that was a bigger change that should not happen too often, so I think that risk is absolutely justifiable, but nevertheless you have been warned! 😉

https://www.kaliber5.de/en/blog/story-based-bdd-an-alternative-approach-to-testing-with-ember/

BDD Testing