Unit testing is a methodology where units of code are tested in isolation from the rest of the application. A unit test might test a particular function, object, class, or module. Unit tests are great to learn whether or not individual parts of an application work. NASA had better know whether or not a heat shield will work before they launch the rocket into space.<\/p>\n\n\n\n
But unit tests don\u2019t test whether or not units work together when they\u2019re composed to form a whole application. For that, you need integration tests, which can be collaboration tests between two or more units, or full end-to-end functional tests of the whole running application (aka system testing). Eventually, you need to launch the rocket and see what happens when all the parts are put together.<\/p>\n\n\n\n
There are multiple schools of thought when it comes to system testing, including Behavior Driven Development (BDD), and functional testing.<\/p>\n\n\n\n
Behavior Driven Development<\/strong> (BDD) is a branch of Test Driven Development (TDD). BDD uses human-readable descriptions of software user requirements as the basis for software tests. Like Domain Driven Design (DDD), an early step in BDD is the definition of a shared vocabulary between stakeholders, domain experts, and engineers. This process involves the definition of entities, events, and outputs that the users care about, and giving them names that everybody can agree on.<\/p>\n\n\n\n BDD practitioners then use that vocabulary to create a domain specific language they can use to encode system tests such as User Acceptance Tests (UAT).<\/p>\n\n\n\n Each test is based on a user story written in the formally specified ubiquitous language based on English. (A ubiquitous language is a vocabulary shared by all stakeholders.)<\/p>\n\n\n\n A test for a transfer in a cryptocurrency wallet might look like this:<\/p>\n\n\n\n Notice that this language is focused exclusively on the business value that a customer should get from the software rather than describing the user interface of the software, or how the software should accomplish the goals. This is the kind of language you could use as input for the UX design process. Designing these kinds of user requirements up front can save a lot of rework later in the process by helping the team and customers get on the same page about what product you\u2019re building.<\/p>\n\n\n\n From this stage, there are two paths you can venture down:<\/p>\n\n\n\n Either way, it\u2019s generally a good idea to treat your tests as black box tests, meaning that the test code should not care about the implementation details of the feature you\u2019re testing. Black box tests are less brittle than white box tests because, unlike white box tests, black box tests won\u2019t be coupled to the implementation details, which are likely to change as requirements get added or adjusted, or code gets refactored.<\/p>\n\n\n\n Proponents of BDD use custom tools such as Cucumber<\/a> to create and maintain their custom DSLs.<\/p>\n\n\n\n For contrast, proponents of functional tests generally test functionality by simulating user interactions with the interface and comparing the actual output to the expected output. In web software, that typically means using a test framework which interfaces with the web browser to simulate typing, button presses, scrolling, zooming, dragging, etc, and then selecting the output from the view.<\/p>\n\n\n\n I typically translate user requirements into functional tests rather than keep up BDD tests, mostly because of the complexity of integrating BDD frameworks with modern applications, and the cost of maintaining custom, English-like DSL whose definitions may end up spanning several systems, and even several implementation languages.<\/p>\n\n\n\n I find the layman-readable DSL useful for very high-level specifications as a communications tool between stakeholders, but a typical software system will require orders of magnitude more low-level tests in order to produce adequate code and case coverage to prevent show-stopping bugs from reaching production.<\/p>\n\n\n\n In practice, you have to translate \u201cI transfer $20 to my friend\u201d<\/em> into something like:<\/p>\n\n\n\n A layer below that, you\u2019re maintaining state for the \u201ctransfer money\u201d workflow, and you\u2019ll want unit tests that ensure that the correct amount is being transferred to the correct wallet address, and a layer below that, you\u2019ll want to hit the blockchain APIs to ensure that the wallet balances were actually adjusted appropriately (something that the client may not even have a view for).<\/p>\n\n\n\n These different testing needs are best served by different layers of tests:<\/p>\n\n\n\n I have never met a layman stakeholder who is remotely aware of all of the functional tests verifying even the top-most level UI behavior, let alone one who cares about all of the lower level behaviors. Since laymen are not interested, why pay the cost of maintaining a DSL to translate for them?<\/p>\n\n\n\n Regardless of whether or not you practice the full BDD process, it has a lot of great ideas and practices we should not lose sight of. Specifically:<\/p>\n\n\n\n Another approach to system testing is functional testing.<\/p>\n\n\n\n The term \u201cfunctional testing\u201d can be confusing because it has had several meanings in software literature.<\/p>\n\n\n\n IEEE 24765<\/a> gives two definitions:<\/p>\n\n\n\n 1. testing that ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions [i.e., black box testing]<\/p><\/blockquote>\n\n\n\n 2. testing conducted to evaluate the compliance of a system or component with specified functional requirements<\/p><\/blockquote>\n\n\n\n The first definition is general enough to apply to almost all popular forms of testing, and already has a perfectly suitable name that is well understood by software testers: \u201cblack box testing\u201d. When I\u2019m talking about black box testing, I\u2019ll use that term, instead.<\/p>\n\n\n\n The second definition is usually used in contrast to testing that is not directly related to the features and functionality of the app, but instead concentrates on other characteristics of the app, such as load times, UI response times, server load testing, security penetration testing, and so on. Again, this definition is too vague to be very useful on its own. Usually, we want to get more specific about what kind of testing we\u2019re doing, e.g., unit testing, smoke testing, user acceptance testing?<\/p>\n\n\n\n For those reasons, I prefer another definition that has been popular recently. IBM\u2019s Developer Works says:<\/p>\n\n\n\n Functional tests are written from the user\u2019s perspective and focus on system behavior that users are interested in.<\/p><\/blockquote>\n\n\n\n That\u2019s a lot closer to the mark, but if we\u2019re going to automate tests, and those tests are going to test from the user\u2019s perspective, that means we\u2019ll need to write tests which interact with the UI.<\/p>\n\n\n\n Such tests can also go by the names \u201cUI testing\u201d or \u201cE2E testing\u201d, but those names don\u2019t replace the need for the term \u201cfunctional tests\u201d because there is a class of UI tests which test things like styles and colors, which are not directly related to user requirements like \u201cI should be able to transfer money to my friend\u201d.<\/p>\n\n\n\n Using \u201cfunctional testing\u201d to refer to testing the user interface to ensure that it fulfills the specified user requirements is usually used in contrast to unit testing, which is defined as:<\/p>\n\n\n\n the testing of individual units of code (such as functions or modules) in isolation from the rest of the application<\/p><\/blockquote>\n\n\n\n In other words, while a unit test is for testing individual units of code (functions, objects, classes, modules) in isolation from the application, a functional test is for testing the units in integration with the rest of the app, from the perspective of the user interacting with the UI.<\/p>\n\n\n\n I like the classification of \u201cunit tests\u201d for developer-perspective code units, and \u201cfunctional tests\u201d for user-perspective UI tests.<\/p>\n\n\n\n Unit tests are typically written by the implementing programmer, and test from the programmer\u2019s perspective.<\/p>\n\n\n\n Functional tests are informed by the user acceptance criteria and should test the application from the user\u2019s perspective to ensure that the user\u2019s requirements are met. On many teams, functional tests may be written or expanded on by quality engineers, but every software engineer should be aware of how functional tests are written for the project, and what functional tests are required to complete the \u201cdefinition of done\u201d for a particular feature set.<\/p>\n\n\n\n Unit tests are written to test individual units in isolation from the rest of the code. There are two major benefits to this approach:<\/p>\n\n\n\n Functional tests on the other hand:<\/p>\n\n\n\n Functional tests without unit tests can never provide deep enough code coverage to be confident that you have an adequate regression safety net for continuous delivery. Unit tests provide code coverage depth. Functional tests provide user requirement test case coverage breadth.<\/p>\n\n\n\n Functional tests help us build the right product. (Validation) You need both.<\/p>\n\n\n\n Note: See Validation vs Verification<\/a>. Build the right product vs build the product right distinction was succinctly described by Barry Boehm<\/a>.<\/p><\/blockquote>\n\n\n\n There are lots of frameworks that allow you to create functional tests for web applications. Many of them use an interface called Selenium<\/a>. Selenium is a cross-platform, cross-browser automation solution created in 2004 which allows you to automate interactions with the web browser. The trouble with Selenium is that it is an engine external to the browsers which relies on Java, and getting it to work together with your browsers can be harder than it needs to be.<\/p>\n\n\n\n More recently, a new family of products has popped up which integrate far more smoothly with browsers with fewer pieces to worry about installing and configuring. One of those solutions is called TestCafe. It\u2019s the one that I currently use and recommend.<\/p>\n\n\n\n Let\u2019s write a functional test for the TDD Day website<\/a>. First, you\u2019ll want to create a project for it. In a terminal:<\/p>\n\n\n\n Now we\u2019ll need to add a You can run the tests by typing Create a new file at TestCafe automatically makes the Now you can select from the page and make assertions using the TestCafe will launch the Chrome browser, load the page, wait for the page to load, and wait for your selector to match a selection. If it doesn\u2019t match anything, the test will eventually time out and fail. If it does match something, it will check the actual selected value against the expected value, and the test will fail if they don\u2019t match.<\/p>\n\n\n\n TestCafe provides methods to test all sorts of UI interactions, including clicking<\/a>, dragging<\/a>, typing text<\/a>, and so on.<\/p>\n\n\n\n TestCafe also supplies a rich selector API<\/a> to make DOM selections painless.<\/p>\n\n\n\n Let\u2019s test the registration button to ensure that it navigates to the correct page on click. First, we\u2019ll need a way to check the current page location. Our TestCafe code is running in Node, but we need it to run in the client. TestCafe supplies a way for us to run code in the client. First, we\u2019ll need to add Now we can use it to test the window location:<\/p>\n\n\n\n If you\u2019re not sure how to do what you\u2019re trying to do, TestCafe Studio<\/a> lets you record and replay tests. TestCafe Studio is a visual IDE for interactively recording and editing functional tests. It\u2019s designed so that a test engineer who may not know JavaScript can build a suite of functional tests. The tests it generates automatically await asynchronous jobs like page loads. Like the TestCafe engine, TestCafe Studio can produce tests which can be run concurrently across many browsers, and even remote devices.<\/p>\n\n\n\n TestCafe Studio is a commercial product with a free trial. You do not need to purchase TestCafe studio to use the open source TestCafe engine, but the visual editor with built-in recording features is definitely a tool worth exploring to see if it\u2019s right for your team.<\/p>\n\n\n\n TestCafe has set a new bar for cross-browser functional testing. Having endured many years of trying to automate cross-platform tests, I\u2019m happy to say that there is finally a fairly painless way to create functional tests, and there is now no good excuse to neglect your functional tests, even if you don\u2019t have dedicated quality engineers to help you build your functional test suite.<\/p>\n\n\n\nStory: Transfers change balances<\/code><\/pre>\n\n\n\nAs a wallet userIn order to send moneyI want wallet balances to update<\/code><\/pre>\n\n\n\nGiven that I have $40 in my balanceAnd my friend has $10 is their balanceAnd I transfer $20 to my friendWhen the transfer is completeThen I should have $20 in my balanceAnd my friend should have $30 in their balance.<\/code><\/pre>\n\n\n\n[Send money]<\/code><\/li>What is Functional Testing?<\/h3>\n\n\n\n
Unit Tests vs Functional Tests<\/h3>\n\n\n\n
Unit tests help us build the product right. (Verification)<\/p><\/blockquote>\n\n\n\nHow to Write Functional Tests for Web Applications<\/h3>\n\n\n\n
mkdir tdddaycd tdddaynpm init -y # initialize a package.jsonnpm install --save-dev testcafe<\/code><\/pre>\n\n\n\n\"testui\"<\/code> script to our package.json<\/code> in the scripts<\/code>block:<\/p>\n\n\n\n{
\"scripts\": {
\"testui\": \"testcafe chrome src\/functional-tests\/\"
}
\/\/ other stuff...
}<\/pre>\n\n\n\nnpm run testui<\/code>, but there aren’t any tests to run yet.<\/p>\n\n\n\nsrc\/functional-tests\/index-test.js<\/code>:<\/p>\n\n\n\nimport { Selector } from 'testcafe';<\/pre>\n\n\n\nfixture<\/code> and test<\/code> functions available. You can use fixture<\/code> with the tagged template literal syntax to create titles for groups of tests:<\/p>\n\n\n\nfixture `TDD Day Homepage`
.page('https:\/\/tddday.com');<\/pre>\n\n\n\ntest<\/code> and Select<\/code> functions. When you put it all together, it looks like this:<\/p>\n\n\n\nimport { Selector } from 'testcafe';<\/pre>\n\n\n\nfixture `TDD Day Homepage`
.page('https:\/\/tddday.com');<\/pre>\n\n\n\ntest('Page should load and display the correct title', async t => {
const actual = Selector('h1').innerText;
const expected = 'TDD DAY 2019';
await t.expect(actual).eql(expected);
});<\/pre>\n\n\n\nClientFunction<\/code> to our import line:<\/p>\n\n\n\nimport { Selector, ClientFunction } from 'testcafe';<\/pre>\n\n\n\nconst getLocation = ClientFunction(() => window.location.href);<\/pre>\n\n\n\n
test('Register button should navigate to registration page',
async t => {
\/\/ Flexible selectors let us select arbitrary things on the page,
\/\/ regardless of how the page was marked up.
const registerButton = Selector('span').withText('REGISTER NOW');
const expected =
'https:\/\/zoom.us\/webinar\/register\/WN_rYdjYdXFTPiHCsiWsnq0jA';<\/pre>\n\n\n\n \/\/ Wait for the button click navigation
await t.click(registerButton);<\/pre>\n\n\n\n \/\/ Now check the location.
await t.expect(getLocation())
.eql(expected);
});<\/pre>\n\n\n\nDos and Don\u2019ts of Functional Tests<\/h3>\n\n\n\n