What do you want to test today

Why, when and what to test

bit.ly/dckyiv-test

Slides are hosted on github, press 's' if you want to read speaker notes too.

About myself

Valery "valthebald" Lourie

Core mentoring coordinator

Agenda

Why do we care about quality
What affects the quality
Before tests
Test types depending on project
Quality vs budget
Q & A

Why

Pride
Reputation
Balance quality and budget

Cost of poor quality

Customer support specialist costs
Solution costs
Brand impact costs

Quality KPI

Mean time to failure
Build stability
Defect density
Customer complains
Customer satisfaction

Testing the testing

Defect density detected by automated tests
Phase based defect removal
Defect removal effectiveness
Fix backlog - BMI
Fix response time

Static analysis

Possible bugs
Overcomplicated expressions
Unused and unreachable code

Setting up PHPStorm

Test types

Unit
Integration
Functional
Performance
Smoke

What is web application

Before going into details of testing, let's understand what we are testing.
        In its basic form, web application exists in the following context:
        User agent (browser, mobile application, other application types) sends request to web server
        for specific resource (page, script, media file).
        Web server either serves request directly (i.e. static files from the file system, cacheable pages
        from web server cache), or bypasses request to application.
        Application uses internal logic to build the response, which travels back to user agent through web server

What is web application

Most non-trivial applications use some kind of database to store data.
        When it takes significant resources to build response or its certain parts on every request,
        it makes sense that application caches responses (or parts of it).
        It also becomes more and more common that application uses external resources.

What is web application

Real picture can be more complex, if you add CDN, load balancers.
      Which brings up the most important question.

You test what you are responsible for

(Optionally) load balancer
Web server
Application
External connectors (DB, search servers, web services etc.)

The whole picture is too complex. Let's try to focus on what is under our control.

What is your project?

Core - unit
Contrib - unit and integration
Client project - functional

Application in detail

        Let's take a closer look at a typical Drupal application.
        Combined, the data that it specific for the project (custom code, theme + configuration)
        are less than 2% of the project.
        But... the client does not care what part of the project is written by you.
        You need to ensure that:
        - your custom code behaves well
        - your custom theme displays everything well
        - used modules are configured, do not conflict one with another and the core
        By this simple chart you can see that integration checks are much more important than
        correct functioning of each subsystem (also, do not forget that core has very good test coverage)

Functional testing tools

PHPUnit
Nightwatch
Behat
Selenium

What the client cares about?

Overall performance
End user experience
Site builder experience

Nothing in this list implies any knowledge about application structure.
      No matter what issue tracking software we use (JIRA, Redmine, Github, Gitlab),
        issue is an atom of client requirements.

Behat scenarios

Written in Gherkin - close to natural English
Actions are methods of PHP classes
Has a wide range of ready, reusable snippets

Smoke test: administrative permissions

      As a site editor
      I can use administrative pages.

      As anonymous user or user without site editing permissions
      I get access denied when attempting to enter administrative pages.

      Behat solution:
      1. Go to /admin as anonymous user
      2. Check that application returns 'access denied'
      3. Login as site editor
      4. Check that application returns administration page

You should never, ever have regressions in this check, right?

Behat feature

@javascript
Feature: Check that administrator has access to the content list
Scenario:
      Given that I am a user with administrator role
      And I visit page /admin/content
      Then I should see `Next page` link

      Given that I am user with anonymous role
      And I visit page /admin/content
      Then I should see `Access denied`

Client issue: new feature

      As a site editor or site administrator.
      When I create a new node, I can use WYSIWYG editor for body field.
      I can switch WYSIWYG on and off.

      Behat solution:
      1. Login is site editor
      2. Go to /node/add/page
      3. Check that page contains WYSIWYG editor

Behat feature

      Given that I am a user with administrator role
      And I visit page /node/add/page
      Then I should see 'View Source' button
      And I click on 'View Source' button
      Then I should see 'View Text' button
      And I should not see 'View Source' button

Client issue: bug

      News page is empty

      Behat solution:
      1. Go to /news as anonymous user
      2. Check that there are news.

      Developer performs investigation, and finds that the problem is caused by the version of jQuery used
      by WYSIWYG. News are loaded with AJAX, so developer decides to downgrade jQuery.
      Regression! WYSIWYG stopped working!

Client issue: bug

      News page is empty again!

      There were no change in jQuery, but news stopped working.
      Another developer has modified web service that exposes news, to match the future launch of mobile apps.
      News page takes news from the same service, AJAX feed now returns rows of different structure.

Lessons learned?

      + We see that there is an issue! Awesome! Breaking change in seemingly unrelated functionality
      of the site was not rolled out.
      - It took time to find the source of the problem.
      - Relation between news page and news web service was not obvious
      We don't know the application design, and this was the source of the problem.

How things work

FeatureContext class(-es)
behat.yml

FeatureContext

class FeatureContext {
                    ...
    /**
     * Push a Submit, Delete, Run etc. button.
     *
     * @Given I push the :button button
     * @param $button
     */
    public function iPushTheButton($button) {

        $buttons = array(
            'Submit' => 'edit-submit',
            'Delete' => 'edit-delete',

Questions?

bit.ly/dckyiv-test

https://london2011.drupal.org/sites/default/files/Code%20Smells_0.pdf
https://bscdesigner.com/quality-kpis.htm
https://bscdesigner.com/complexity-metrics.htm
https://diceus.com/top-7-software-quality-metrics-matter/
https://www.atlassian.com/continuous-delivery/software-testing/types-of-software-testing
https://docs.behat.org/en/latest/