I’m really excited about today’s show because this is one of those topics that I know that I need to know, but don’t know where to start. So, I’m super excited that you agreed to come on and teach me what you know about regression testing!

Regression Testing

• What do you mean when you say “Regression Testing”?
• Why is regression testing important?
• Is there anything to install on your computer? (Testing server)
• What types of regression testing are there?
  • Unit/Functional (Mocha/Chai)
  • Behavioral (WebdriverIO, Behat)
    • Authenticated/Anonymous
  • Qualitative (Sasslint, JS lint, JSON lint)
  • Performance (Pagespeed, Perfbudget)
  • Visual Regression (wraith, phantomcss, webdrivercss)
    • How does this work on pages that have dynamic content. Homepage photos, comments etc.
• Where would you start?
  • Front-end
  • Back-end
  • Site-builder

Where to learn

• https://visualregressiontesting.com/
• https://learn.visualregressiontesting.com/
• Free 6 day email course

Automated Testing

• Before we actually get started talking about Red Test, can we take a step back and talk about what automated testing is in general?
  • I am sure you are all aware of manual testing and QA. Once you build a site or a feature in the site, you go to the browser and test it out. As an example, you added a workflow to your blog post. You will go to the browser and test whether the blog you create gets the draft state. Then you will log in as an editor and you will move it to published state. Now you will log out and check whether the blog post is visible to anonymous users. What you are doing is manual testing. It works very well for smaller projects.
  • For bigger projects, and especially for projects with multiple developers, what starts happening is adding a feature may break an older feature. Let’s take the example above. Earlier all the blog posts used to be published by the writer directly as soon as he saved the form and an email used to go out to all your blog subscribers that a new blog has been created. Now after adding the functionality to save the blog post as a draft state, you want the email to go out only after the blog post has been published. Suppose the developer changed and tested the workflow functionality without changing the email sending rule and pushed the feature to live site. You will soon be flooded by complaints from your blog subscribers.
  • This is a very simple example but such problems increase exponentially when the software grows in size over time. Such problems can be prevented by having automation testing in place. For each feature in your software, you write test code and you run all these tests often, and especially after working on a feature. You push your new feature to production only after all the tests pass. This way, as long as you have ample test coverage, you minimize your chances of finding a bug on the production site.
• Who generally does this testing, the developer? site-builder? site-owner?
  • There are different schools of thought here. In practice, I have seen two different approaches:
    • If you are following test-driven development, then the developer writes these automated tests. In fact, he writes the tests before he even writes the code for the feature and makes sure that the tests fail. Then he implements the feature and makes sure that all the tests pass. In our company, our developers write the tests but that’s usually done after he has implemented the feature. We are experimenting with test driven development as of now.
    • The other approach I have seen in having a separate QA team for testing. They write automated tests as well as do manual QA. The advantage of this approach is that you can hire QA person, who is usually much cheaper than a developer. The problem I have seen in this approach is miscommunication between the developer and tester. Because of the delay due to communication and also since tester writes the tests after developer has completed the task, as compared to in parallel, it takes more time for a feature to get out of the door.
• What do these automated tests test for?
  • Ideally everything but practically that’s not possible. As everything in life, this follows pareto rule. 20% of tests will cover 80% of the functionality and you should definitely write these 20% tests. In addition to that, we follow a rule that no bug should appear on production twice. So if a bug is found on production, we write an automated test for it so that it doesn’t appear on production again. What these tests should cover is very dependent on your application. If you are building an information blog site with a beautiful theme and you are making changes mostly to the theme, then you should write regressions tests for your CSS and JS. On the other hand, if your site has a lot of custom business logic, access permissions and rules, then focus on testing the logic.
• I always assumed tests were for functionality, can you give me an example of something you would test on the theme layer?
• I have to admit that I haven’t ever written an automated test for any of the sites I’ve built. I did have an experience a little over a year ago where the videos on my site (that were supposed to be behind a pay-wall) weren’t because I made a change to panels that messed up the access control. I didn’t realize it for two months! So, if I had had tests in place to check the access to those videos, I would have been in better shape. So, even though my site is a small site in the grand scheme of things, even I can bennefit from writting appropriate tests.

RedTest

• Red Test, as I understand it, is an open source integration testing framework for Drupal that you developed at Red Crackle. Can you tell us briefly what it does and why we should use it?
  • Red Test is an open-source integration testing framework for testing your Drupal applications. If you have multiple developers working on a big Drupal application stretching into months, you know that you need automated testing to keep adding more and more functionality without breaking old code. Red Test is ideal for testing:
    • Site structure and configuration
    • Custom business logic of your application
    • User access, roles and permissions
    • Rules
• Drupal 7 supports Simpletest by default. Simpletest has unit testing and functional testing. Why do we need another automated testing solution? How is Red Test different from Simpletest and why should people use it?
  • You correctly mentioned that Drupal 7 supports Simpletest by default. In real life, when you are working on a big project, there are quite a few challenges when you test Drupal sites.
    • Drupal assumes that there is a persistent database connection available so any hook or function is free to access the database at will. This means that any function that accesses the database, unit testing is not possible. You can obviously refactor your code but that takes a long time. Also since Drupal stores all the configuration in the database, most of your custom code will need to access the database anyway.
    • For every test, simpletest creates a new test environment that has only minimal modules enabled. So if you want to test your current site with all the configuration, you have to first set all that configuration in code and then run the tests. That is too much of an overhead.
    • Functional testing in Simpletest is very slow because it uses a headless browser and every request needs to bootstrap Drupal. It’s not uncommon for a test suite on a large site to take multiple hours to finish.
  • Red Test alleviates these problems. It is an integration testing framework so it tests your site with the current configuration. It actually depends on the database so there is no need to refactor your code to make it work with Red Test. In fact, Red Test code is totally separate from Drupal code. We have created Red Test so that it runs in parallel on multiple processors of your machine and bootstraps Drupal only once at the start so it is 60 times faster than Simpletest.
• A lot of Drupal developers have started using Behat which helps in testing your site functionally through a browser. With Behat gaining traction, is there still a need for Red Test?
  • Behat is an excellent tool and in fact, we use it in addition to Red Test. Since Red Test is an integration testing framework written in PHP and resides on the server, it can’t really test the javascript. So wherever JS functionality needs to be tested, Behat is the right tool. On the other hand, for testing all the other business logic in the application, we use Red Test. If you have used Behat on a big project, you will know that creating and especially maintaining Behat tests takes a long time. Every time an HTML id changes, Behat test needs to be changed.
  • Similarly, when you add a new required field in a form, Behat test needs to be updated to fill that field in the form. Red Test, on the other hand, knows about Drupal and its entities, nodes, people, roles and permissions. So it does a lot of tasks automatically in the background. As an example, if you added a new required field to a node form, Red Test will automatically fill suitable values in that field without you having to change anything in your tests. This makes it very easy for you to create and develop tests on Red Test.
  • In fact, we have done some measurement over the last couple of months and found that with the latest version of Red Test, creating and developing automated tests takes about 12% of total development time.
• Is it easy to get started with writing automated tests using Red Test?
  • Yes, we are using all the standard PHP tools so it’s pretty easy for a developer to get started. Red Test uses Composer for installation and is based on PHPUnit. In fact, we measured how much time it takes a newbie to create the first integration test using Red Test and it comes to be a little less than 15 minutes. Below is the blog post you can follow to get started: http://redcrackle.com/red-test/documentation/first-integration-test

Use Cases

• What are some concrete examples of what you should test?
  • Red Test is ideal for testing:
    • Site structure and configuration
    • Custom business logic of your application
    • User access, roles and permissions
    • Rules
  • If you are building just a basic informational site on Drupal without much functionality, you may not want to use Red Test. Use it only if you are building a large Drupal application.
• What’s in the future for Red Test? Do you have improvement plans, or plans to integrate with D8?
  • Currently a developer needs to create tests using Red Test. We are considering whether enhancing it so that it works using Gherkin language makes sense. The next obvious step is to migrate it to Drupal 8.