jeudi 16 août 2018

How to set up working automated browser tests on Travis CI?

I am trying to set up automated tests for my web app on Travis CI. I am talking about automated browser tests - running unit tests is simple, but I need a process/system test that will start a headless browser and perform scripted tests of various use cases.

My application is in PHP so I decided to go with PHPUnit, Selenium and headless Firefox.

After a lot of research and trial and error I ended with following .travis.yml file:

language: php
php:
  - '7.1'
services:
  - mysql
addons:
  firefox: latest
env:
  - MOZ_HEADLESS=1
    DISPLAY=:99.0
    SELENIUM_FIREFOX_DRIVER=/home/travis/build/lotcz/zSample/geckodriver
before_install:
  - sudo apt-get update > /dev/null
  - wget https://selenium/download/url -O selenium-server.jar
  - wget https://github.com/mozilla/geckodriver/releases/download/v0.21.0/geckodriver-v0.21.0-linux32.tar.gz
  - tar -xzf geckodriver-v0.21.0-linux32.tar.gz
install:
  - sudo apt-get install apache2
  - sudo service apache2 start
  - mysql -e 'CREATE DATABASE IF NOT EXISTS zsample;'      
before_script:      
  - nohup java -jar -Dwebdriver.gecko.driver=$SELENIUM_FIREFOX_DRIVER selenium-server.jar &
  - composer install
script:
  - phpunit --fail-on-risky --fail-on-warning --stop-on-skipped --stop-on-incomplete --verbose --debug --colors
after_failure:     
  - cat nohup.out

I edited out some pieces specific to my application. Just believe me that I set my application correctly before running the test.

Now a very simple test may look something like this:

class VisitorLoginTest extends PHPUnit_Extensions_Selenium2TestCase {

  public function setUp() {
    $this->setHost('localhost');
    $this->setPort(4444);
    $this->setBrowserUrl('http://localhost');
    $this->setBrowser('firefox');
  }

  public function tearDown() {
    $this->stop();
  }

  public function testFrontPage() {
    $this->url('/');
    $content = $this->byClass('main-title')->text();
    $this->assertEquals('Hello', $content);
  }

}

When my test is run I get this:

The Selenium Server is not active on host localhost at port 4444.   
OK, but incomplete, skipped, or risky tests!
Tests: 1, Assertions: 0, Skipped: 1.
The command "make test" exited with 0.

Now my problems are these:

  • this file seems to be really long to me and too complicated given that I want to do something rather standard (I assume that automated browser tests are common these days). I would expect Travis CI to provide an easier way to perform automated browser tests. Here I have to download, install and start Apache, Selenium driver, Selenium server, use composer to get PHPUnit Selenium plugin etc...
  • Selenium server doesn't seem to be running and I can't find out why.
  • In the end, because PHPUnit ingeniously return 0 even when it couldn't even run the tests, Travis reports this test as successful. All those flags like --fail-on-risky or --stop-on-skipped still don't force PHPUnit to report a test failure which is in my opinion the only logical result as test clearly failed.

I know this is too broad and contains multiple questions. I am afraid that I took wrong direction somewhere and I am probably trying to do something simple in a complicated way.

Can somebody provide working example of .travis.yml file for automated browser tests? My application is in PHP, but I can write tests in Node.js, Python, Java or anything else as long as tests will really work and failure will be reported if anything goes wrong.

Aucun commentaire:

Enregistrer un commentaire