CI made dead simple with Drupal 8, GitLab and Platform.sh

Hi

Branislav Bujisic, engineering director

Family guy

Bad accent included

Why Drupal?

• It's PHP
• It's a mishmash of custom and contrib code
• It's complicated (database, caching, search engines...)

Why Gitlab

• CE - open source (MIT License)
• SaaS or self-hosted
• Merge requests
• Built-in CI/CD

Why Platform.sh

• PaaS
• Use git to manage the hosting
• An environment per git branch
• Branching = cloning the parent environment state (db, services, everything)

PHP Codesniffer

Setup Drupal coding standards:

    composer global require drupal/coder
    export PATH=”$PATH:$HOME/.composer/vendor/bin”
                        

Usage:

phpcs --standard=Drupal modules/custom

Results:

-----------------------------------------------------------------------------------------------------------------------
FOUND 13 ERRORS AND 3 WARNINGS AFFECTING 16 LINES
-----------------------------------------------------------------------------------------------------------------------
   5 | WARNING | [x] Unused use statement
   7 | WARNING | [x] Unused use statement
   9 | ERROR   | [x] Missing class doc comment
  13 | ERROR   | [x] Missing function doc comment
  33 | ERROR   | [x] You must use "/**" style comments for a function comment
  35 | ERROR   | [x] Data types in @var tags need to be fully namespaced
  44 | WARNING | [ ] Line exceeds 80 characters; contains 91 characters
  45 | ERROR   | [ ] Doc comment short description must be on a single line, further text should be a separate paragraph
  78 | ERROR   | [ ] Doc comment short description must be on a single line, further text should be a separate paragraph
  92 | ERROR   | [ ] Missing short description in doc comment
  93 | ERROR   | [x] Data types in @param tags need to be fully namespaced
  98 | ERROR   | [x] You must use "/**" style comments for a function comment
 118 | ERROR   | [ ] Missing short description in doc comment
 125 | ERROR   | [x] You must use "/**" style comments for a function comment
 139 | ERROR   | [x] Expected 1 blank line after function; 0 found
 140 | ERROR   | [x] The closing brace for the class must have an empty line before it
 -----------------------------------------------------------------------------------------------------------------------
 PHPCBF CAN FIX THE 11 MARKED SNIFF VIOLATIONS AUTOMATICALLY
 -----------------------------------------------------------------------------------------------------------------------

Time: 2.38 secs; Memory: 10MB

When are you doing it?

1. Before each commit?
2. Before each push?
3. Before each merge request?
4. When you remember to do it?
5. Never?

Manual work is tough...

If you automated your testing, you are doing it right.

...because, honestly, do you even want to waste your code review time on trivialities such as coding standards.

The goal:

ensure quailty of software,
through frequent and comprehensive testing,
while maintaining high productivity

Quality control

• Coding standards
  because we need consistency
• Unit tests
  cover individual componets, smallest possible things to test
• Integration tests
  cover the interaction between components
• Acceptance tests
  evaluate the compliance with the business requirements
• E2E tests
  cover the entire flow of an app, from the start to the end

What can be automated?

Good quality control

Continuous Integration allows you to integrate the code into a shared repository and build and test each change automatically, as early as possible, usually several times a day.

Source: https://about.gitlab.com/product/continuous-integration

Even better quality control

Continuous Delivery ensures that the software can be released to production at any time, often by automatically pushing changes to a staging system.

Source: https://about.gitlab.com/product/continuous-integration

Quality control that you really trust

Continuous Deployment takes the process a step further and pushes changes to the production automatically.

Source: https://about.gitlab.com/product/continuous-integration

The pipeline

Build > Test > Deliver > Deploy

Source: https://about.gitlab.com/product/continuous-integration

Code review tools are logical candidates to run CI/CD pipelines

While humans read the diff, machines run the automation.

Lots of tools

Setting up Gitlab

Ubuntu 18.04, 2 cores, 8GB RAM

curl -LO https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh
sudo bash script.deb.sh
sudo apt install gitlab-ce

# /etc/gitlab/gitlab.rb
external_url 'https://example.com'
letsencrypt['contact_emails'] = ['admin@example.com']

sudo gitlab-ctl reconfigure

Source: https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-gitlab-on-ubuntu-18-04

Gitlab runners

• An isolated machine (a VM, a VPS, a bare-metal machine, a docker container, or a cluster of containers).
• Picks up jobs from the coordinator API of GitLab and runs them.
• Ideally, it should not run on the same server as GitLab.

Source: https://docs.gitlab.com/runner/

Setting up a GitLab runner

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
sudo apt-get install gitlab-runner

sudo gitlab-runner register

Sources:
https://docs.gitlab.com/runner/install/linux-repository.html
https://docs.gitlab.com/runner/register/index.html

Docker image

docker pull bbujisic/drupal8-phpunit:1.0

• PHP 7.2, composer, Drush
• PHPCS, Drupal Coder, PHPLint
• PHP Unit
• Open source

The recipe

image: bbujisic/drupal8-phpunit:1.0

stages:
  - test

phpcs:
  stage: test
  script:
    - phpcs --standard=Drupal web/modules/custom web/themes/custom

phpunit:
  stage: test
  script:
    - composer install
    - phpunit web/modules/custom

#.gitlab-ci.yml
image: bbujisic/drupal8-phpunit:1.0
stages:
  - test
   
phpcs:
  stage: test
  script:
    - phpcs --standard=Drupal web/modules/custom web/themes/custom
phpunit:
  stage: test
  script:
    - composer install
    - phpunit web/modules/custom

PHPUnit can solve the unit and integration testing

The problem of acceptance and end-to-end testing

Solvable by behat, for example

Behat

@api
  Scenario: An anonymous should see the hello page
    Given I am an anonymous user
    When I go to "hello"
    Then I should see "Hello world"

Acceptance and end-to-end tests are expensive

• As close to production environment as possible.
• Database, files, search server, caching etc.

The setup is complicated

• Behat in the Docker container
• Database in the Docker container
• Dump the from the staging environment
  ... and import it
• Rsync files from the staging environment
• What about the search server, redis, mongodb?

... skip it

Deploy to staging

Platform.sh solves the problem of staging environments and stakeholder acceptance

A new staging environment

$ git push platform my-feature-branch
$ platform environment:activate
                    

$ platform push
                    

Problem: platform-cli requires users to authenticate, and git requires an ssh key pair.

Gitlab + Platform.sh

1. Use the environment variables to store the ssh key pair and API token.
2. Setup ssh on docker container build.
3. Push to platform.

Gitlab environment variables

SSH setup and push

stages:
  # ...
  - deploy
# ...
psh-deploy:
  stage: deploy
  script:
    - mkdir -p $HOME/.ssh
    - echo "$SSH_KEY" > $HOME/.ssh/id_rsa
    - echo "$SSH_KEY_PUB" > $HOME/.ssh/id_rsa.pub
    - echo "$SSH_KNOWN_HOSTS" > $HOME/.ssh/known_hosts
    - chmod go-r $HOME/.ssh/id_rsa
    - platform project:set-remote "$PSH_PROJECT_ID"
    - platform push --force --activate --target=$CI_BUILD_REF_NAME

Continuous delivery!

The problem of Behat tests remains

Can I make the available tools work for me at a cost of paradigm shift?

Do I really have to run all the tests before the delivery to staging?

I could deliver to Platform.sh staging first, and then run Behat tests.

Setting up behat

// composer.json
"require": {
  "drupal/drupal-extension": "^3.4",
},
"config": {
    "bin-dir": "bin/"
}

composer update
mkdir behat
cd behat
../bin/behat --init

Problematic part of setting up behat

# behat/behat.yaml
default:
  suites:
    default:
      contexts:
        - FeatureContext
        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MinkContext
        - Drupal\DrupalExtension\Context\MessageContext
        - Drupal\DrupalExtension\Context\DrushContext
  extensions:
    Behat\MinkExtension:
      goutte: ~
      selenium2: ~
      base_url: http://mysite.local
    Drupal\DrupalExtension:
      blackbox: ~
      api_driver: 'drupal'
      drush:
        alias: 'local'
      drupal:
        drupal_root: '../web/'
      region_map:
        footer: "#footer"
                    

No way to know default.extensions.Behat\MinkExtension.base_url,
use the environment variable instead!

Add a step to the pipeline

• Ensure the availability of the ssh key to the platform-cli tool
• Get the URL of the platform environment

  platform url --pipe | head -n 1

• create environment variable BEHAT_PARAMS containing the above URL:

  {"extensions": {
      "Behat\\MinkExtension":
        {"base_url":"[platform-url]"}
      }
    }

Complete CI stage

stages:
  # ...
  - test-staging
# ...
behat:
  stage: test-staging
  script: |
    mkdir -p $HOME/.ssh
    echo "$SSH_KEY" > $HOME/.ssh/id_rsa
    echo "$SSH_KEY_PUB" > $HOME/.ssh/id_rsa.pub
    echo "$SSH_KNOWN_HOSTS" > $HOME/.ssh/known_hosts
    chmod go-r $HOME/.ssh/id_rsa
    platform project:set-remote "$PSH_PROJECT_ID"
    platform variable:create \
        --name=env:BEHAT_PARAMS \
        --value="{\"extensions\":{\"Behat\\\\MinkExtension\":{\"base_url\":\"`platform url --environment=$CI_BUILD_REF_NAME --pipe | head -n 1`\"}}}" \
        --level=environment --json=true \
        --environment=$CI_BUILD_REF_NAME \
        --yes || true
    platform ssh "cd behat; behat" -e $CI_BUILD_REF_NAME

The revised pipeline

What next?

Run all tests after the deployment?

Continuous deployment?

Automated release notes? Slack integration?

Whatever you want, really!