Engineering

Continuous Integration Setup for Validated Products: a universal Jenkins Workflow

Continous Integration (CI) is a best practice of professional software developers that automates building, testing and deployment. CI doesn’t just save time. By automatically performing these steps each time a developer commits code, the team gets early feedback, and early feedback helps avoid unnecessary context switching – adjustments can be made before the developer moves on to the next thing. At Little Green Software, we use continuous integration on every project.

Additionally, many of Little Green Software’s customers are in regulated industries that require formal validation of the software. Formal validation requires traceability through the software design, development, testing and publishing process, so we’ve designed our CI workflow to facilitate traceability. In this article we explain our continuous integration model, the strategy behind it, and how it helps maintain traceability.

Goals

We defined the following goals for the Little Green Software continous integration process:

  1. Commits to the source control repository automatically trigger a build so that developers get immediate feedback on the code they’ve just written.
  2. Each build has a unique version number used to identify the build and facilitate traceability.
  3. Each successful build is deployed to an appropriate environment, either automatically or manually-upon-review, depending on the environment.

Development Workflow

Little Green Software developers follow a gitflow-based workflow for development:

  1. Developer writes code and commits it to a personal feature branch.
  2. Developer submits pull request to merge the feature branch to the develop branch.
  3. Reviewer reviews the code. If changes are needed, the process repeats from step 1. When the reviewer approves the pull request, the code is automatically merged to develop.

The development workflow feeds into the continuous integration workflow.

Jenkins CI

Little Green Software uses Jenkins CI and its Workflow plugin to implement our CI workflow. We like this combination because:

  • Jenkins supports every platform
  • We could create a “universal” workflow, meaning one that is shared by all applications on all platforms, web and mobile.
  • The entire workflow is stateful, meaning that it can survive a restart and each step can be handled by any available node.

Jenkins Workflow for Web Applications

Web app builds can be deployed to any valid web hosting environment. Web platforms offer various mechanisms for specifying environment-specific configuration parameters – environment variables, config files, launch parameters, etc.

Jenkins Workflow: Internal Builds
Workflow Step
1. Pull latest code from the develop branch of the source control repository.
2. Set the version number using semantic versioning scheme (Major.minor.patch.build) and increment the build number.
3. Build and stash the build artifact.
Dev Phase Workflow Step Test Phase Workflow Step
4. Deploy to Dev 7. Deploy to Test
5. Review by Developer 8. Review by QA
6. Promote to Test Phase -
Jenkins Workflow: Client Builds

Preparing a build for the client is very similar. When the release manager is ready to prepare a client release, a developer merges develop to master, which triggers the Master workflow. The Master workflow is identical to the Develop workflow except that the build is deployed first automatically to Staging and then manually upon release manager approval to Production.

Workflow Step
1. Pull latest code from the master branch of the source control repository.
2. Set the version number. Note: client builds are based on the same numbering scheme as the internal builds so that version numbers are unique per project build regardless of the branch from which they were built.
3. Build and stash the build artifact.
Staging Phase Workflow Step Production Phase Workflow Step
4. Deploy to Staging 7. Deploy to Production
5. Review by QA, release manager, then client 8. Review by all
6. Promote to Production Phase -

Jenkins Workflow - Web App Workflow

Jenkins Workflow for Mobile Apps

The Mobile app workflow is identical to the Web app workflow except that mobile apps need to be built specifically for the deployment environment. Mobile platforms do not offer any mechanism for configuring the environment; instead, build “flavors” are used to include environment-specific configuration in the build itself.

Jenkins Workflow - Mobile App Workflow

In the mobile app workflow, the Dev flavor can only be built and deployed to Dev, the Test flavor is built can only be deployed to Test, the Staging flavor can only be built and deployed to Staging, and the Production flavor can only be deployed to Production.

Traceability

Validated apps, such as medical apps requiring FDA approval or apps for clinical research, require traceability: an auditable and unbroken chain linking every step in the development proceses:

  • Requirements
  • Design
  • Committed code
  • Code review
  • Versioned build
  • Tested build
  • Reviewed approved and deployed build

Our Jenkins workflow ensures every build is versioned and available to the tester for review. Because we only publish builds via our CI system (ie, instead of from a developer’s local machine) it ensures that the build that was tested, reviewed and approved is the exact build that will get deployed and published. Not only does that meet the requirements of traceability, it avoids confusion and lowers risk by having a managed version numbering system and a consistent build machine environment.

We enforce that all commit messages reference a JIRA issue. JIRA is integrated very nicely with Bitbucket, our git repository, so that each commit and pull request is linked to the JIRA issue.

JIRA Bitbucket Integration

Those links, along with JIRA’s auditable history log, are exactly what we need to trace the full chain of history from requirements to deployed build.

Slack

No developer tool is complete without Slack integration. Unfortunately as of this writing Slack is not currently supported by Jenkins Workflow which means we aren’t notified when the workflow succeeds or fails via Jenkins. Fortunately, the deployment targets we use most frequently, iTunes Connect and Crashlytics Beta, have their own notification mechanism, so the team is notified when a new build is ready for review.

The advantages of a universal workflow

The Jenkins Workflow plugin is a major paradigm shift for the Jenkins CI tool. Cloudbees, the primary sponsor of Jenkins CI, is promoting Workflow as “a major step in the evolution of Jenkins which takes it from CI to full dev-to-production continuous delivery.” While it’s great that Jenkins is building support for delivery / deployment into its core product, the ability to share the workflow across all platforms and client products is the primary reason we chose to migrate our company to the tool. Previously each application required its own Jenkins job. If a platform update occured that affected all applications, each job required updating. With Jenkins Workflow we can easily make changes to our workflow across all projects, meaning we can efficiently maintain our customer’s applications. The Workflow plugin is still relatively new – it will take some time before it reaches maturity and full support with other Jenkins plugins – but we are already experiencing its benefits.

What’s next?

Of course we have ideas on how to make our CI workflow even better. The Jenkins JIRA plugin is not yet compatibile with Jenkins workflow. It would be great if builds could automatically link to JIRA issues and vice versa – currently this is still a manual step.

Are you developing software that requires validation? How is your continous integration and delivery workflow set up to facilitate traceability? If you have tried Jenkins Worfklow, how did you like it?