CICD Process for Developers

Taking a look at how we implemented CICD for TigerGraph GSQL code deployment at an enterprise scale with multiple teams.

Background

At one of our customers they had 15 team’s developing on top of TigerGraph Solutions. Having multiple teams with multiple developers means we need to look at the process in which we co-develop and deploy code for our TigerGraph Solutions. This document outlines the basics of the process we implemented to manage code between these developers.

Solution

To assist with programmatically deploying GSQL code in a multitude of environments, We developed Giraffle, a plugin to Gradle that facilitates the execution of GSQL against TigerGraph machines. This was a key aspect of reconstruction of entire solutions with a single Gradle Task. The rest of the CICD process is pretty basic and can be done through any number of tools.

Basic Flow

  1. Developer is working on adding a new schema change, loading jobs, queries, etc on a forked version of the official project repository. The developer would then execute their changes on their own developer box. These changes not only included the GSQL, unit tests, but also any modifications to the build process including new tasks in Gradle.
  2. Once the developer was ready to migrate code to the Master repository they would issue a Pull Request. When the pull request was made this would then activate a sequence of events using Jenkins as an orchestration tool.
  3. Code would first be run through SonarQube for a code analysis and quality check.
  4. After checks passed a Docker image with TigerGraph on it would be spun up for testing.
  5. Jenkins would then execute a sequence of Gradle commands that developers authored to build the solution using the Giraffle plugin.

6. Once a solution was deployed the solution would be verified with a suite of tests.
7. After all checks and tests passed GitHub Pull Request would then give you a green checkmark that everything was ready to be merged.

Resources:

Additional Notes

Other enterprise companies have used the pyTigerGraph tool for DevOps deployments. This is another option if your company is looking to use a Python specific solution. I’ve included a notebook that has a series of functions used to deploy GSQL code against environments HERE. For more information on pyTigerGraph and its functionalities please checkout pyTigerGraph Documentation and pypi. An experimental edition of pyTigerGraph is being developed called pyTigerGraph which removes the Java dependency of the GSQL client called pyTigerGraph Beta.

2 Likes

Thanks @Jon_Herke. This is exactly some what I am looking for. I could feel a disconnect where a developer updates the GSQL using graph studio, but misses in pushing the updates to code repo. This creates a problem later on when we want to triage any issue based on what version of GSQL was running and eventually reverting back to some prior version. So an automated process here would have ensured this.

I will go into the details of the use cases provided by you. Thanks.

1 Like