Taking a look at how we implemented CICD for TigerGraph GSQL code deployment at an enterprise scale with multiple teams.
Background
At one of our customers they had 15 team’s developing on top of TigerGraph Solutions. Having multiple teams with multiple developers means we need to look at the process in which we co-develop and deploy code for our TigerGraph Solutions. This document outlines the basics of the process we implemented to manage code between these developers.
Solution
To assist with programmatically deploying GSQL code in a multitude of environments, We developed Giraffle, a plugin to Gradle that facilitates the execution of GSQL against TigerGraph machines. This was a key aspect of reconstruction of entire solutions with a single Gradle Task. The rest of the CICD process is pretty basic and can be done through any number of tools.
Basic Flow
- Developer is working on adding a new schema change, loading jobs, queries, etc on a forked version of the official project repository. The developer would then execute their changes on their own developer box. These changes not only included the GSQL, unit tests, but also any modifications to the build process including new tasks in Gradle.
- Once the developer was ready to migrate code to the Master repository they would issue a Pull Request. When the pull request was made this would then activate a sequence of events using Jenkins as an orchestration tool.
- Code would first be run through SonarQube for a code analysis and quality check.
- After checks passed a Docker image with TigerGraph on it would be spun up for testing.
- Jenkins would then execute a sequence of Gradle commands that developers authored to build the solution using the Giraffle plugin.
6. Once a solution was deployed the solution would be verified with a suite of tests.
7. After all checks and tests passed GitHub Pull Request would then give you a green checkmark that everything was ready to be merged.
Resources:
- Using Giraffle To Manage Deployments At An Enterprise Scale
- A Step-by-Step Introduction to Giraffle How to Programmatically Create Graphs on TigerGraph
- Giraffle GitHub
- Giraffle Documentation
- Integration Testing with TigerGraph CI/CD with Docker, and GitHub Actions for a TigerGraph-backed Gradle application
Additional Notes
Other enterprise companies have used the pyTigerGraph tool for DevOps deployments. This is another option if your company is looking to use a Python specific solution. I’ve included a notebook that has a series of functions used to deploy GSQL code against environments HERE. For more information on pyTigerGraph and its functionalities please checkout pyTigerGraph Documentation and pypi. An experimental edition of pyTigerGraph is being developed called pyTigerGraph which removes the Java dependency of the GSQL client called pyTigerGraph Beta.