Pass results directly from a Cloud Build pipeline without storing/retrieving an artifact
Background
The typical pattern for using a CI/CD pipeline is to provide some input (like a git repo), and the pipeline will perform a combination of commands such as API calls or software compilations.
By the end, the pipeline may produce some ‘artifact’. This could be the compiled software binary, X-as-Code State, or a log file.
Google Cloud Build is a managed ‘serverless’ CI/CD pipeline service allowing you to run pipelines without the need to maintain any infrastructure and only pay for the compute over the time the pipeline is running.
Solution
A seemingly obscure feature of Cloud Build, (I don’t even know if it has a name, let’s call it:) buildStepOutputs.
When you get the details of a Cloud Build run, you can see many details about it, including the status (running, failed, successful, etc.), duration, parameters used, and more.
In particular, there is a value for ‘results’, under which is ‘buildStepOutputs’.
Here is what Google’s documentation has to say about this:
buildStepOutputs[]
string (bytes format)
List of build step outputs, produced by builder images, in the order corresponding to build step indices.
Cloud Builders can produce this output by writing to $BUILDER_OUTPUT/output. Only the first 4KB of data is stored.
A base64-encoded string.
This is the entirety of the documentation on this feature and I can find no references to people using it.
In fact, it’s so underappreciated that $BUILDER_OUTPUT is an environment variable that’s not even listed in the documentation of available environment variables.
Even so, it’s an incredibly useful tool for passing data that persists with the build history, and with no need for additional permissions (if you can trigger the build, you likely have permissions to see its status) or medium to store an artifact.
Use Case
I was writing some automation that would trigger a pipeline and I wanted it to get the results of the pipeline.
The typical pattern would be for the pipeline to output an ‘artifact’, in this case, it could have been a small text file saved to Google Cloud Storage. However, this would have required granting additional permissions to both the pipeline and automation to access GCS.
Another option would be to have the pipeline write to the logs and then have the automation read the whole log looking for certain messages. This can get messy.
But using the buildStepOutputs I was able to write the results of the pipeline to the Cloud Build service and have the automation pick up those results in the same API call checking if the pipeline ran successfully or not.
How To Use
Using this feature is pretty simple: for any step during the build that you want to pass data from, simply write to the file “/builder/outputs/output” during that step, as you would any other file. ($BUILDER_OUTPUT points to “/builder/outputs”)
Then when the pipeline is complete, you can access the output from the Cloud Build API.
Here is an example using the google-cloud-sdk CLI tool:
| |
Note: $STEP_INDEX is zero-indexed.
