How long does it take your developers to create a new application with your current tooling stack?
This is one of my favorite questions to start a platform conversation. How long does it take to create the repository? How long does it take to implement a simple functionality? How long does it take to get the application running in the runtime environment? Depending on the organization’s maturity and degrees of automation and self-service the answer to these questions can range from a few minutes to several days, or even weeks (sadly this is not an exaggeration). Why so long? Because creating a new service typically involves more than just writing code. It often means means:
At an application component scope (i.e. things that need to be done for each application)
- Provisioning repositories
- Setting up initial project structure
- Configuring build pipelines (CI)
- Defining deployment processes (CD)
At a Shared platform level (i.e. things that are provided as part of the developer platform)
- Onboard in CI Infrastructure
- Onboard in CD Infrastructure
- Align with Multi-Tenant runtime infrastructure with guardrails built-in
- Provide application telemetry for the runtime
- Setup monitoring and enforce compliance and security standards
Each of these steps, when done manually, introduces friction, consumes valuable developer time, and is prone to human error. This friction directly impacts developer velocity, job satisfaction, and ultimately, the speed at which an organization can innovate.
A key capability of a developer platform is therefore to efficiently onboard new applications and help developers through a thoughtfully cultivated combination of golden paths and platform guardrails.
Golden paths are meant to reduce cognitive load for developers by providing them with a curated selection of framework stacks. They can support onboarding of new applications through templating and pre-populating values, initializing the code repository etc. Even though many of these tasks could be achieved via simple templating functionality, richer experiences like Backstage software templates are commonly used to let developers hydrate a pre-configured template with their use case specific values.
On the other hand, many components of the shared developer platform like artifact repositories, runtimes or shared CI/CD infrastructure are often provisioned centrally and configured though IAC pipelines that are owned by the platform team.
But how do I glue these two worlds together? How do I ensure the repository that is dynamically created following the golden paths of the developer platform can also trigger builds and deployments in the shared CI/CD infrastructure?
Depending on the CI tooling selection this can get even tricky. In the example we look at in this blog post we decide we want to use Cloud Build as a CI solution. To automatically launch Cloud Builds you typically use Cloud Build triggers. The difficulty is that the Cloud Build triggers are created explicitly for each application repository. Logically they are in the application component scope even though conceptually they are closely tied to the shared CI platform.
To bridge the gap between dynamically created application repositories and shared CI/CD infrastructure and runtime, several key building blocks of a modern developer platform come into play. These components should integrate as seamlessly as possible to provide the “golden paths” and implement “platform guardrails”.
Self-Service Developer Portal
Whilst a developer portal solution like Backstage is not strictly required for a functional developer platform, it is often used as a front door for developers and provides a central view of the platform capabilities like software catalogs, documentation, and of course the application templates.
Application Templates
Application templates are pre-defined “recipes” for creating new applications and services. They encapsulate best practices, standard tooling, and “golden path” configurations. The templates are usually exposing a configuration interface that exposes just the right amount of complexity to the user and hide the remaining configuration parameters behind sensible defaults.
Source Code Management
Once applications are created from application templates they usually live in a source code repository. The example mentioned in this blog post will work in any of the repository types supported in Cloud Build.
Shared CI/CD Infrastructure
The inherent complexity of this example lies in the fact that many customers will want to use a centralized CI/CD infrastructure as part of their developer platform to standardize tooling and apply consistent supply chain security practices. In our case we create CI builds with Cloud Build and roll out the application artifacts to the runtime environments using Cloud Deploy pipelines. If you use decentralized CI tooling like Github Actions or Gitlab CI to build your artifacts then the integration points of your application component specific resources and the shared platform resources will not happen at the CI trigger but at a later stage.
This first option allows you to create builds with Cloud Build without the need to create Cloud Build Trigger resources in Google Cloud at all. In this case we leverage the built-in commit triggers of the SCM platform like GitHub actions or Gitlab CI to submit a build in Cloud Build instead. As the configuration of the CI actions is usually part of a specific sub folder in the repository this trigger configuration can easily be included in the templating capabilities of the application template.
By using workload identity federation for authenticating a CI pipeline for Cloud Build a simple GitHub Action script like the one below can be used to create a snapshot of the repository and submit a Cloud Build with a simple gcloud builds submit command.
name: Create Run a Cloud Build in GCPon:
push:
branches:
- main
jobs:
run_cloud_build:
runs-on: ubuntu-latest
permissions:
contents: 'read'
id-token: 'write'
steps:
- name: Checkout Repository
uses: actions/checkout@v4
- name: Authenticate to Google Cloud
uses: google-github-actions/auth@v2
with:
project_id: ${{ values.projectId }}
workload_identity_provider: projects/${{ values.projectNumber }}/locations/global/workloadIdentityPools/identity-pool/providers/github-actions
- name: Set up Google Cloud SDK
uses: google-github-actions/setup-gcloud@v2
- name: Run Cloud Build
run: |
gcloud builds submit . \
--config="cloudbuild.yaml" \
--service-account=projects/${{ values.projectId }}/serviceAccounts/cloud-build@${{ values.projectId }}.iam.gserviceaccount.com
On the upside this is a very simple approach and can be achieved by centrally setting up workload identity federation and IAM permissions as part of the platform and thus not requiring any further configuration on the Cloud Build side for different applications.
On the other hand, this approach creates a lot of overhead and adds additional latency compared to directly triggering cloud build triggers. It first needs to instantiate the bootstrapping pipeline in Github Actions before it submits the Cloud Build trigger. From a usability perspective we lose some of the handy integrations in Cloud Build that allow us to automatically link a Git Repo content hash to a Cloud Build. It’s also important to call out that with workload identity federation we needed to delegate permissions to an external identity provider which might not be permitted in all scenarios.
To avoid these issues we should look at triggering builds using Cloud Build triggers instead.
So far any operation we performed including the creation of the Github Action in the bootstrapped repository was using the built-in application template functionality of Backstage. The hydrated repository contained the configuration of the Cloud Build and the Github Action that triggered it.
If we want to use the Cloud Build native triggers instead of the GitHub action we need to extend our template to not only create and push a Code Repository but also create the necessary infrastructure components in Google Cloud.
Specifically in our case we would need to create a Cloud Build Repository that points to the newly created Git repository and a Cloud Build Trigger that triggers a build upon pushes to the main branch of that repository.
If your development platform allows developers to provision cloud resources using IAC in a self-service manner, adding these two resources can be as simple as adding the necessary Terraform resources for a Cloud Build Repository and Trigger and triggering a Terraform apply during the template initiation process. Terraform is just used as an example here and any other IAC process that allows you to create the necessary resources would work equally well.
This declarative infrastructure solution is elegant and often preferred as it is not dependent on the Developer Portal implementation and follows industry best practices for IAC. On the other hand if all you want to achieve is to allow developers to trigger Cloud Builds then operating a full IAC infrastructure that integrates with the application templates might be considered a bit of an overkill and introduce too much operational overhead for the platform team.
If you do not have any plans to support a full IAC pipeline and simply want to create Cloud Build triggers as part of the application template you could also do this by implementing a custom action in Backstage that can later be used in your application template.
Because the Cloud Build trigger depends on a Cloud Build repository that is also only known after the application template has created the repository in the SCM we would create two custom actions that allow you to:
- Create a Cloud Build repository (technically a 2nd gen repository that allows you to use an existing connection resource)
- Create a Cloud Build trigger for that repository
A simple proof of concept for such an action can be found in the following repository:
With these custom actions you can extend your pipeline with two steps that can be added after the repository creation to wire :
- id: create-repository
name: Create Cloud Build Repository
action: gcp:cloudbuild-repository:create
input:
projectId: ${{ parameters.projectId }}
gcpRegion: ${{ parameters.gcpRegion }}
repositoryName: ${{ parameters.repositoryName }}
remoteUri: ${{ parameters.remoteUri }}
connection: ${{ parameters.connection }}
- id: create-trigger
name: Create Cloud Build Trigger
action: gcp:cloudbuild-github-trigger:create
input:
projectId: ${{ parameters.projectId }}
gcpRegion: ${{ parameters.gcpRegion }}
triggerName: ${{ parameters.triggerName }}
repositoryName: ${{ parameters.repositoryName }}
connection: ${{ parameters.connection }}
branchPattern: ${{ parameters.branchPattern }}
buildConfigPath: ${{ parameters.buildConfigPath }}
Embedded into the out-of-the-box templating capabilities of the Backstage Application templates this would allow you to build pipelines like the one shown below where the generated repository is hooked up to the Cloud Build pipeline:
In an enterprise environment, the creation of new applications is often slowed by manual provisioning and unnecessary formalities. As we’ve discussed, “golden paths” implemented with Backstage software templates can streamline these architectural decisions and automate the end-to-end process. This post demonstrated how to integrate these templates with centrally managed CI/CD environments in Google Cloud, bridging the critical gap between application scaffolding and the software development lifecycle. By connecting these two worlds, you can transform a slow, error-prone procedure into a rapid, standardized, and self-service experience for your developers.
If you are ready to connect your golden paths to a CI/CD pipeline in Google Cloud you could follow the following steps:
- Review the Options: Evaluate the integration strategies discussed in this post and select the one that best fits your organization’s technical maturity and requirements.
- Implement a Pilot: Start by building one complete end-to-end golden path for a common service type. Focus on making the journey from the Backstage template to a triggered Cloud Build pipeline completely seamless.
- Gather Feedback and Iterate: Onboard the first set of application teams to use the new template. Use their feedback to refine the process and improve the developer experience before rolling it out more broadly.
- Expand the pattern: Expand the automation to other parts of the shared platform like continuous deployments, load balancer configurations or even creating documentation assets to further reduce friction and speed up the creation process for your applications.
Source Credit: https://medium.com/google-cloud/streamline-app-creation-with-backstage-templates-and-automated-ci-cd-in-google-cloud-72923f4ab1dc?source=rss—-e52cf94d98af—4
