Once the solution is designed and the code is written, it’s “just” about deploying, and then we’re done, right? We often use CI/CD solutions to build and deploy, run tests, and much more. If someone can compromise pipelines, the build agent building the solution, or the connection to the resources we deploy to, we will have major problems.
Even though it’s challenging to cover everything in a few short articles, we still try to provide insights into the issues that delivery teams should address.
1 - CI/CD
CI/CD is the overarching framework for automating builds, verification, and deployment. A good pipeline provides traceability, quality assurance, and controlled rollout across environments.
Continuous Integration and Continuous Delivery (CI/CD) are practices for automating the flow from code change to running solution. Pipelines provide a standardized process for building, verification, and rollout, and reduce the risk of manual errors.
The point of CI/CD is not just speed but control. Each run should be traceable, repeatable, and have clear stopping points before anything goes to production.
The CI/CD flow from code change to production. The same artefact travels the whole way, and the stopping point before production is deliberate: every run should be traceable, repeatable and approved before it reaches end users.
Principles for CI/CD
Safe CI/CD practice is built on a few simple foundational principles:
Least privilege: The pipeline should only have the access it actually needs.
Traceability: You should be able to see what was run, by whom, and against what code.
Automated rules: Quality and security requirements should be in the pipeline, not in people’s heads.
Controlled flow: Production should have its own control points with tests and approvals.
Environment separation: Build, test, and production should not share identity and access.
Security controls in the pipeline
A pipeline should have at minimum:
Secret scanning: Find hardcoded keys, tokens, and other sensitive information, see Secrets.
Static code analysis (SAST): Find common weaknesses early, see Security Testing.
Automated tests: Run unit, integration, and relevant security tests, including checks of authorization, 401/403 responses and JWT handling.
Audit logging: Keep logs and metadata for audit trails.
Critical findings should stop deployment, not just give a warning.
Several of these controls require additional tools and licenses. Be particularly mindful of tools that send code to external services for analysis — that is a flow of data out of the project, and it has to be cleared.
Branching and promotion model
Branching is closely connected to CI/CD, but the details belong in the Git article.
In CI/CD context, the main points are:
Code to production should go via pull request and protected main branch
Deployment to production should require passing controls and approval
Secrets and credentials in the pipeline
Secrets used in pipelines must be handled with particular care:
Never in code: Keys and tokens should not be in the repository.
Use secure storage: Use dedicated secret storage in your platform.
Rotate regularly: Change keys and tokens at fixed intervals.
Avoid leakage in logs: Ensure secrets don’t end up in logs.
Third-party components in the pipeline
Pipeline configuration is part of the supply chain. Third-party components in the pipeline (actions, orbs, templates, build images) run with pipeline access and must be treated as code you’re responsible for.
This means a compromised or malicious third-party component can:
Exfiltrate secrets and credentials available to the run
Manipulate artifacts being built without it being visible in your code
Change behavior when a tag is updated to point to new code without your approval
Measures:
Pin to commit hash, not tag: A tag can be moved without you noticing.
Limit access for actions: Give a component only the permissions it actually needs.
Review third-party actions: Review the source code of actions you use, especially those handling secrets or deploying to production.
Use internal or official actions: Prefer actions from the same organization or from verified actors with good track records.
Monitor changes: Use Dependabot or similar to notify you when actions are updated, so updates are explicit and tracked.
Using AI in pipelines: Be sure you understand the risks and potential challenges before considering using AI-based actions in your pipeline.
For AI solutions, the same pipeline principles should apply to code, models, datasets, and configuration. Deployment should stop if verification fails.
Details about safe building of artifacts (including signing) are described in Building, and safe deployment to environments in Deployment.
Practical adoption in teams
CI/CD implementation should be done gradually:
Start with basic building and testing, add security controls gradually.
Document pipeline configuration as code so it can be reviewed and versioned.
Give the team time to learn the tool and practice before enforcing stricter requirements.
Follow up on findings in practice. Alerts without follow-up have little value.
Building is about producing a reproducible artifact that can be verified and trusted before deployment. Build environment, dependencies, and signing are critical control points.
Building is the step where source code becomes a deployable artifact. The goal is for the artifact to be reproducible, verifiable, and traceable back to a specific commit and pipeline run.
This article focuses on the building process itself. Overall pipeline management is described in CI/CD, and deployment to environments is described in Deployment.
Build environments and build agents
Build agents typically come in two main forms:
Cloud provider-managed agents
Self-hosted agents (in the cloud or on-premise)
Cloud provider-managed agents often provide lower operating costs and simpler maintenance. Self-hosted agents give you more control but full responsibility for hardening, patching, and access management.
Regardless of which model you choose, the build environment is a high-risk point. If it’s compromised, all artifacts produced there become unreliable.
Requirements for secure building
A robust build setup should have at least:
Reproducibility: Same input should produce the same artifact. Avoid hidden dependencies and undocumented manual steps.
Pinned dependencies: Lock versions explicitly and avoid “latest” in builds.
Isolation: Separate build jobs between projects with different risk levels.
Control over build tools: Compilers, base images, and plugins should be versioned and approved.
Traceability: The artifact must be traceable back to the commit, pipeline run, and build configuration.
Yes, signing naturally belongs in the build phase. Signing should happen right after the artifact is produced and verified.
Practical controls:
Sign artifacts: Sign containers, binaries, and packages with a controlled key.
Verify signature on deployment: Deployment should stop if the signature is missing or invalid.
Generate SBOM: Create an inventory of what’s in the artifact for traceability and vulnerability management.
Build provenance attestation: Document what built the artifact, where it came from, and under what conditions.
Building for AI systems
For AI solutions, building encompasses more than just application code. Trained models, supporting files for text processing, data representations, and runtime containers are all artifacts that need controlled handling throughout the process.
Version model artifacts: Treat model versions the same way as code versions.
Sign model and runtime artifacts: Use the same integrity requirements for models as for other artifacts.
Document dataset and model versions: Clearly describe what’s included in each release.
Avoid dynamic downloads in production: Don’t download model files outside of controlled release workflows.
Deployment is about controlled movement of a verified artifact to runtime environments. The goal is safe rollout, fast rollback, and predictable operations.
Deployment starts after the artifact has been built and verified. The main principle is that the same artifact is promoted across environments without rebuilding in between.
This article focuses on rollout to environments. Overall pipeline management is covered in CI/CD, and details about safe artifact production in Building.
The CI/CD flow from code change to production. The same artefact travels the whole way, and the stopping point before production is deliberate: every run should be traceable, repeatable and approved before it reaches end users.
Principles for safe deployment
To reduce operational and security risk, the deployment process should be built on:
Promotion of the same artifact: Don’t rebuild for each environment.
Verification before rollout: Deployment should stop if required checks don’t pass.
Automated and idempotent rollout: Same deployment command should produce the same result.
Traceability: Each deployment should be traceable to artifact, commit, and approval.
The same artefact is promoted between environments with no rebuild along the way. The approvals between each step are what separate a promotion from a fresh deployment — it is the same code already verified in the environment before.
What a deployment process should cover
A mature deployment process should cover at minimum:
Environment promotion (dev-test-prod): Clear gates between environments.
Approval and change control: Requirements for approval before production.
Release strategy: Support for blue/green, canary, or rolling deployment where appropriate.
Rollback: Defined and tested procedure for fast reversal.
Smoke tests after deployment: Verify that the solution actually works in the target environment.
Observability: Monitoring and alerting that catch errors early after rollout.
Access management and environment security
Deployment access must be treated as a privileged operation:
Least privilege for deployment identities: Give only access needed for deployment.
Separate roles: Separate roles for development and production deployment where possible.
Limit production deployment triggers: Restrict who can start deployment to production.
Traceability on deployments: Log who deployed what, when, and to which environment.
Deployment of AI systems
For AI solutions, the deployment process must also handle model behavior and data-driven risk:
Model rollout with control: Use gradual rollout (for example canary) to reduce impact if model behavior is wrong.
Clear version linkage: Your running solution should be traceable to the exact model version, code version, and configuration.
Verification in target environment: Run relevant quality and security checks in the target environment after deployment, not just during the build phase.
Operational monitoring: Watch model quality, response time, and signs of data or output drift.
Fast rollback: Have clear procedure to switch back to previous model version if issues occur.