Contact us anytime to know more - Kuldeep K., Founder & CEO CISIN
Building an efficient software manufacturing facility relies on an effective DevSecOps pipeline, including continuous integration and deployment, monitoring/logging/auditing/governance, and operations processes in place. Automating vulnerability identification during early software development stages can lower costs while speeding the implementation of changes faster.
Organizations may utilize various cloud-based and third-party services to identify vulnerabilities in their DevSecOps pipelines, including AWS native tools and third-party ones from third parties. Aggregating findings manually is complex. AWS makes the task easy by offering tools and services needed for successful vulnerability identification and integration between them all - whether native to AWS or third-party tools from various vendors; its services even aggregate security findings.
This post presents an AWS DevSecOps reference architecture that encompasses all the practices mentioned previously, such as SCA (Software Composite Analysis), SAST(Static Application Security Testing), DAST(Dynamic Application Security Testing), aggregation and vulnerability identification into one pane; it also covers concepts like the security of pipelines as well as within pipelines.
This pipeline can be deployed within any standard AWS Region, AWS Region US in particular; however, all listed AWS services are currently only authorized to run FedRAMP High workloads within this Region with AWS CodePipeline and Security Hub both currently undergoing JAB reviews and will hopefully soon receive their final approval statuses.
Developers working on software development at a lightning pace need reliable deployment pipelines that ensure reliable workflows while upholding high levels of reliability and security. DevSecOps combines Development, Security, and Operations into one approach to help integrate software security practices in the development process and its deployment from inception. Businesses often choose Amazon Web Services (AWS) when developing and deploying applications on cloud infrastructures like AWS. To effectively implement DevSecOps with AWS, building an effective Continuous Integration and Continuous Deployment pipeline (CI/CD) with open-source tools is critical for success - this article demonstrates this method.
Want More Information About Our Services? Talk to Our Consultants!
What is DevSecOps?
DevSecOps is an evolution of DevOps, an emerging culture that strives to bridge the divide between operations and development teams. Recognizing security's centrality throughout software lifecycle development processes early, DevSecOps fosters collaboration among development, security, and operations teams to make security part of every software development effort process.
Service and Tools
This section discusses the AWS services, third-party tools, and other components utilized to build this solution.
Services CI/CD
We utilize AWS Services for Continuous Integration/Continuous Deployment:
- AWS CodeBuild is a fully managed continuous integration service that compiles Code, performs tests, and packages software ready for deployment.
- AWS CodeCommit provides fully managed source control services hosting Git repositories.
- AWS CodeDeploy - is a managed service to automate software deployment across various computing services such as Amazon Elastic Compute Cloud, AWS Fargate, and AWS Lambda.
- AWS CodePipeline provides a fully managed continuous delivery service designed to automate release pipelines to provide fast, dependable updates for applications and infrastructure.
- AWS Lambda provides an effortless way to run Code on AWS without needing to manage or provision servers yourself, paying only for compute time consumed.
- Amazon Simple Notification Service - Amazon SNS provides a fully managed messaging service designed for application-to-application and person-to-person communication, amongst other uses.
- Amazon Simple Storage Service (S3) provides internet-based data storage that you can easily use from anywhere worldwide at any given time.
- Amazon S3 makes retrieving and storing any kind of file easy from any place online at any given moment - the AWS Systems Manager Parameter store gives you control and visibility into your AWS infrastructure for maximum control over it.
Testing Tools For Continuous Testing
- Your GovCloud subscription does not currently support Amazon CodeGuru for static code analyses - as an open-source static code analysis tool, it may provide valuable assistance - however, other tools better suit your needs, such as Java or Python development environments.
- OWASP Dependency-Check - This software composition analysis (SCA) tool attempts to locate publicly disclosed vulnerabilities within your project's dependencies. At the same time, SonarQube SAST's thousands of rules for automated Static Code Analysis help detect bugs quickly.
- It specializes in discovering bugs in Code without actually running it. It can detect entire categories of errors before any tests are written. The OWASP Zap is an automated vulnerability assessment solution that assists web application developers as they create and test them to quickly detect vulnerabilities automatically as part of development/test cycles.
Services Of Continuous Monitoring And Logging
AWS provides several Monitoring and logging services to keep an eye on ongoing activities:
- AWS CloudWatch Logs gives you access to log files generated from Amazon EC2 Instances, AWS CloudTrail and Route 53 instances in near real-time, and cloudWatch events, which detail changes made in AWS resources in near real-time.
Services Of Auditing And Governance
AWS provides auditing and governance services:
- AWS CloudTrail Provides AWS accounts with audit capabilities for governance, compliance, and operational risk auditing.
- AWS Identity and Access Management (IAM) allows you to control access to AWS resources and services securely. IAM allows you to create AWS groups and users, administer them, and assign permissions so they may gain entry.
- With this tool, you can assess and audit AWS resource configurations.
Operation Services
AWS provides operations with various services:
- AWS Security Hub - gives a consolidated overview of all AWS Security Alerts in one convenient place, making vulnerability information readily available in one window. AWS CloudFormation gives an easy, consistent way of managing resources throughout their lifespan by treating them like Code.
- AWS System Manager Parameter Store - Provides secure, hierarchical data storage to manage configuration data, secrets, and sensitive information securely and reliably. As parameter values, you can store information like database strings,
- Amazon Machine Image IDs (AMIs), license codes, and passwords relating to Amazon Machine Image IDs (AMIs).
- AWS Elastic Beanstalk - an intuitive service that makes deploying and scaling web apps written in Java,.NET PHP, Node.js, or Node.js easy - allows for the rapid deployment and scaling of Java-,.NET PHP- or Node. Js-powered websites developed on popular servers like Apache, Nginx Passenger, or IIS. Elastic
- Beanstalk was used here to deploy WordPress with Amazon Aurora MySQL using Elastic Beanstalk as part of the LAMP stack deployment process - although Elastic Beanstalk can also be configured for deployment elsewhere on AWS environments or anywhere else as desired.
Pipeline Architecture
This diagram depicts the AWS DevSecOps CICD pipeline architecture.
- These are the main steps: CodePipeline is initiated when a CodeCommit repository receives a CloudWatch Event.
- CodeBuild uploads artifacts into an S3 bucket for CodeBuild to use with authentication data (like scanning tool tokens retrieved from Parameter Store) to begin scanning them. However, we recommend keeping them in artifact repositories like AWS CodeArtifact for added ease and simplicity. S3 will still be utilized.
- CodeBuild performs code quality analyses using SCA (OWASP dependency-check) and SAST tools (SonarQube, PHPStan). CloudFormation Template allows selecting an SCA tool; however, CodeBuild allows for any third-party tool to be brought along too.
- CodeBuild will invoke a Lambda Function when vulnerabilities are identified during SCA or SAST analyses. It will convert AWS Security Finding Format (ASFF) reports into Security Hub aggregate and display them on one pane conveniently. Lambda also uploads scanning results directly into an S3 bucket.
- CodeDeploy can deploy Code directly into Elastic Beanstalk staging without security vulnerabilities being present. At the same time, CodeBuild will initiate DAST scans with the OWASP ZAP Tool after successful deployment (this feature may also be activated for Bring Your Tool approaches).
- Lambda functions are used if there are vulnerabilities, parsing ASFF documents into ASFF format before uploading results directly into Security Hub via the Security Hub Upload function, similar to Step 4. Additionally, the S3 bucket upload functionality uploads scanning results as per Step 4 below.
- Approval will only occur if no security vulnerabilities are present; an email notification will then be sent out to the approver for review and code deployment on the Elastic Beanstalk production environment by CodeDeploy. CloudWatch Events records any changes to build states as they occur and notifies users via SNS of those events.
- CloudTrail is an API call tracking and monitoring solution. The tool notifies of critical events like updates to pipelines (updatePipeline/deletePipeline), creates Projects with CodeBuild (createProject and deleteProject), as well as audit trail projects created from them (CreateProject/DeleteProject).
- AWS Config provides an audit trail of all changes made to AWS services. It has added, as part of a security best practice, these AWS Configuration rules:
- CODEBUILD_PROJECT_ENVVAR_AWSCRED_CHECK - Checks whether a project contains environment variables AWS_ACCESS_KEY_ID and
- AWS_SECRET_ACCESS_KEY for AWS ACCESS Key ID/Access Key Secret Access Key credentials in its environment variables; otherwise, this rule will fail as NON_COMPLIANT.
- CLOUD_TRAIL_LOG_FILE_VALIDATION_ENABLED - Verifies whether CloudTrail creates a signed digest file containing its logs. AWS suggests file validation should be enabled on all trails; otherwise, this rule won't comply.
IAM roles, S3 bucket policies, and IAM roles can all help restrict pipeline access. Encryption and SSL-secured transport protect pipeline data during transit and storage; Parameter Store stores sensitive information securely, such as API tokens or passwords, while other forms of MFA might also be necessary to remain compliant with frameworks like FedRAMP.
Checks, including SCA, SAST, and DAST, implement security in pipeline processes. This pipeline could also employ IAST techniques (Interactive Application Security Testing), which combine SAST with DAST testing methods. Encryption must always be employed when transmitting or transporting Code or artifacts containing sensitive material - including in transit.
The following section provides instructions on how to run and deploy the CloudFormation pipeline template used in this example. Clicking any links provided can give more detail regarding each service that makes up this pipeline. For extra safety when creating pipelines using CloudFormation, we advise using software like CFN Nag Linting, which checks for security issues before creating infrastructure pipelines.
Prerequisites
Starting, you will require:
- Elastic Beanstalk environments with applications deployed are known as environments. Other apps may also be supported; please see Deploying high-availability WordPress websites with an Amazon RDS external database onto
- Elastic Beanstalk for further details.
- Your application code should be stored in an AWS CodeCommit repository - for more information, see Create an AWS CodeCommit Repository.
- Input the build spec files (.yml and sonar-project properties file), sonar project properties file (sonar project properties), JSON file, and PHPstan.neon into its root repository directory for upload.
- Lambda code uploaded to an S3 bucket is used to analyze scan reports and upload results into Security Hub.
- Security Hub also accepts SonarQube API tokens and URLs for scanning Code, an OWASP ZAP URL dynamic website scanner, and application testing URLs (see below for examples).
- Email notifications regarding approvals, pipeline changes, CloudTrail events, and notifications about CloudTrail will also be delivered here.
- AWS Configuration Recorder or manually Enabling Security Hub could already be activated; for details, please see either article (Managing Configuration Recorder versus Manually Enabling the Security Hub ).
Pipeline Deployment
Following these steps is necessary to deploy your pipeline successfully: Download the CloudFormation Template and pipeline code from the Google Cloud Platform.
If you don't already have an AWS account, create one now by going through these steps:
- Detail your code repository name, the branch initiating the pipeline, SAST Tool URL (if using PHPStan), API Token, and API Tool URL under SonarQube information as necessary.
- Select DAST and OWASP Zap, its dynamic testing tool, as required for dynamic analysis.
- Enter an API token, DAST tool URL address, application domain to be tested, and Lambda Function S3 Bucket Name, Filename, and Handler name under Lambda Functions to complete testing.
- Enter Elastic Beanstalk environments and details regarding applications used for production and staging, along with email notifications of approvals or changes in pipeline status.
CloudFormation Template deployment is now complete. Once the pipeline has been installed, confirm your subscription using the link in your notification email and complete this step. Please be aware that this post's CloudFormation Template was intended to run in AWS Government Cloud; to run in other regions, adjust the partition name accordingly, for example, from arn-aws:us-gov to arn-aws in CloudFormation template partition name settings.
The Pipeline
Add changes to application repository files to trigger CloudWatch events and activate your pipeline. CodeBuild then scans Code for vulnerabilities before calling Lambda to upload results to Security Hub.
The severity level must be indicated when providing information to the Security Hub about vulnerabilities. Security Hub will assign labels according to this parameter; you may choose a value suited to your company's needs by altering this field in your Code.
- INFORMATIONAL
- 1 to 39 Low
- 40- 69 - MEDIUM
- High-end (70-89)
- 90 to 100 CRITICAL
Read More: Best 15 CI/CD Tools You Must Know 2023
SCA scanning and SAST scan
CodeBuild can automatically trigger SCA scanning and SAST scanning simultaneously; this AWS CI/CD Architecture section discusses their usage using OWASP SonarQube and PHPStan as examples of such scans.
SCA (OWASP Dependency Check):
Conducting Scans using OWASP Dependency-Check: Within this Lambda Function code fragment, which parses SCA results and posts them directly into Security Hub for further processing, an equivalent Security Hub severity level (normalized_severity) is assigned based on those results.
Scan with SonarQube SAST:
Here is an excerpt of Code from a Lambda Function, which parses SonarQube results and posts them directly into Security Hub; each SonarQube result is assigned its equivalent severity level within Security Hub as determined by SonarQube results.
PHPStan Scanning Code Example:
Here is an excerpt of the Code used for parsing PHPStan scan results and posting them directly into Security Hub from a Lambda Function.
DAST scan
CodeBuild is responsible for initiating DAST scans and tools in our architecture. Once no security vulnerabilities were identified in SAST scanning, manual approval would follow if no security vulnerabilities were present in SAST analysis. An email would then be sent notifying an approver who can review or reject deployment at their discretion - with deployment proceeding forward once approved or rejected based on these decisions.
Scanning with OWASP Zap:
CodeBuild initiates DAST scanning after successful deployment and, should any vulnerabilities be discovered, will activate a Lambda function similar to SAST, which posts results of parsing back to Security Hub - here is an example code sample for such functions.
The Security Hub can automate remediation by consolidating vulnerability reports. You can, for instance, trigger a Lambda Function when finding vulnerabilities to implement remediation efforts quickly and seamlessly - while at the same time alleviating operational and security staff workload as they no longer need to log into separate dashboards for monitoring purposes.
DevSecOps: Key principles
- Automated security: tests and checks should be implemented throughout the CI/CD pipeline, including vulnerability scans, code analyses, and infrastructure verifications.
- Shift Right: Security processes should be placed on the left-hand side when planning for development. Early identification and correction of vulnerabilities is of critical importance.
- Immutable infrastructure: Switch to immutable infrastructure where resources can be replaced instead of changed to reduce attack surfaces and streamline rollback processes.
- Continuous Monitoring: This can be implemented to continuously assess applications and infrastructure to detect security concerns and address them immediately and quickly.
- Coding to Achieve Compliance: Code can be used to define security and compliance policies, which can then be checked automatically.
AWS DevSecOps Pipeline for CI/CD
Open source tools allow developers to quickly build an AWS DevSecOps CI/CD Pipeline that integrates seamlessly. Here are the essential steps necessary for setting up such an environment.
- Source Code Repository: Host your source code on an authoritative repository such as GitLab or GitHub for versioning control purposes and use Infrastructure as Code to manage AWS resources to maintain version control over them.
- Continuous Integration (CI): You have various continuous integration servers at your disposal: Travis CI, CircleCI, or Jenkins can serve as an excellent CI server; open source tools can automatically test and build your Code while also being integrated with AWS for automating deployments.
- Static Code Analysis: Use tools such as SonarQube and ESLint for static code analysis in your pipeline to quickly spot issues with code quality in an early development cycle. These will assist with quickly spotting any potential code-based issues or quality concerns early.
- Containerization: Containerization can be achieved via Docker, an open-source tool. Creating images created with Docker ensures consistency across environments by packaging apps and their dependencies in an image file created through this approach.
- Vulnerability Scanning: Clair and Trivy can help detect security vulnerabilities within container images stored on the AWS Elastic Container Registry with services like Amazon ECR Scan.
- Infrastructure as Code: Tools like Terraform or AWS CloudFormation should be utilized to manage and provision your infrastructure effectively. Having auditable control is vital when using infrastructure as code solutions.
- Automated Deployment: For reliable deployment automation tools such as Ansible and Chef to ensure consistent, secure rollouts on AWS resources. Furthermore, CodeDeploy can also help deploy applications onto this cloud service platform.
- Scanners for Security: Security scanners like AWS GuardDuty and Security Hub can continuously monitor your AWS environment to detect security risks, such as vulnerabilities and threats in AWS environments. These tools offer 24/7 surveillance.
- Complying With The Code: Use tools like AWS Config Rules or Open Policy Agent to enforce and define security policies within the Code to guarantee that all infrastructure you employ meets all compliance standards and is compliant.
- Orchestrated By Orchestration: Manage complex workflows using AWS Step Functions or Apache Airflow for effortless management.
- Continuous Monitoring: Prometheus or Grafana tools, such as AWS CloudWatch, are the go-to choices for comprehensive application performance monitoring and infrastructure security assessments.
The Key Components in an AWS DevSecOps Pipeline for CI/CD
Infrastructure Automation:
Automation as Code is an integral component of DevSecOps Pipelines. It facilitates provisioning AWS services while preserving traceability and consistency across environments, with tools like Terraform and AWS CloudFormation offering solution creation with versioned infrastructure management for easier administration.
Management Secrets:
Proper credentials and secrets management are paramount, especially with sensitive data. Tools like HashiCorp Vault (or AWS Secrets Manager) provide secure storage solutions for this. Incorporating these tools into your CI/CD process will protect sensitive data.
Tests and Quality Assurance:
A comprehensive CI/CD system should include unit, integration, and end-to-end tests as part of its quality assurance plan. You can validate your application using open-source frameworks like JUnit and Selenium to validate it before going live with it.
Regulation and Governance:
For optimal standards compliance, implementing a Governance-as-Code approach is of critical importance. AWS Organizations, AWS Configuration, and open source tools like Open Policy Agent can all assist with managing standards and policies across your infrastructure.
Logging and Monitoring:
Real-time application monitoring and performance are paramount, which makes utilizing open-source tools like ElasticStack (Elasticsearch Logstash Kibana), Prometheus Grafana, or AWS CloudWatch with powerful monitoring/logging features essential.
End-to-end AWS DevSecOps Pipeline CI/CD Benefits
Increase Security:
Integrating security early into development processes allows security risks and vulnerabilities to be identified early, with consequent mitigation efforts implemented and decreased chances of AWS CI/CD Services data and security breaches occurring later in production.
Quick and Efficient:
While DevSecOps streamlines software development processes and promotes more rapid updates, its automation of testing and deployment helps minimize human intervention while reducing errors substantially.
Improved Collaboration:
DevSecOps encourages cross-functional collaboration across development, operations, and security departments for enhanced accountability and responsibility sharing among teams within these fields.
Cost Efficiency:
Scalability and provisioning within cloud infrastructure can optimize resources while simultaneously decreasing costs. Utilizing software-controlled scaling allows for the more effective allocation of resources and more cost savings overall.
Continuous Feedback:
DevSecOps encourages an environment for continuous improvement by giving teams real-time access to real-time feedback and monitoring systems that quickly identify issues so that issues can be fixed faster for improved software quality.
Scalability:
AWS' cloud-native architectural design makes scaling your pipeline with your application easy; adjust accordingly without manual intervention as workloads increase.
Auditing and Compliance:
By adopting compliance as a Code, organizations can automate their security and compliance policy checks - simplifying auditing and making compliance demonstration easier for regulatory agencies.
Want More Information About Our Services? Talk to Our Consultants!
The Conclusion Of The Article Is:
DevSecOps on AWS, with its open-source tools and end-to-end Continuous Integration/Continuous Deployment, is no longer a luxury; it has become essential in today's rapidly advancing tech environment. DevSecOps helps organizations deploy software rapidly while still protecting privacy - businesses should adopt this practice to respond to market needs faster and increase security posture significantly while cutting costs with cloud-native and automated solutions.
DevSecOps tools and methods must evolve as the technology landscape and security shift. Acquia CI/CD Staying abreast of new tools, technologies, and best practices for DevSecOps projects on AWS remains essential to their continued success.
DevSecOps pipelines that integrate open-source tools with AWS DevSecOps platforms offer an efficient way of improving security while increasing efficiency during software deployment and development processes. Automating security checks and integrating security practices from the beginning will allow you to identify vulnerabilities more quickly, reduce security risks more efficiently, increase the reliability of AWS apps, and ensure maximum uptime - DevSecOps is much more than simply a method or toolkit; rather, it represents a cultural shift that represents collaboration in an approach which prioritizes speed while protecting security - something crucial given today's fast-evolving threat landscape where speed must not compromise security.
