Author: devopsy.kinsta.cloud

Kubernetes has created a whole new world for running infrastructure at scale. With the right setup, an application can go from serving a few users to millions effortlessly. Setting up Kubernetes can be tasking and can require a lot of expertise to put all the pieces together. You’ll need set up virtual or bare metal machines to use as nodes and manage SSL certificates, networking, load balancers and many other moving parts.

The introduction of Amazon Elastic Kubernetes Service (EKS) was widely applauded as it streamlines the abstraction of the complexities in an environment most organizations are already familiar with and on a provider they already trust. Amazon EKS makes creating and managing Kubernetes clusters easier with more granular controls around security and straightforward policies of how resources are used.

GitLab strives to increase developer productivity by automating repetitive tasks and allowing developers to focus on business logic. We recently introduced support for auto-creating Kubernetes clusters on Amazon EKS. In a few clicks with the right permissions, you’ll have a fully functional Kubernetes cluster on Amazon EKS. It doesn’t stop there however – GitLab also gives you the power to achieve the following use cases and more :

• Highly scalable CI/CD system using GitLab Runner: There are times like holidays when little to no changes to code are pushed to production, so why keep resources tied down? With the Amazon EKS integration with GitLab, you can install GitLab Runner with just a click and your CI/CD will run effortlessly without worrying about running out of resources.
• Shared Cluster: Maintaining multiple Kubernetes clusters can be a pain and capital intensive. With Amazon EKS, GitLab allows you to setup a cluster at Instance, Group and Project levels. Kubernetes Namespaces are created for each GitLab project when the Amazon EKS is integrated at Instance and Project level, allowing isolation and ensuring security.
• Review Apps: Reviewing changes to code or design can be tricky, you’ll need to check out your branch and run the code in a test environment. GitLab integrated with Amazon EKS deploys your app with new changes to a dynamic environment and all you need to do is click on a “View App“ button to review changes.
• AutoDevOps takes DevOps to a whole new level. AutoDevOps detects, builds, tests, deploys, and monitors your applications, leveraging the Amazon EKS integration. All you have to do is push your code and the magic happens. In this tutorial, we will deploy a sample application to the Amazon EKS cluster we will be creating using AutoDevOps.

To show you how easy it is to create an Amazon EKS cluster from GitLab, the rest of this tutorial will walk you through the steps of the integration, starting with a one-time setup of necessary resources on AWS.

One-time setup on AWS to access resources

First, we need to create a “provision” role and a “service” role on AWS to grant GitLab access to your AWS resources and set up the necessary permissions to create and manage EKS clusters. You only need to perform these steps once and you can reuse them anytime you want to perform another integration or create more clusters.

Step 1 – Create Provision Role

To grant GitLab access to your AWS resources, a “provision role” is required. Let’s create one:

1. Access GitLab Kubernetes Integration Page by clicking on the ”Kubernetes” menu for groups and Operations > Kubernetes menu for projects and click the “Add Kubernetes Cluster” button.
2. Select “Amazon EKS” in the options provided under the “Create new cluster on EKS” tab.
3. You are provided with an Account and External ID to use for authentication. Make note of these values to be used in a later step.
4. Open IAM Management Console in another tab and click on “Create Role”
5. Click on the “Another AWS account” tab and provide the Account and External ID obtained from GitLab and click Next to set permissions as shown below:
6. On the permissions page, click on “Create policy.” This will open a new tab where you can set either of the permissions below using JSON:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "autoscaling:*",
                    "cloudformation:*",
                    "ec2:*",
                    "eks:*",
                    "iam:*",
                    "ssm:*"
                ],
                "Resource": "*"
            }
        ]
    }
   

   This gives GitLab full access to create and manage resources, as seen in the image below:

   

   If you prefer limited permission, you can give GitLab the ability to create resources, but not delete them with the JSON snippet below. The drawback here is if an error is encountered during the creation process, changes will not be rolled back and you must remove resources manually. You can do this by deleting the relevant CloudFormation stack.

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "autoscaling:CreateAutoScalingGroup",
                    "autoscaling:DescribeAutoScalingGroups",
                    "autoscaling:DescribeScalingActivities",
                    "autoscaling:UpdateAutoScalingGroup",
                    "autoscaling:CreateLaunchConfiguration",
                    "autoscaling:DescribeLaunchConfigurations",
                    "cloudformation:CreateStack",
                    "cloudformation:DescribeStacks",
                    "ec2:AuthorizeSecurityGroupEgress",
                    "ec2:AuthorizeSecurityGroupIngress",
                    "ec2:RevokeSecurityGroupEgress",
                    "ec2:RevokeSecurityGroupIngress",
                    "ec2:CreateSecurityGroup",
                    "ec2:createTags",
                    "ec2:DescribeImages",
                    "ec2:DescribeKeyPairs",
                    "ec2:DescribeRegions",
                    "ec2:DescribeSecurityGroups",
                    "ec2:DescribeSubnets",
                    "ec2:DescribeVpcs",
                    "eks:CreateCluster",
                    "eks:DescribeCluster",
                    "iam:AddRoleToInstanceProfile",
                    "iam:AttachRolePolicy",
                    "iam:CreateRole",
                    "iam:CreateInstanceProfile",
                    "iam:CreateServiceLinkedRole",
                    "iam:GetRole",
                    "iam:ListRoles",
                    "iam:PassRole",
                    "ssm:GetParameters"
                ],
                "Resource": "*"
            }
        ]
    }
   

   The image below visualizes what permissions are granted:

   

7. Once the policy is created, return to the “Create Role” browser tab and refresh to see the policy we created listed. Select the policy and click “Next.”
8. In the Tags section, we don’t need to set any Tags, except if it’s required in your organization. Let’s proceed to Review.
9. Specify a Name for your new Role. You will see the policy we created listed under policies and click “Create Role” to complete the process.
10. Click on the new Role you created in the list of Roles to view its details. You may have to search for it in the list of Roles if it’s not listed in the first view. Copy the Role ARN provided – we will need it on the GitLab Kubernetes Integration page.

Step 2 – Create Service Role

The Service Role is required to allow Amazon EKS and the Kubernetes control plane to manage AWS resources on your behalf.

1. In the IAM Management Console, click on “Create Role” and select the “AWS service” tab.
2. Select EKS in the list of services and Use Cases as shown below and click Next.
3. You will notice the “AmazonEKSClusterPolicy” and “AmazonEKSServicePolicy” permissions are selected; these are all we need. Click through the Tags step and create if necessary, then click Next to get to the Review step. Click “Create Role” to complete the process.

GitLab EKS Integration

This is the easy part! As mentioned earlier, you only need to create the Provision and Service role once if you don’t already have them in your organization’s AWS setup. You can reuse the roles for other integrations or cluster creations.

1. Return to the GitLab Kubernetes Integration page and provide the Role ARN of the Provision Role we created earlier and click “Authenticate with AWS.”
2. Once authenticated, you’ll have a page to set the parameters needed to set up your cluster as shown in the image below and click on “Create Kubernetes Cluster” to let GitLab do its magic!The parameters you’ll need to provide are:
   • Kubernetes cluster name – The name you wish to give the cluster.
   • Environment scope – The GitLab environment associated with this cluster; * denotes the cluster will be used for deployments to all environments.
   • Kubernetes version – The Kubernetes version to use. Currently, the only version supported is 1.14.
   • Role name – The service role we created earlier.
   • Region – The AWS region in which the cluster will be created.
   • Key pair name – Select the key pair that you can use to connect to your worker nodes if required.
   • VPC – Select a VPC to use for your EKS Cluster resources.
   • Subnets – Choose the subnets in your VPC where your worker nodes will run.
   • Security group – Choose the security group to apply to the EKS-managed Elastic Network Interfaces that are created in your worker node subnets. AWS provides a default group, which can be used for the purpose of this guide. However, you are advised to setup up the right rules required for your resources.
   • Instance type – The AWS instance type of your worker nodes.
   • Node count – The number of worker nodes.
   • GitLab-managed cluster – Leave this checked if you want GitLab to manage namespaces and service accounts for this cluster.

   

3. The cluster creation process will take approximately 10 minutes. Once done you can proceed to install some predefined applications. At the very least, you need to install the following:
   • Helm Tiller: This is required to install the other applications.
   • Ingress: This provides SSL termination, load balancing and name-based virtual hosting you your applications. It acts as a web proxy for your application, which is useful when using AutoDevOps or deploying your own apps.
   • Cert Manager: This is a native Kubernetes certificate management controller, which helps in issuing certificates using Let’s Encrypt. You don’t need this if you want to use a custom Certificate issuer.
   • Prometheus: GitLab uses the Prometheus integration for automatic monitoring of your applications to collect metrics from Kubernetes containers allowing you to understand what is going on from within the GitLab UI.

   

4. To make use of Auto Review Apps and Auto Deploy stages of AutoDevOps, you will need to specify a Base Domain name with a wild card DNS pointing to the Ingress Endpoint generated when you Install Ingress in the list of predefined apps.

Summary

In this tutorial, we looked at how GitLab integrates with Amazon EKS, allowing Kubernetes clusters to be created easily from the GitLab UI after setting the right permissions. As we’ve seen, developer productivity is greatly improved by no longer having to manually set up clusters. Also, the same cluster can be used for multiple projects when Amazon EKS is integrated with GitLab at the Group and Instance levels, thus making onboarding new projects a breeze. After integration, the possibilities of what developers can achieve is enormous.

In the next part of this tutorial, we will look at how to deploy your applications on an Amazon EKS cluster using AutoDevOps.

In order to deliver serverless applications, customers often turn to DevOps principles to efficiently build, deploy, operate, and iterate on features and changes.

Continuous integration and continuous deployment (CI/CD) is one of the major components of DevOps that helps deliver code faster and more reliably to production.

The continuous integration offering from GitLab, an AWS Partner Network (APN) Advanced Technology Partner with the AWS DevOps Competency, provides a rich set of features for automating how new code is incorporated into your software and how new versions of your software get built and deployed.

In this post, you will learn how to create a sample serverless application using AWS Serverless Application Model (SAM) and how GitLab continuous integration helps with building, testing, and deploying the application in your Amazon Web Services (AWS) account.

About AWS Serverless Application Model

A serverless application is comprised of multiple resources, which can include functions, event sources, datastores, and other resources that work together to perform tasks.

Serverless applications are more than just AWS Lambda functions—they can include additional resources, such as APIs, databases, and event source mappings.

AWS Serverless Application Model is an open source framework for building serverless applications. SAM includes the AWS SAM Template Specification and AWS SAM Command Line Interface (CLI).

The AWS SAM Template Specification is an extension of AWS CloudFormation used to define your serverless application resources as a template. It provides a more convenient and concise syntax to express functions, APIs, databases, and event source mapping when compared to typical CloudFormation template.

The CLI enables you to build, test, and deploy serverless applications that are defined using a SAM template.

Getting Started

To get started, ensure you have your own GitLab account and AWS account. This post assumes you have required permissions to configure Gitlab CI/CD pipelines and create AWS resources in your account.

We will create a sample serverless application using the SAM CLI and deploy it to an AWS account using GitLab CI/CD pipeline. We’ll use GitLab as the repository for our source code.

When a developer checks in their new code, the GitLab continuous integration fetches the source code from the repository, and then build, package, and deploy to the AWS account.

These are the steps in this procedure:

1. Install and configure the SAM CLI.
2. Create an AWS serverless application using SAM CLI.
3. Craft the .gitlab-ci.yml file.
4. Set up your AWS credentials in your GitLab account.
5. Test locally.
6. Deploy your application.
7. Test your deployment.

Step 1: Install and Configure the SAM CLI

Some steps in this documentation use the SAM CLI. Please follow the instructions on installing SAM CLI.

If you use AWS Cloud9 as your integrated development environment (IDE), the AWS Command Line Interface (CLI), SAM CLI, Docker, and necessary Docker images are installed for you.

Create an Amazon Simple Storage Service (Amazon S3) bucket in your AWS account to store the built package for deployment. Provide required permissions to the user you will be using in later steps to deploy the SAM application to write to this bucket.

Step 2: Creating an AWS SAM Application Using SAM CLI

You can create a serverless application by defining all required resources in an AWS SAM template. AWS SAM also provides some quick-start templates to create an application. In this post, we’ll use one of those templates to create a sample application.

Before you create a SAM application, create a new GitLab project and clone it to your local environment. To create a SAM app from the CLI, open a terminal, change to the newly cloned project, and enter the following text:

sam init -r python3.8 -n gitlab-example --app-template "hello-world"

This creates a SAM app named gitlab-example using the default configuration, a single Python 3.8 function invoked by an Amazon API Gateway endpoint.

To see additional runtimes supported by SAM and options for sam init, enter sam init -h.

Git push the application back to the GitLab project.

Step 3: Crafting the .gitlab-ci.yml File

GitLab CI/CD pipelines are configured using a YAML file called .gitlab-ci.yml within each project. This defines the structure and order of the pipelines.

In a .gitlab-ci.yml file in the root of your project, place the following code and replace the #S3Bucket# with the Amazon S3 bucket name you created to store the package:

image: python:3.8

stages:
  - deploy

production:
  stage: deploy
  before_script:
    - pip3 install awscli --upgrade
    - pip3 install aws-sam-cli --upgrade
  script:
    - sam build
    - sam package --output-template-file packaged.yaml --s3-bucket #S3Bucket#
    - sam deploy --template-file packaged.yaml --stack-name gitlab-example --s3-bucket #S3Bucket# --capabilities CAPABILITY_IAM --region us-east-1
  environment: production

Let’s examine the config file more closely:

• The Docker image to use for this build is specified and pulled from the Docker hub. The image with Python 3.8 is used in this example since the sample application is written in Python 3.8.
• AWS Command Line Interface and AWS SAM CLI are installed in the script section.
• SAM builds, packages, and deploys the application.

As part of the CI/CD process, we recommend you scan your code for quality and vulnerabilities in bundled libraries. You can find these security offerings from our AWS Lambda Technology Partners.

Step 4: Setting Up Your AWS Credentials with GitLab

To interact with your AWS account, the GitLab CI/CD pipelines require both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be defined in your GitLab project settings. You can do this under Settings > CI/CD > Variables. For more information, please see this GitLab ReadMe.

The AWS credentials you provide must include AWS Identity and Access Management (IAM) policies that provision correct access control to AWS Lambda, Amazon API Gateway, AWS CloudFormation, and IAM resources.

Step 5: Testing Locally

SAM provides functionality to test your applications locally. As we mentioned earlier, you must have SAM CLI installed locally for you to test locally.

SAM provides a default event in events/event.json that includes a message body of {\"message\": \"hello world\"}. If you pass that event into the HelloWorldFunction, it should respond with the same body:

sam local invoke HelloWorldFunction -e events/event.json

Output should be:

{"message": "hello world"}

After you confirm the Lambda function is working as expected, you can test the API Gateway using the following steps.

First start the API locally:

sam local start-api

SAM again launches a Docker container, this time with a mocked API Gateway listening on localhost:3000. Now, you can call the hello API as follows:

curl http://127.0.0.1:3000/hello

Output again should be:

{"message": "hello world"}

Step 6: Deploying Your Application

Git push the changes to your GitLab repository, and the GitLab pipeline will automatically deploy your application. If your build and deploy are successful, go to the next step to test your deployed application.

If your build fails, please take a look at the build log to see why it failed. Some common reasons are:

• Incompatible versions of software (for example, the Python runtime version may be different from the Python on the build machine). Address this by installing the proper versions of the software. If your build machine has an incompatible version of the software, install the proper version via the .gitlab-ci.yml file.
• You may not be able to access your AWS account from GitLab. Check the environment variables you configured with AWS credentials under the CI/CD settings in GitLab.
• You may not have permission to deploy a serverless application. Make sure you provide all required permissions to deploy a serverless application. Your credentials must have permissions to the S3 bucket, in addition to being able to create the necessary resources of the sample SAM app.

Step 7: Testing Your Application

To test the application you deployed, go to the build log and click on Show Complete Raw in the upper right corner.

Look for the HelloWorldApi – API Gateway endpoint similar to shown below.

Finally, use curl to test the API:

curl https://<api-id>.execute-api.us-east-1.amazonaws.com/Prod/hello/

Output should be:

{"message": "hello world"}

Conclusion

In this post, you used GitLab continuous integration and AWS Serverless Application Model to create, build, test, and deploy a serverless application. With GitLab, users can build and deploy serverless applications with minimal configuration to take advantage of continuous integration for serverless application development.

Learn more about GitLab and AWS Serverless Application Model (SAM).

.

.

GitLab – APN Partner Spotlight

GitLab is an AWS DevOps Competency Partner. Its continuous integration offering provides a rich set of features for automating how new code is incorporated into your software and how new versions of your software get built and deployed.

Contact GitLab | Solution Overview | AWS Marketplace

*Already worked with GitLab? Rate this Partner

*To review an APN Partner, you must be an AWS customer that has worked with them directly on a project.

Ask Media Group operates over 30 websites and provides enriched search results, articles, galleries, and shopping sites to over 100 million unique visitors each month. About two years ago, Ask Media was looking for ways to grow the business, draw advertisers, and expand its audience. Routine tasks like onboarding developers or releasing software took too long. The monolithic system that was in place had limited capabilities and added financial burdens for services that went unused.

Chenglim Ear, principal software engineer at Ask Media, recently sat down with Trevor Hansen, solutions architect at AWS, to discuss how adopting GitLab empowered developers to improve the customer experience, release software quicker, and leverage AWS cloud services.

Building microservices from monoliths

Ask Media was looking to move away from a monolithic system to microservices in order to modernize workflow and improve the overall business process. “We wanted to move over to microservices. We wanted to leverage Kubernetes. It was a new container world that was shaping. When we looked at GitLab, it was very complete in providing what we needed to be able to build images, to run on containers,” according to Chenglim. “That was a very big deciding factor. GitLab had everything that we needed.”

Developers can now break services into multiples and develop them independently, own the code, and have full visibility prior to deployment. “We’re making the hidden logic transparent and we enable the parts of the logic to be independently developed in parallel. So you can have developers all working on their own, with different skillsets,” Chenglim says.

Containers, cost, and scalability

“We needed a system that could handle change. When we look at what we did to speed up development, make it simple and transparent, and control the cost, we see a paradigm shift. GitLab gave us push-button releases. Docker and Kubernetes enabled us to switch to a microservices architecture and AWS enabled auto scaling,” says Chenglim. “On Amazon, we started building Kubernetes clusters and GitLab became our command and control interface.”

Ask Media was looking for a tool that could scale and grow as needed. Cost, speed, and functionality are the tenets that AWS focuses on providing to its customers, according to Hansen. AWS works closely with Ask Media to ensure that the containers in place offer the scalability, flexibility, and timeliness they need.

With GitLab and AWS, Ask Media developers built out a platform that enables the knowledge from all members of the teams. “With AWS, we wanted a product that was fairly complete and mature. AWS has a lot of history and lots of services. We definitely wanted to be able to leverage those services and to build on a platform that was a solid,” Chenglim says. “We set off to build Kubernetes clusters, right on EC2 instances. We continue to look at opportunities to leverage the resources available through AWS.”

To learn more about how Ask Media made the transition to cloud native, check out the full webcast.