Amazon’s Elastic Compute Cloud (or EC2, for short) allows you to deploy and run any application you can build in the cloud. Here at Administrate, we make extensive use of it for a number of our applications, with one of the the most exciting ones being our Cloudboot development environment.
You can read all about it in our previous blog post. To sum it up, we’ve built a Github CodeSpaces-esque secure cloud-based development environment running on an individual EC2 machine for each of our developers. This has enabled us to ensure development environment consistency across all of our teams, which in turn helped us speed up troubleshooting of developer environments and has also allowed us to work around other barriers that some developers faced, such as slow internet speeds.
How we got to EC2 Image Builder
We designed Cloudboot with the idea that it would be a secure environment that is very easy to tear down and spin back up so that a fresh environment could be spun up every time a developer starts working on a new task. With such an approach, we could guarantee that all tasks are worked from a baseline and there wouldn’t be any doubt about whether a specific issue that arises with a task is related to a dev environment misconfiguration.
In our first iteration of Cloudboot, the startup times were fairly long. Numerous steps were necessary in order for a development environment to be online and usable by a developer. On top of all the components that needed to be installed and configured (interactive shells, node, python, etc) we also needed to check out the code for each of our repositories. This drove our “time to code from nothing” up to around twenty minutes.
We needed a solution for bringing down our “time to code from nothing”, so we decided to automate the operations that would have to be done every time a development instance would be spun up.
Enter EC2 Image Builder!
The EC2 Image Builder proved to be the Amazon tool that would allow us to achieve this. The Image Builder is an AWS service that allows you to automate the generation of base EC2 images on a schedule or manually. It essentially allows you to provide it with a list of instructions on what it needs to do to generate an image that you can then use to launch an EC2 instance from. It allows users to build multiple types of images, however we are only using it for Amazon Machine Images or AMI.
For example, in the case of our developer environments, we needed the code for our applications to be checked out every time an environment was spun up. This was a time-consuming operation that we knew would have to be done on every time a developer spun up an environment, so we decided it would be a perfect candidate for automation.
We explored several ways in which this could be done, but ultimately we decided on the EC2 image builder.
How the Image Builder can help you automate tasks for your EC2 Images
The image builder relies on pipelines for creating base EC2 images which are composed of an image recipe, a build schedule, and several configuration options.
The image recipes are where most of the magic happens. The recipes allow you to define a number of steps or instructions that need to be carried out in order for the pipeline to pass successfully. The steps are referred to as components and can either be build or test components. AWS offers a wide range of pre-defined build and test components, but you also have the possibility to define your own custom components to fit any needs you might have around the generated images.
For example, the update-linux/1.0.2 component is a pre-defined AWS build component. The update-linux/1.0.2 component simply ensures that the version of Linux (we use a distribution of Linux as a base image for our Cloudboot virtual machines) is up to date. This was a step that we’d always have to do when we were launching our Cloudboot developer instances but is now done when the daily base Cloudboot image is generated via the image builder, thus saving us time when we actually launch our containers. Below are the contents of the definition file for the update-linux/1.0.2 component:
name: Update Linux
description: Updates Linux by installing all available updates via the UpdateOS action module.
schemaVersion: 1.0
phases:
- name: build
steps:
- name: UpdateOS
action: UpdateOS
timeoutSeconds: -1
inputs:
exclude:
- amazon-ssm-agent
Example of custom components we have created include dev-cloudboot-image-clone-repos, dev-cloudboot-image-cloudboot-dev-containers and dev-cloudboot-image-pull-administrate-images. They handle the cloning of all of our repositories and the pulling of the docker images for our repositories to the AMI. Below is a sample of the contents of the dev-cloudboot-image-clone-repos component:
name: clone-repos
description: Clone all ExampleCompany123 repositories.
schemaVersion: 1.0
phases:
- name: build
steps:
- name: a-front-end-repoitory
action: ExecuteBash
inputs:
commands:
- gh repo clone ExampleCompany123/ExampleCompany123 projects/a-front-end-repoitory
- git -C projects/a-front-end-repoitory remote set-url origin git@github.com:ExampleCompany123/ExampleCompany123.git
Each step in the component is defined under the steps section and they represent the sequential set of instructions that must be executed in order for a successful run of the pipeline to be registered. The steps visible in the above image include cloning one of the repositories necessary for the development environment.
In a similar fashion, the test components also include a series of steps that are sequentially executed and are aimed at ensuring that the AMI generated during the build phase is usable and has been set up as expected.
Following our ethos of defining all of our infrastructure as code, both the image builder pipelines, the image recipes, and the individual components of the recipes are defined in CDK, AWS’ newer infrastructure-as-code offering.
How to leverage the scheduling capabilities of the Image Builder
The component handling the cloning of our repos discussed above has to be run regularly in order for it to present any benefit to us, as the repos are constantly changing, with updates being made to them every day. Therefore, we set up our EC2 Image Builder pipelines to run daily
new ImageManagementStack(app, "CloudbootImage", {
env: {
account: Accounts.DEV,
region: "eu-west-1",
},
imageRecipeConstruct: CloudBootImageRecipe,
imageName: "Cloudboot Image",
schedule: Schedule.cron({ hour: "3", minute: "0" }),
instanceType: InstanceType.of(InstanceClass.BURSTABLE3, InstanceSize.XLARGE),
launchTemplate: {
name: "Cloudboot-Instance",
volumeSize: 100,
encryptionKeyArn:
"-",
},
});
Above you can see part of our infrastructure-as-code CDK code. The schedule property determines when the pipeline is run for building a new AMI. We’ve set the pipeline up to run on a daily basis, at 3 AM UTC. This means that every morning we have a new AMI available for our developers, which contains all the latest updates that have been made to our repos in the previous day.
Conclusion
EC2 Image Builder is a fantastic service for buildings AMIs that handle running all the repetitive, time-consuming scripts that you would normally have to run every time you spin up an EC2 instance.