Infrastructure as Code (IaC) is the managing and provisioning of infrastructure through code instead of through manual processes. It enables the automated and repeatable creation and maintenance of all virtualized infrastructures. Both Terraform and Crossplane support multiple providers and are great for operating cross-cloud. In this blog post, I wanted to give you a brief overview of both and to give you some insight into how we can keep Infrastructure as Code as clean as possible.
Terraform
Terraform is a tool for building, changing, and versioning infrastructure in a quick and efficient manner. It can manage existing and popular service providers and custom in-house solutions.
I’d like to begin by stating that I’m a big fan of Terraform with its overlay in the form of Terragrunt. To me, they’re absolutely essential when it comes to creating Cloud components. Of course, there are also great tools such as AWS Cloud Formation or Azure Resource Manager but the great thing about Terraform is that you can manage almost all Cloud Providers via one tool. I started using it when it was in its infancy and throughout my journey. There were good days and bad days - just like in a typical relationship.
Even though the tool is very good I found some cons that may overshadow its pros. One of them is a lack of so-called environment state observability, which in my scenario would be something like:
If the created Cloud component suddenly and not intentionally disappears, please recreate it immediately.
The second is the very lengthy command execution - when I run terragrunt apply and there is no local cache, sometimes I have to wait a few minutes to get the yes | no prompt. Even if my module is very small, Terraform needs to connect to external servers and get the dependencies, handle them, test the code, plan it and so on. While doing this once a day is acceptable, but when you are testing something and you need to delete the infrastructure’s components and apply them again and again, it is quite irritating.
Finding a new way to set the Cloud Infrastructure
Those two cons were the trigger to finding a new way to set the Cloud Infrastructure. To handle events I could create some event based actions which could be triggered by the change in the infrastructure. The event could then perform some action such as recreating the part of an invalid cloud component. This could be done via AWS CloudTrail or Lambda but as we see, it adds additional effort and complexity into our environment.
To get rid of long execution time we may write smaller modules or choose not to delete the cached data in the .terragrunt-cache. Of course it may help but when we perform changes in our code, we want to test it on the remote Cloud infrastructure and waiting N-times multiplied by e.g. 60s gives us a nice couple of minutes when we are taking a look at the Terraform log. That is a little bit of waste that we want to avoid. So maybe there is another way to create a Cloud infrastructure and have an event-based and less execution time approach.
Using an API to configure the Cloud infrastructure
I came across an interesting presentation in which the so-called Kubernetes Cluster API as well as declare Configuration as Data were mentioned. In short, if you do declare your Kubernetes configuration in YAML syntax, why not use its API to configure the Cloud infrastructure and declare it via YAML too? Wow! That is something new which is worth reading and trying… but it reveals some downsides - you must have a Kubernetes up-and-running. So for all of those scenarios when we do not use that kind of service and even if we want to we do not have knowledge about it - that may be a problem in implementing such an approach. Let’s skip it for now and assume that we use Kubernetes in our infrastructure and using the Cluster API for provisioning an infrastructure will be possible. Now it is time to find out which tools will make it happen.
As always we may find some Cloud Provider specific tools like Kubernetes Cluster API Provider AWS, AWS Controllers, GCP Config Connector or Azure Service Operator. But such services are good for one Cloud provider which is fine, but we have to use separate logic in usage for AWS or Azure. It would be great to have something like Terraform - a Swiss Army Knife that could manage many providers at one time. I found that the Crossplane offers these options, at least for the main providers like AWS, Azure, GCP and Alibaba. So that will give us the possibility to cover such scenario like:
Run our startup in AWS and ...oh! Wait, our Business decided to move it immediately to Azure - apply it then!
The above scenario is quite complicated for those who use Terraform mainly for AWS and do not have the code for Azure. In this situation they have to write modules, run tests and so on. It is possible but it takes some time. With declarative YAML and Cloud Operator it should be much easier and faster. Let’s test the Crossplane with AWS and the local Kubernetes cluster based on kind.
Crossplane
Testing Crossplane with AWS, Kubernetes and kind
First we have to have an AWS IAM User which credentials we have to provide into Crossplane’s configuration. This step needs to be done before we run the Crossplane.
$ aws iam create-user --user-name crossplane-admin
{
"User": {
"Path": "/",
"UserName": "crossplane-admin",
"UserId": "AIDA3N32HB6CP7WBGF7TM",
"Arn": "arn:aws:iam::123456789012:user/crossplane-admin",
"CreateDate": "2021-04-09T11:20:31+00:00"
}
}
$ aws iam attach-user-policy --user-name crossplane-admin --policy-arn arn:aws:iam::aws:policy/AdministratorAccess
$ aws iam create-access-key --user-name crossplane-admin
{
"AccessKey": {
"UserName": "crossplane-adminx",
"AccessKeyId": "AKIA<secret>",
"Status": "Active",
"SecretAccessKey": "<secret>",
"CreateDate": "2021-04-09T11:30:43+00:00"
}
}
After above we have a crossplane-admin user and AWS Access Keys which will be used in the Crossplane’s configuration. Now we may jump into our local Kubernetes server and install the Crossplane.
- Create valid cluster with v1.19.7 kind Docker image
$ kind create cluster --image kindest/node:v1.19.7 --wait 5m
...
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
- Install Crossplane CLI
$ curl -sL https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh | sh- Install Crossplane
$ kubectl create namespace crossplane-system
$ kubectl get namespaces
NAME STATUS AGE
crossplane-system Active 58s
...
$ helm repo add crossplane-stable https://charts.crossplane.io/stable
$ helm repo update
$ helm install crossplane --namespace crossplane-system crossplane-stable/crossplane --version 1.1.1
$ helm list -n crossplane-system
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
crossplane crossplane-system 1 2021-04-09 13:57:42.518737 +0200 CEST deployed crossplane-1.1.1 1.1.1
$ kubectl get all -n crossplane-system
NAME READY STATUS RESTARTS AGE
pod/crossplane-6c9c8bd9c7-dscb9 1/1 Running 0 40s
pod/crossplane-rbac-manager-5c8fff46f8-qj5l4 1/1 Running 0 40s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/crossplane 1/1 1 1 40s
deployment.apps/crossplane-rbac-manager 1/1 1 1 40s
NAME DESIRED CURRENT READY AGE
replicaset.apps/crossplane-6c9c8bd9c7 1 1 1 40s
replicaset.apps/crossplane-rbac-manager-5c8fff46f8 1 1 1 40s
- Install Configuration Package (v1.1.1)
$ kubectl crossplane install configuration registry.upbound.io/xp/getting-started-with-aws-with-vpc:v1.1.1
$ kubectl get configuration
NAME INSTALLED HEALTHY PACKAGE AGE
xp-getting-started-with-aws-with-vpc True False registry.upbound.io/xp/getting-started-with-aws-with-vpc:v1.1.1 84s
- Configure AWS Account data and create the secret
$ AWS_PROFILE=default && echo -e "[default]\naws_access_key_id = $(aws configure get aws_access_key_id --profile $AWS_PROFILE)\naws_secret_access_key = $(aws configure get aws_secret_access_key --profile $AWS_PROFILE)" > creds.conf
$ kubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./creds.conf
- Configure the Provider
$ cat ./provider.yaml
---
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
name: default
spec:
credentials:
source: Secret
secretRef:
namespace: crossplane-system
name: aws-creds
key: creds
$ kubectl apply -f ./provider.yaml
$ kubectl get providerconfigs -o wide
NAME AGE SECRET-NAME
default 19s
Provisioning the AWS infrastructure
After completing the above steps, we need to provision the AWS infrastructure! Now it is time to provision the VPC with Security Group for the MySQL access. To do this we may use an available example configuration such as the crossplane/provider-aws repository but unfortunately there is no such example for the VPC so we have to handle it ourselves. That is good training to get familiar with Crossplane’s Custom Resource Definitions (CRD). To get into, we have to find out how to define our own YAML for the VPC. To do this we must go to package/crds/ec2.aws.crossplane.io_vpcs.yaml file and look closer into it.
We see that the spec.group is equal ec2.aws.crossplane.io and spec.versions.name is v1beta1. For metadata we may put typical fields like name and labels. But where to define VPC’s parameters like cidr or even tags?
This is defined in the section: spec.versions.schema.openAPIV3Schema.properties.spec.properties.forProviders field. We may see the keys like cidrBlock, enableDnsHostNames, enableDnsSupport or tags. This is all we need to create our YAML for the VPC.
---
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: VPC
metadata:
name: mgmt-vpc
labels:
provider: aws
vpc: “mgmt-vpc”
spec:
forProvider:
cidrBlock: “10.1.0.0/16”
enableDnsHostNames: true
enableDnsSupport: true
instanceTenancy: default
region: eu-central-1
tags:
-
key: Name
value: mgmt-vpc
-
key: Manager
value: crossplane
providerConfigRef:
name: default
The value providerConfigRef contains the reference to the ProviderConfig. In the same way we have to configure our SecurityGroup resource and the result should be like below:
---
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: SecurityGroup
metadata:
name: db-mysql-access
spec:
forProvider:
region: eu-central-1
vpcIdSelector:
matchLabels:
vpc: mgmt-vpc
groupName: db-mysql-access
description: Allow access to MySQL
ingress:
-
fromPort: 3306
toPort: 3306
ipProtocol: tcp
ipRanges:
-
cidrIp: "10.1.10.0/0"
description: Production
Please note the parameter spec.forProvider.vpcIdSelector.matchLabels that has a value vpc: mgmt-vpc It is needed because we want to create our Security Group after the VPC.
The above configuration should be placed in the vpc_sg.yaml file and executed by
$ kubectl apply -f vpc_sg.yaml
Let’s check the result!
$ kubectl get vpc -o wide
NAME READY SYNCED ID CIDR AGE
mgmt-vpc True True vpc-0ec630f4ffb2d3d03 10.1.0.0/16 11s
$ kubectl get securitygroup -o wide
NAME READY SYNCED ID VPC AGE
db-mysql-access False True sg-011c1557dd5278c97 vpc-0ec630f4ffb2d3d03 20m
$ kubectl describe vpc
Name: mgmt-vpc
Namespace:
Labels: provider=aws
vpc=mgmt-vpc
Annotations: crossplane.io/external-name: vpc-0ec630f4ffb2d3d03
...
Spec:
For Provider:
Cidr Block: 10.1.0.0/16
Enable Dns Host Names: true
Enable Dns Support: true
Instance Tenancy: default
Region: eu-central-1
Tags:
Key: Manager
Value: crossplane
Key: Name
Value: mgmt-vpc
Key: crossplane-kind
Value: vpc.ec2.aws.crossplane.io
Key: crossplane-name
Value: mgmt-vpc
Key: crossplane-providerconfig
Value: default
...
$ kubectl describe securitygroup
Name: db-mysql-access
...
Spec:
For Provider:
Description: Allow access to MySQL
Group Name: db-mysql-access
Ingress:
From Port: 3306
Ip Protocol: tcp
Ip Ranges:
Cidr Ip: 10.1.10.0/0
Description: Production
To Port: 3306
Region: eu-central-1
Vpc Id: vpc-0ec630f4ffb2d3d03
Vpc Id Ref:
Name: mgmt-vpc
Vpc Id Selector:
Match Labels:
Vpc: mgmt-vpc
$ aws ec2 describe-vpcs --region eu-central-1 --output yaml
Vpcs:
- CidrBlock: 10.1.0.0/16
CidrBlockAssociationSet:
- AssociationId: vpc-cidr-assoc-08b287574b16e1243
CidrBlock: 10.1.0.0/16
CidrBlockState:
State: associated
DhcpOptionsId: dopt-d36cc9b9
InstanceTenancy: default
IsDefault: false
OwnerId: '123456789012'
State: available
Tags:
- Key: Manager
Value: crossplane
- Key: Name
Value: mgmt-vpc
- Key: crossplane-name
Value: mgmt-vpc
- Key: crossplane-kind
Value: vpc.ec2.aws.crossplane.io
- Key: crossplane-providerconfig
Value: default
Awesome, we see that there is a new VPC configured via our Cluster API manifest. As I wrote at the beginning, one problem which is seen with a typical Terraform usage is non-event based approach. So if our VPC is created by an ordinary Terraform module and it will be deleted, it will not be recreated automatically. Crossplane CRD has an event-loop which monitors the state between the local configuration and the remote setup. So I assume that if we delete the VPC (accidently) we should get it back (almost) immediately. Let’s check this.
$ kubectl get vpc -o wide --watch
NAME READY SYNCED ID CIDR AGE
mgmt-vpc True True vpc-0ec630f4ffb2d3d03 10.1.0.0/16 13s
mgmt-vpc True True vpc-061a6702eb5130c35 10.1.0.0/16 23s
$ aws ec2 delete-vpc --vpc-id vpc-0ec630f4ffb2d3d03 --region eu-central-1
There was a fresh VPC vpc-0ec630f4ffb2d3d03 which was deleted and after a few seconds the new vpc-061a6702eb5130c35 was created. Hurray! We reached our goals
- the resources were created immediately and we did not have to wait for downloading any external dependencies
- the resources were recreated immediately after it was deleted
That was nice example but someone may ask:
Where is the state located, what if I lose my Kubernetes cluster?
What happens if you lose your Kubernetes cluster
Terraform may keep its state locally or in e.g. AWS S3 but Crossplane does not store it anywhere. The YAML manifest is the form of state, so if we lose our Kubernetes cluster we will lose the event-based approach. However, once the cluster is online again we should get the state too, particularly if we use the State Importing feature. All we have to do is add a special annotation to the previously defined resource and apply it. For our AWS VPC and Security Groups it should be as follows:
---
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: VPC
metadata:
name: mgmt-vpc
annotations:
crossplane.io/external-name: <vpc-id>
...
---
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: SecurityGroup
metadata:
name: db-mysql-access
annotations:
crossplane.io/external-name: <sg-id>
...
But how do we know what kind of annotations to use? It’s all in the CRD’s code. For the VPC we have to take a look in the
package/crds/ec2.aws.crossplane.io_vpcs.yaml and for the Security Groups into the package/crds/ec2.aws.crossplane.io_securitygroups.yaml where the content looks like this:
- jsonPath: .metadata.annotations.crossplane\.io/external-name
name: ID
type: string
It is unfortunately not standardized because for some components we should use an external-name instead of id. After the code, with the new annotation, we wrote before the cluster’s outage is applied we will import the current state of our AWS components in the cluster.
What if we want to create a more complex setup?
That part was pretty straightforward because we used a configuration package which defined a VPC. But what if we want to create a complex setup like VPC with Security Groups or maybe create some AWS RDS instance with Security Groups as well. Plus we would like to not stress users and give them the possibility to define only a few parameters of the resource instead of all. That is possible by so-called Crossplane Composition - but this will be a part of another article where we will compare it with the Terragrunt DRY approach.
For fans of Terraform (including me), there is good information that this tool has also the possibility to operate on the resources via Kubernetes Operators. The project hashicorp/terraform-k8s gives that kind of possibility but it is in the alpha testing state, so be careful when you want to use it in production.
A handy summary of Terraform and Crossplane
I think that we may go into some summary and create a user friendly table. I do not want to put any Pros and Cons because it is very relative - for some my Pros will be Cons and vice versa.
Terraform:
- It's quite mature and with lot of community support in GitHub
- It gives you the chance to provision a lot of Cloud providers and applications
- HCL is a pretty straightforward language but sometimes the syntax lacks clarity and might be problematic to use
- It's not possible to act based on events after the infrastructure is created
- All you need to start using it is to install the Terraform application and write the code
- With Terragrunt is acquires the ability to have a DRY code structure
- The knowledge (state) about the infrastructure may be kept in the AWS S3
Crossplane:
- It's very young but continuously growing
- It currently covers only the main services for each Cloud provider
- It is possible to extend the application by new Cloud resources (Go language knowledge needed)
- It offers the possibility to provision major Cloud providers only
- It needs a Kubernetes cluster to operate on the infrastructure
- There is no need to learn a new programming language, as the configuration is based on the YAML syntax
- The state is kept in the current Kubernetes cluster only
And there you have it. I hope that you have found the above examples and summary useful.
Looking for an experienced team to bring your digital product to life?
Get in touch for a free consultation on hello@10Clouds.com. Our friendly team will get back to you within one working day!
