Terraform vs Crossplane - How to provide IaC with the environment state observability

Circles illustrationCircles illustration

Infrastructure as Code (IaC) is the managing and provisioning of infrastructure through code instead of through manual processes. It enables the automated and repeatable creation and maintenance of all virtualized infrastructures. Both Terraform and Crossplane support multiple providers and are great for operating cross-cloud. In this blog post, I wanted to give you a brief overview of both and to give you some insight into how we can keep Infrastructure as Code as clean as possible.

Terraform

Terraform is a tool for building, changing, and versioning infrastructure in a quick and efficient manner. It can manage existing and popular service providers and custom in-house solutions.

I’d like to begin by stating that I’m a big fan of Terraform with its overlay in the form of Terragrunt. To me, they’re absolutely essential when it comes to creating Cloud components. Of course, there are also great tools such as AWS Cloud Formation or Azure Resource Manager but the great thing about Terraform is that you can manage almost all Cloud Providers via one tool. I started using it when it was in its infancy and throughout my journey. There were good days and bad days - just like in a typical relationship.

Even though the tool is very good I found some cons that may overshadow its pros. One of them is a lack of so-called environment state observability, which in my scenario would be something like:

If the created Cloud component suddenly and not intentionally disappears, please recreate it immediately.

The second is the very lengthy command execution - when I run terragrunt apply and there is no local cache, sometimes I have to wait a few minutes to get the yes | no prompt. Even if my module is very small, Terraform needs to connect to external servers and get the dependencies, handle them, test the code, plan it and so on. While doing this once a day is acceptable, but when you are testing something and you need to delete the infrastructure’s components and apply them again and again, it is quite irritating.

Finding a new way to set the Cloud Infrastructure

Those two cons were the trigger to finding a new way to set the Cloud Infrastructure. To handle events I could create some event based actions which could be triggered by the change in the infrastructure. The event could then perform some action such as recreating the part of an invalid cloud component. This could be done via AWS CloudTrail or Lambda but as we see, it adds additional effort and complexity into our environment.

To get rid of long execution time we may write smaller modules or choose not to delete the cached data in the .terragrunt-cache. Of course it may help but when we perform changes in our code, we want to test it on the remote Cloud infrastructure and waiting N-times multiplied by e.g. 60s gives us a nice couple of minutes when we are taking a look at the Terraform log. That is a little bit of waste that we want to avoid. So maybe there is another way to create a Cloud infrastructure and have an event-based and less execution time approach.

Using an API to configure the Cloud infrastructure

I came across an interesting presentation in which the so-called Kubernetes Cluster API as well as declare Configuration as Data were mentioned. In short, if you do declare your Kubernetes configuration in YAML syntax, why not use its API to configure the Cloud infrastructure and declare it via YAML too? Wow! That is something new which is worth reading and trying… but it reveals some downsides - you must have a Kubernetes up-and-running. So for all of those scenarios when we do not use that kind of service and even if we want to we do not have knowledge about it - that may be a problem in implementing such an approach. Let’s skip it for now and assume that we use Kubernetes in our infrastructure and using the Cluster API for provisioning an infrastructure will be possible. Now it is time to find out which tools will make it happen.

As always we may find some Cloud Provider specific tools like Kubernetes Cluster API Provider AWS, AWS Controllers, GCP Config Connector or Azure Service Operator. But such services are good for one Cloud provider which is fine, but we have to use separate logic in usage for AWS or Azure. It would be great to have something like Terraform - a Swiss Army Knife that could manage many providers at one time. I found that the Crossplane offers these options, at least for the main providers like AWS, Azure, GCP and Alibaba. So that will give us the possibility to cover such scenario like:

Run our startup in AWS and ...oh! Wait, our Business decided to move it immediately to Azure - apply it then!

The above scenario is quite complicated for those who use Terraform mainly for AWS and do not have the code for Azure. In this situation they have to write modules, run tests and so on. It is possible but it takes some time. With declarative YAML and Cloud Operator it should be much easier and faster. Let’s test the Crossplane with AWS and the local Kubernetes cluster based on kind.

Crossplane

Testing Crossplane with AWS, Kubernetes and kind

First we have to have an AWS IAM User which credentials we have to provide into Crossplane’s configuration. This step needs to be done before we run the Crossplane.

$ aws iam create-user --user-name crossplane-admin

{
    "User": {
        "Path": "/",
        "UserName": "crossplane-admin",
        "UserId": "AIDA3N32HB6CP7WBGF7TM",
        "Arn": "arn:aws:iam::123456789012:user/crossplane-admin",
        "CreateDate": "2021-04-09T11:20:31+00:00"
    }
}

$ aws iam attach-user-policy --user-name crossplane-admin --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

$ aws iam create-access-key --user-name crossplane-admin

{
    "AccessKey": {
        "UserName": "crossplane-adminx",
        "AccessKeyId": "AKIA<secret>",
        "Status": "Active",
        "SecretAccessKey": "<secret>",
        "CreateDate": "2021-04-09T11:30:43+00:00"
    }
}

After above we have a crossplane-admin user and AWS Access Keys which will be used in the Crossplane’s configuration. Now we may jump into our local Kubernetes server and install the Crossplane.

  • Create valid cluster with v1.19.7 kind Docker image
$ kind create cluster --image kindest/node:v1.19.7 --wait 5m
...
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind
  • Install Crossplane CLI
$ curl -sL https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh | sh
  • Install Crossplane
$ kubectl create namespace crossplane-system

$ kubectl get namespaces
NAME                 STATUS   AGE
crossplane-system    Active   58s
...

$ helm repo add crossplane-stable https://charts.crossplane.io/stable

$ helm repo update

$ helm install crossplane --namespace crossplane-system crossplane-stable/crossplane --version 1.1.1

$ helm list -n crossplane-system

NAME          NAMESPACE            REVISION    UPDATED                                  STATUS      CHART               APP VERSION
crossplane    crossplane-system    1           2021-04-09 13:57:42.518737 +0200 CEST    deployed    crossplane-1.1.1    1.1.1

$ kubectl get all -n crossplane-system
NAME                                           READY   STATUS    RESTARTS   AGE
pod/crossplane-6c9c8bd9c7-dscb9                1/1     Running   0          40s
pod/crossplane-rbac-manager-5c8fff46f8-qj5l4   1/1     Running   0          40s

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/crossplane                1/1     1            1           40s
deployment.apps/crossplane-rbac-manager   1/1     1            1           40s

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/crossplane-6c9c8bd9c7                1         1         1       40s
replicaset.apps/crossplane-rbac-manager-5c8fff46f8   1         1         1       40s
  • Install Configuration Package (v1.1.1)
$ kubectl crossplane install configuration registry.upbound.io/xp/getting-started-with-aws-with-vpc:v1.1.1

$ kubectl get configuration
NAME                                   INSTALLED   HEALTHY   PACKAGE                                                           AGE
xp-getting-started-with-aws-with-vpc   True        False     registry.upbound.io/xp/getting-started-with-aws-with-vpc:v1.1.1   84s
  • Configure AWS Account data and create the secret
$ AWS_PROFILE=default && echo -e "[default]\naws_access_key_id = $(aws configure get aws_access_key_id --profile $AWS_PROFILE)\naws_secret_access_key = $(aws configure get aws_secret_access_key --profile $AWS_PROFILE)" > creds.conf

$ kubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./creds.conf
  • Configure the Provider
$ cat ./provider.yaml

---
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-creds
      key: creds

$ kubectl apply -f ./provider.yaml

$ kubectl get providerconfigs -o wide
NAME      AGE   SECRET-NAME
default   19s

Provisioning the AWS infrastructure

After completing the above steps, we need to provision the AWS infrastructure! Now it is time to provision the VPC with Security Group for the MySQL access. To do this we may use an available example configuration such as the crossplane/provider-aws repository but unfortunately there is no such example for the VPC so we have to handle it ourselves. That is good training to get familiar with Crossplane’s Custom Resource Definitions (CRD). To get into, we have to find out how to define our own YAML for the VPC. To do this we must go to package/crds/ec2.aws.crossplane.io_vpcs.yaml file and look closer into it.

We see that the spec.group is equal ec2.aws.crossplane.io and spec.versions.name is v1beta1. For metadata we may put typical fields like name and labels. But where to define VPC’s parameters like cidr or even tags?

This is defined in the section: spec.versions.schema.openAPIV3Schema.properties.spec.properties.forProviders field. We may see the keys like cidrBlock, enableDnsHostNames, enableDnsSupport or tags. This is all we need to create our YAML for the VPC.

---
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: VPC
metadata:
  name: mgmt-vpc
  labels:
    provider: aws
    vpc: “mgmt-vpc”
spec:
  forProvider:
    cidrBlock: “10.1.0.0/16”
    enableDnsHostNames: true
    enableDnsSupport: true
    instanceTenancy: default
    region: eu-central-1
    tags:
      -
        key: Name
        value: mgmt-vpc
      -
        key: Manager
        value: crossplane
  providerConfigRef:
    name: default

The value providerConfigRef contains the reference to the ProviderConfig. In the same way we have to configure our SecurityGroup resource and the result should be like below:

---
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: SecurityGroup
metadata:
 name: db-mysql-access
spec:
 forProvider:
   region: eu-central-1
   vpcIdSelector:
     matchLabels:
       vpc: mgmt-vpc
   groupName: db-mysql-access
   description: Allow access to MySQL
   ingress:
     -
       fromPort: 3306
       toPort: 3306
       ipProtocol: tcp
       ipRanges:
         -
           cidrIp: "10.1.10.0/0"
           description: Production

Please note the parameter spec.forProvider.vpcIdSelector.matchLabels that has a value vpc: mgmt-vpc It is needed because we want to create our Security Group after the VPC.

The above configuration should be placed in the vpc_sg.yaml file and executed by

$ kubectl apply -f vpc_sg.yaml

Let’s check the result!

$ kubectl get vpc -o wide
NAME       READY   SYNCED   ID                      CIDR          AGE
mgmt-vpc   True    True     vpc-0ec630f4ffb2d3d03   10.1.0.0/16   11s

$ kubectl get securitygroup -o wide
NAME              READY   SYNCED   ID                     VPC                     AGE
db-mysql-access   False   True     sg-011c1557dd5278c97   vpc-0ec630f4ffb2d3d03   20m

$ kubectl describe vpc
Name:         mgmt-vpc
Namespace:
Labels:       provider=aws
              vpc=mgmt-vpc
Annotations:  crossplane.io/external-name: vpc-0ec630f4ffb2d3d03
...
Spec:
  For Provider:
    Cidr Block:             10.1.0.0/16
    Enable Dns Host Names:  true
    Enable Dns Support:     true
    Instance Tenancy:       default
    Region:                 eu-central-1
    Tags:
      Key:    Manager
      Value:  crossplane
      Key:    Name
      Value:  mgmt-vpc
      Key:    crossplane-kind
   Value:  vpc.ec2.aws.crossplane.io
      Key:    crossplane-name
      Value:  mgmt-vpc
      Key:    crossplane-providerconfig
      Value:  default
...

$ kubectl describe securitygroup
Name:         db-mysql-access
...
Spec:
  For Provider:
    Description:  Allow access to MySQL
    Group Name:   db-mysql-access
    Ingress:
      From Port:    3306
      Ip Protocol:  tcp
      Ip Ranges:
        Cidr Ip:      10.1.10.0/0
        Description:  Production
      To Port:        3306
    Region:           eu-central-1
    Vpc Id:           vpc-0ec630f4ffb2d3d03
    Vpc Id Ref:
      Name:  mgmt-vpc
    Vpc Id Selector:
      Match Labels:
        Vpc:  mgmt-vpc


$ aws ec2 describe-vpcs --region eu-central-1 --output yaml

Vpcs:
- CidrBlock: 10.1.0.0/16
  CidrBlockAssociationSet:
  - AssociationId: vpc-cidr-assoc-08b287574b16e1243
    CidrBlock: 10.1.0.0/16
    CidrBlockState:
      State: associated
  DhcpOptionsId: dopt-d36cc9b9
  InstanceTenancy: default
  IsDefault: false
  OwnerId: '123456789012'
  State: available
  Tags:
  - Key: Manager
    Value: crossplane
  - Key: Name
    Value: mgmt-vpc
  - Key: crossplane-name
    Value: mgmt-vpc
  - Key: crossplane-kind
    Value: vpc.ec2.aws.crossplane.io
  - Key: crossplane-providerconfig
    Value: default

Awesome, we see that there is a new VPC configured via our Cluster API manifest. As I wrote at the beginning, one problem which is seen with a typical Terraform usage is non-event based approach. So if our VPC is created by an ordinary Terraform module and it will be deleted, it will not be recreated automatically. Crossplane CRD has an event-loop which monitors the state between the local configuration and the remote setup. So I assume that if we delete the VPC (accidently) we should get it back (almost) immediately. Let’s check this.

$ kubectl get vpc -o wide --watch
NAME       READY   SYNCED   ID                      CIDR          AGE
mgmt-vpc   True    True     vpc-0ec630f4ffb2d3d03   10.1.0.0/16   13s
mgmt-vpc   True    True     vpc-061a6702eb5130c35   10.1.0.0/16   23s

$ aws ec2 delete-vpc --vpc-id vpc-0ec630f4ffb2d3d03 --region eu-central-1

There was a fresh VPC vpc-0ec630f4ffb2d3d03 which was deleted and after a few seconds the new vpc-061a6702eb5130c35 was created. Hurray! We reached our goals

  • the resources were created immediately and we did not have to wait for downloading any external dependencies
  • the resources were recreated immediately after it was deleted


That was nice example but someone may ask:

Where is the state located, what if I lose my Kubernetes cluster?

What happens if you lose your Kubernetes cluster

Terraform may keep its state locally or in e.g. AWS S3 but Crossplane does not store it anywhere. The YAML manifest is the form of state, so if we lose our Kubernetes cluster we will lose the event-based approach. However, once the cluster is online again we should get the state too, particularly if we use the State Importing feature. All we have to do is add a special annotation to the previously defined resource and apply it. For our AWS VPC and Security Groups it should be as follows:

---
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: VPC
metadata:
  name: mgmt-vpc
  annotations:
    crossplane.io/external-name: <vpc-id>
...
---
apiVersion: ec2.aws.crossplane.io/v1beta1
kind: SecurityGroup
metadata:
  name: db-mysql-access
  annotations:
    crossplane.io/external-name: <sg-id>
...

But how do we know what kind of annotations to use? It’s all in the CRD’s code. For the VPC we have to take a look in the
package/crds/ec2.aws.crossplane.io_vpcs.yaml and for the Security Groups into the package/crds/ec2.aws.crossplane.io_securitygroups.yaml where the content looks like this:

    - jsonPath: .metadata.annotations.crossplane\.io/external-name
      name: ID
      type: string

It is unfortunately not standardized because for some components we should use an external-name instead of id. After the code, with the new annotation, we wrote before the cluster’s outage is applied we will import the current state of our AWS components in the cluster.

What if we want to create a more complex setup?

That part was pretty straightforward because we used a configuration package which defined a VPC. But what if we want to create a complex setup like VPC with Security Groups or maybe create some AWS RDS instance with Security Groups as well. Plus we would like to not stress users and give them the possibility to define only a few parameters of the resource instead of all. That is possible by so-called Crossplane Composition - but this will be a part of another article where we will compare it with the Terragrunt DRY approach.

For fans of Terraform (including me), there is good information that this tool has also the possibility to operate on the resources via Kubernetes Operators. The project hashicorp/terraform-k8s gives that kind of possibility but it is in the alpha testing state, so be careful when you want to use it in production.

A handy summary of Terraform and Crossplane

I think that we may go into some summary and create a user friendly table. I do not want to put any Pros and Cons because it is very relative - for some my Pros will be Cons and vice versa.

Terraform:

  • It's quite mature and with lot of community support in GitHub
  • It gives you the chance to provision a lot of Cloud providers and applications
  • HCL is a pretty straightforward language but sometimes the syntax lacks clarity and might be problematic to use
  • It's not possible to act based on events after the infrastructure is created
  • All you need to start using it is to install the Terraform application and write the code
  • With Terragrunt is acquires the ability to have a DRY code structure
  • The knowledge (state) about the infrastructure may be kept in the AWS S3

5 technology trends likely to continue booming in a post-COVID world

Crossplane:

  • It's very young but continuously growing
  • It currently covers only the main services for each Cloud provider
  • It is possible to extend the application by new Cloud resources (Go language knowledge needed)
  • It offers the possibility to provision major Cloud providers only
  • It needs a Kubernetes cluster to operate on the infrastructure
  • There is no need to learn a new programming language, as the configuration is based on the YAML syntax
  • The state is kept in the current Kubernetes cluster only

And there you have it. I hope that you have found the above examples and summary useful.

Have any DevOps queries for us? Want to discuss a piece of work?

It would be great to hear from you. Just drop us a line on hello@10clouds.com

You may also like these posts

Start a project with 10Clouds

Hire us