In this article I am penning down what I learned about "automating Ansible Tower install" - using a CloudFormation template.

Remember, it's not "automating with Ansible" instead it's "how to deploy Ansible Tower with a CloudFormation Template". I have used RHEL7.4 for this template.

First things first. A very brief introduction on CloudFormation template

Few words about CloudFormation

AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion

It is cool, really! You can write your templates in JSON or YAML. And, so quite honestly I haven't had to deal with YAML, but nothing to it except it's very picky about how you indent your code :D :D. But we will catch malformed template by validating the template. It's super easy.

On a side note, I use Atom to write my templates, but you are free to use an editor of your choice. I also am a huge fan of VSCode.

And a bit about template syntax

A CloudFormation template has,

• Description - describes a template
• Metadata - has information about ParameterGroups, Parameters & Parameter Labels
• Mappings - Region maps, Region AMI maps etc..
• Conditions - Some boolean conditions (optional)
• Parameters - parameters with default values (if any).
• Resources - Definition of AWS resources to be spun up by the template
• Outputs - Output from newly spun resources

AWS has decent tutorials / listing of templates to get familiarized with the CloudFormation syntax. YAML or JSON, pick your poison :D

How I write my template

This is not mandatory, but I am a developer first and so I like to use appropriate prefix to identify parameters, resources etc...I will identify parameters with a prefix "p" and resources with "r". So here is how it goes. Very simple example of a parameter and a resource.

#  parameter definition
pAuthorName:
    Description: Who is the author?
    Type: String
    Default: 'ritesh patel'

# resource definition
rElasticIP:
    Type: AWS::EC2::EIP
    Properties:
      InstanceId: !Ref rEC2InstanceAnsibleTower
      Domain: vpc

Don't worry about the details of defining a resource. We will dissect resources in a few.

How to validate a template?

I use AWS CLI to validate my templates. It's super easy with only one requirement. You must install and configure AWS CLI.

aws cloudformation validate-template --profile $PROFILE --region $REGION --template-body $TEMPLATE

$PROFILE is your AWS credentials, $REGION is your AWS region and $TEMPLATE is the template file you wish to validate. That's it.

Now let's start building our template for Ansible Tower.

Important parameters for Ansible Tower instance

In this section I will mainly focus on important parameters (a.k.a must haves) for Ansible Tower.

Network Parameters

These are the parameters for VPC, subnets & availability zones.

- Label:
    default: Network Configurations
    Parameters:
    - pVPC
    - pSubnet
    - pAvailabilityZone

Server Configurations

These parameters will define instance type, AMI, key pair, instance profile and so forth.

- Label:
    default: Server Security Configurations
    Parameters:
    - pInstanceType
    - pAMIId
    - pKeyPair
    - pIAMInstanceProfile

Rabbit MQ Configurations

Ansible Tower uses Rabbit MQ for clustering and therefore these parameters will handle Rabbit MQ requirements.

- Label:
    default: RabbitMQ Configurations
    Parameters:
    - pRabbitMQPort
    - pRabbitMQVHost
    - pRabbitMQUserName
    - pRabbitMQPassword
    - pRabbitMQCookie

Ansible Tower Credentials

Ansible Tower has a Web UI. We must pass credentials for Ansible Web UI during the install and therefore these parameters will be used to set Tower credentials.

- Label:
    default: Ansible Tower Credentials
    Parameters:
    - pAnsibleTowerPassword

There are few more parameters for which please refer to the final template in my Github repository. But for now this should do it.

Does Ansible Tower use a database? You bet. I am using PostgreSql RDS instance with Ansible Tower and so you guessed it right! Next up we will look at the parameters required for RDS instance. We are almost done with the parameters so just stay with me.

RDS Parameters

Quickly, what is RDS? It is a Relational Database Service offered by AWS. Here is an excerpt straight from AWS.

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easier to set up, operate, and scale a relational database in the cloud. It provides cost-efficient, resizable capacity for an industry-standard relational database and manages common database administration tasks.

Below are the parameters we will use for our PostgreSql instance.

- Label:
    default: Ansible Tower Credentials
    Parameters:
    - pRDSAvailabilityZone
    - pRDSInstanceClass
    - pRDSMasterUsername
    - pRDSMasterUserPassword
    - pRDSDatabaseName
    - pRDSPort
    - pRDSSubnetGroupName

Phew! Quite a few parameters, but we are done with the parameters. Next, let's see how to assign a default value to one of the parameters. All parameters follow similar syntax for assigning default values.

A Sample Parameter Definition

Let's look at sample RDS parameter with a default value.

Parameters:
    pRDSDatabaseName:
        Description: Database name
        Type: String
        Default: towerdb

Notice "Default: towerdb". That's it. Simple, right? Review the final template for additional parameters.

UserData and Init Scripts

Yeah so CloudFormation is great, I get it. It allows me to spin up an instance or a cloud resource of my choice. But what if I have specific set of scripts to be executed when booting up a new instance? Meet UserData. Yes, any custom scripts you wish to execute upon instance startup must go under UserData.

And so what can go under UserData? I say anything that will rock your boat :D. Ok ok. In this template, I am installing Python, yum updates, running CloudInit scripts & setting up few wait conditions. (more on wait conditions later).

Sample scripts under UserData.

UserData:
    Fn::Base64: !Sub |
        #!/bin/sh
        export AWS_DEFAULT_REGION=us-east-1
        yum -y update aws-cfn-bootstrap

You get the gist!

ConfigSets

What are the config sets? Nothing but merely a list of tasks to execute via CloudFormation::Init. As the name suggests it's initialization. Yes, I said any initialization should go under UserData. That's correct. CloudFormation::Init gets called from UserData. Let's look at how this happens!

UserData:
    Fn::Base64: !Sub |
        /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --configsets bootstrap

And then under the same resource definition we define CloudFormation::Init like below.

Metadata:
    AWS::CloudFormation::Init:
        configsets:
            bootstrap:
                - install-ansible-engine
                - get-tower
                - create-inventory
                - replace-inventory
                - modify-hosts
                - install-tower

Pay attention to "configsets" defined under UserData and how configsets are executed via Init. In this case configsets tasks:

• will install ansible engine
• download a tower bundle
• create ansible inventory file
• replace inventory in the extracted bundle
• modifies host file to run the playbook
• install tower.

With me?

Before we install Ansible Tower we need a database. And so Ansible Tower install has a tight dependency with the database, right? So how exactly CloudFormation deals with resource dependencies?

Meet DependsOn.

DependsOn - Dependency Management in a Template

To install Ansible Tower we need database name, database endpoint and few additional information from the database instance. So we have a strict dependency on spinning up resources. In other words, template must not create Ansible Tower Instance until a Database Instance is in-place.

CloudFormation is very good at handling implicit dependencies, but in this case we will explicitly use "DependsOn" property to declare a dependency.

And so this is how it goes...

rAnsibleTower:
    DependsOn: rRDSInstance
    ...
    ...
    ...

rRDSInstance:
    Type: AWS::RDS::DBInstance
    Properties:
        AllocatedStorage: '100'
        BackupRetentionPeriod: '30'
        ...
        ...
        ...

Just like above you may use DependsOn for additional resource dependencies. But, what if you have a resource that depends on multiple dependencies? Simple, just provide a list of strings.

rAnsibleTower:
    DependsOn: [ 'rRDSInstance', 'rElasticIP' ]
    ...
    ...
    ...

With me?

Remember UserData & ConfigSets?

My template on GitHub has additional resources to support Ansible Tower install but, we will strictly focus on the tasks for installing and configuring Ansible Tower. Remember the configsets I mentioned in UserData? Let's go through them one-by-one.

install-ansible-engine

install-andible-engine:
    commands:
            0-install-pip:
              command: sudo easy_install pip
            1-sleep:
              command: sleep 10s
            2-install-ansible:
              command: sudo pip install ansible
            3-sleep:
              command: sleep 10s
            4-print-ansible-version:
              command: ansible --version
            5-pause-after-ansible-engine:
              command: sleep 10s

Self explanatory shell commands. YAML above will install ansible engine via pip, create a sample inventory file (/tmp/hosts) and pings localhost via ansible. A simple test to make sure ansible is able to make a connection to localhost.

get-tower

Let's download Ansible Tower tar file. We will download and extract tar ball in /tmp folder.

get-tower:
    commands:
        0-download-tower:
            command: sudo wget https://releases.ansible.com/ansible-tower/setup/ansible-tower-setup-latest.tar.gz
            cwd: '/tmp'
        1-extract-tower:
            command: tar xvzf ansible-tower-setup-latest.tar.gz
            cwd: '/tmp'
        2-pause-after-tower-extract:
            command: sleep 15s

Again, YAML above will download ansible tar ball, extract tar into temporary folder (/tmp) and pause for 15 seconds.

create-inventory

Ansible Tower install is done through a playbook. It comes bundled with an inventory file. But with any config file it doesn't have specifics about our install. And so, we will create a new inventory file which will replace the inventory file bundled with the tar ball.

In this inventory file we will specify,

• Ansible credentials
• RDS parameters - like a database name, port, database credentials, endpoint (RDS URL)
• RabbitMQ parameters - port, credentials etc..

Remember most of these parameters will take the values from RDS resource and template parameters.

And so the new inventory file.

create-inventory:
    files:
        /tmp/inventory:
            content: !Sub |
                [tower]
                localhost ansible_connection=local

                [database]

                [all:vars]
                ansible_become=true
                admin_password=${pAdminPassword}
                pg_host=${rRDSInstance.Endpoint.Address}
                pg_port=${pRDSPort}
                pg_database=${pDatabaseName}
                pg_username=${pDatabaseAdmin}
                pg_password=${pDatabasePassword}

                rabbitmq_port=${pRabbitMQPort}
                rabbitmq_vhost=${pRabbitMQVHost}
                rabbitmq_username=${pRabbitMQAdmin}
                rabbitmq_password=${pRabbitMQPassword}
                rabbitmq_cookie=cookiemonster
                rabbitmq_user_log_name=false
        mode: '000644'
        owner: 'root'
        group: root
    DependsOn: rRDSAnsibleTower

In YAML above notice "ansible_connection=local". By specifying "local" we are instructing ansible engine to run the playbook on a local host.

Another property to notice is, "ansible_become=true". This property will elevate privileges for the playbook execution. In other words, playbook will be executed as a "sudo". Rest should be self explanatory.

replace-inventory

Now, let's replace the inventory file from the tower bundle with the one we created above.

replace-inventory:
    commands:
        0-copy-inventory:
            command: cp /tmp/inventory /tmp/ansible-tower-setup*/inventory
    DependsOn: rRDSAnsibleTower

Very simple. We will copy our new inventory file in the directory where we extracted the tar file.

Notice "ansible-tower-setup*".When you extract the tar file, it will create a directory named "ansible-tower-setup-(version number)". Version number could be 3.1.2, 3.1.3 or some future version. So we will use a "wild-card" for version numbers. After all, we don't want to have a tight coupling with the versions by hardcoding the version number, right?

Now remember I said Ansible Tower install is done through a playbook? Well, then we need to modify the host file used by the ansible engine. We will add a "localhost" to our ansible host file.

modify-hosts

modify-hosts:
    files:
        /etc/ansible/hosts:
            content: !Sub |
                [localhost]
                127.0.0.1

And finally we install tower.

install-tower

install-tower:
    0-move-source-dir:
        command: cd /tmp/ansible-tower-setup*
    1-release-min-var-requirements:
        command: sed -i -e "s/10000000000/100000000/" /tmp/ansible-tower-setup*/roles/preflight/defaults/main.yml
    2-allow-sudo:
        command: sed -i -e "s/Defaults    requiretty/Defaults    \!requiretty/" /etc/sudoers
    3-pause-before-tower-install:
        command: sleep 120s
    4-install-tower:
        command: sh /tmp/ansible-tower-setup-*/setup.sh -e "nginx_disable_https=true"
DependsOn: rRDSAnsibleTower
services:
    sysvinit:
        nginx:
            enabled: true
            ensureRunning: true 

In the YAML above, we move to the install directory, release minimum of 10GB requirement (required for RHEL7), allow sudo in the sudoers, pause for 2 minutes and then install tower.

We also ensure, "nginx" service is enabled and running.

WaitCondition

So it's great how we can run custom scripts in UserData but how exactly do we know the scripts have finished execution? Ah, meet "WaitCondition"s. Very useful. WaitCondition signals notify the CloudFormation templates about stack completion.

So how exactly we implement a WaitCondition? Well, look at the very last line under UserData.

/opt/aws/bin/cfn-signal -e $? '${towerWaitHandle}'

Our script is calling cfn-signal to keep an eye on 'towerWaitHandle'. And here is what the 'towerWaitHandle' looks like.

  towerWaitHandle:
    Type: AWS::CloudFormation::WaitConditionHandle
  cfnWaitCondition:
    Type: AWS::CloudFormation::WaitCondition
    DependsOn: rAnsibleTower
    Properties:
      Handle: !Ref towerWaitHandle
      Timeout: '1200'

In the YAML above, we are creating a waitHandle and a waitCondition. waitHandle is referred by the waitCondition.

Now take a look at the waitCondition. It has a dependency on the resource "rAnsibleTower". In other words, UserData from "rAnsibleTower" will signal the waitCondition once the scripts have completed the execution.

WaitCondition also has a timeout of 1200 seconds. So, our CloudFormation template will wait at least 20 minutes to receive a signal from rAnsibleTower. Remember signal could be triggered sooner than 20 minutes in case of a successful execution.

If no signal received then CloudFormation stack will roll back indicating failure else the CloudFormation status is changed to "CREATE_COMPLETE".

Watch Progress

So how do I track the execution status of UserData scripts? Yes, you do want to keep an eye on the logs as it installs Ansible Tower. As soon as EC2Instance is available, ssh into this new instance. Change user to sudo and go to /var/log directory. You should see few logs under this directory. One log you want to pay attention to is cfn-init-cmd.log.

Run the command below at the terminal. And voila! you should be able to see the progress of UserData scripts.

tail -f cfn-init-cmd.log

Additionally, Ansible Tower install has it's own log, of course. Go to /tmp directory and find the folder where we extracted the tar file. You should see a setup.log file and if you tail this log file you should be able to track tower install as it happens :D

tail -f setup.log

That was a lot, I know. Hey, but it comes with a reward: You can have Ansible Tower up and running behind a load balancer within 30 minutes - give or take. Now that's a deal.

And finally here is the GitHub repo. Clone and enjoy!

Cheers!