Update Route53 on Instance Boot

2022-08-24

Recently, I found myself wanting to host a demonstration of OpenCTI on a single VM. As the system requirements for it are significant, and this is mostly a toy deployment, I decided to save costs (and accept lower availability) by using an AWS Spot Request to host the entire instance. I wanted to still be available from a dedicated domain name, but AWS Spot Requests will receive a new public IP when recycled, which meant having to update the associated resource record in Route53. I’ll discuss how I use the AWS CLI tool to create a systemd start-up script to update the DNS record with the correct IP address when the VM boots up.

Environment Summary

The infrastructure used for this project is a workload that is served via an AWS Spot Instance request. The benefit to doing this is that compute costs can be reduced anywhere from 40%-80%, depending upon the instance type, marketplace demand, and the amount of availability one is willing to sacrifice. Spot Instances will run until AWS needs the assigned compute resources for a higher-paying workload, at which point the spot instance will get shut down, through some mechanism that is configurable. In my case, I set the interruption behavior to be “Stop”, which will power off the instance, but keep it around, and then power it back on when free capacity becomes available in the future. Since I don’t mind an occasional outage for what is ultimately a personal experiment, I don’t mind so much. I still assign an IAM execution role to this instance, so that it can perform some additional AWS API actions, and access AWS-hosted resources dedicated to the project.

The challenge introduced by this is that, when the instance restarts, it will be granted a new AWS Elastic IP (public IP) from the pool, which means updating DNS records whenever it gets a new IP address (on boot up).

Granting IAM Permissions

Like any other AWS service, the Route53 service can be interacted with via API using the AWS CLI tool. In order to be able to modify records, the IAM role that will be attempting to perform those change operations needs to be granted permission to do so. Assuming that the domain name you wish to use is already configured to use Route53 for its DNS records, you will first need to get the "Hosted Zone ID" for your domain from the “Route53” service panel.

Go to “Route53” in the AWS Console, and then click the navigation link for “Hosted zones”. This will present a list of the Domain names that are being hosted by AWS, and in each row there’s an associated Hosted zone ID. This is the unique identifier for your domain name, and is necessary both to tell IAM what domain to grant update permissions on, as well as to tell the AWS CLI tool what domain name an update will apply to. My hosted zones are either 20 or 13 character identifiers, which appear to be randomly generated, except that the first character is always “Z”, and the identifier is always upper-cased. Find this Zone ID, and save it somewhere.

In here, as well, click on the domain name that you want to use for creating the sub-domain for the site hosted by the spot instance. This will bring up the list of Records. If the sub-domain you wish to use to access the site is already present, you shouldn’t need to make any changes, as long as it is an “A” type record. Otherwise, create a new record of type “A”, and give it a unique sub-domain, and then populate the content either with the current IP of the VM, or a random IP such as 0.0.0.0. Write down the full sub-domain name, including the parent domain name of its hosted zone (if the parent domain is yoursite.com, then the full sub-domain might be myserver.yoursite.com).

Next, navigate to the IAM system, and find the execution role that the VM is using (or create a new execution role for the VM, and assign it to the VM in the EC2 service configuration area, and then return to IAM after you’ve completed that).

In IAM, under Roles, find the role matching the execution role assigned to your Spot Instance. Then, “Add Permissions”, and select “Inline policy”, to create a new inline policy that will extend the role’s permissions. Open up the JSON editor, and put the following into it, replacing the text <YOUR HOSTED ZONE ID> with the Z-value you saved in the prior step from the Route53 section.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "route53:ChangeResourceRecordSets",
            "Resource": "arn:aws:route53:::hostedzone/<YOUR HOSTED ZONE ID>"
        }
    ]
}

Save the new inline policy by giving it a short name (and description, if desired). Once saved, it will immediately grant this permission to the execution role on the VM going forward, without needing a reboot.

Testing the Update Works

For the sake of this example, let’s assume that the following Route53 information was collected/configured in the prior step:

Hosted Zone ID: Z0123456789ABCDEFGHI
Domain Name: yoursite.com
Sub-domain: myserver.yoursite.com

Log into the Linux VM, and if it isn’t already installed, make sure the AWS CLI tools are already installed on the VM. Once in, use the following command to get the current IP address:

curl "http://ifconfig.me/" 2> /dev/null

It should report an IP address, and nothing else:

52.1.101.123

Record this, and then use a text editor to create the following file in /tmp/test-r53.json. Make sure to replace the sub-domain and Hosted Zone ID with the correct ones from your Route53 service, or this will not work:

{
    "Comment": "Update record to reflect new IP address for myserver.yoursite.com",
    "Changes": [
        {
            "Action": "UPSERT",
            "ResourceRecordSet": {
                "Name": "myserver.yoursite.com",
                "Type": "A",
                "TTL": 60,
                "ResourceRecords": [
                    {
                        "Value": "52.1.101.123"
                    }
                ]
            }
        }
    ]
}

Finally, save and exit from the text editor, and then run the following command using the aws CLI tool to update the DNS resource record in Route53. Note that the Hosted Zone ID corresponding to your parent domain name needs to be used here:

aws route53 change-resource-record-sets --hosted-zone-id 'Z0123456789ABCDEFGHI' --change-batch "file:///tmp/test-r53.json"

If successful, some more JSON similar to the following should be displayed. Note the "PENDING" Status:

{
    "ChangeInfo": {
        "Id": "/change/xxx123asd123",
        "Status": "PENDING",
        "SubmittedAt": "2021-04-07T04:00:18.348000+00:00",
        "Comment": "Update record to reflect new IP address for myserver.yoursite.com"
    }
}

If the above was successful, then you’ve confirmed that the IAM permissions are correct, and the appropriate tools are installed in your VM to use the AWS API for this.

A good practice, before moving on, is to delete the temporary JSON file that was created:

rm -f /tmp/test-r53.json

Setting Up the Service in systemd

Create a new file named /opt/update-route53.sh and put the following into it:

#!/bin/bash
myip=`curl http://ifconfig.me/ 2> /dev/null`
tfile=`tempfile -s .json -p r53` || exit

cat << ENDJSON > "${tfile}"
{
    "Comment": "Update record to reflect new IP address for a system ",
    "Changes": [
        {
            "Action": "UPSERT",
            "ResourceRecordSet": {
                "Name": "${CHANGE_RECORD}",
                "Type": "A",
                "TTL": 60,
                "ResourceRecords": [
                    {
                        "Value": "${myip}"
                    }
                ]
            }
        }
    ]
}
ENDJSON

aws route53 change-resource-record-sets --hosted-zone-id "${HOSTED_ZONE}" --change-batch "file://${tfile}" > /dev/null

rm -f "${tfile}"

The above script will use the public http://ifconfig.me service to identify the external IP of the VM. Alternately, if we want to use AWS-internal services only, the following endpoint may be used instead: http://169.254.169.254/latest/meta-data/public-ipv4.

Next, create a new “defaults” file that will contain the ${HOSTED_ZONE} and ${CHANGE_RECORD} variables used by this script in /etc/defaults/update-route53. Update the two example values below with the ones collected from Route53 at the beginning of this post:

#!/bin/bash
HOSTED_ZONE='Z0123456789ABCDEFGHI'
CHANGE_RECORD="myserver.yoursite.com"

Finally, create a new file /etc/systemd/system/update-route53.service, with the following content:

[Unit]
Description=Service to update Route53
After=network.target

[Service]
EnvironmentFile=/etc/default/update-route53
Type=oneshot
ExecStart=/bin/bash /opt/update-route53.sh
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

In the above, the Type=oneshot property for the [Service] section tells systemd that this job is expected to run once and then exit. The EnvironmentFile variable tells systemd to use the /etc/defaults/update-route53 file to pre-populate environment variables, which is a common convention used for providing an easy configuration for the service unit.

When a systemd service is added or updated, you’ll need to run the following command to have systemd refresh its service database, so run this next:

sudo sytemctl daemon-reload

Finally, to enable the service on boot, as well as execute it immediately (updating the IP):

sudo sytemctl enable --now update-route53.service

Conclusion

This should help ensure better dynamic availability of a spot instance, by providing a dedicated domain name to reach it that is populated automatically with the public IP address. My recommendation would be to configure the Interruption Behavior to “Stop”, which will stop the running VM, but keep around the EBS volume, reusing it when capacity becomes available again. Alternatively, if “Terminate” is preferred, then additional work will need to be made to create an AMI from the volume (after performing the above steps), in order to preserve these changes across reboots.

Author: Coleman Kane
Permanent Link: https://blog.malware.re/2022/08/24/Update-Route53-on-Instance-Boot/index.html