Rapid MISP Deployment in AWS Serverless

The MISP Project is a popular cyber threat intel (CTI) database that has a very active user and development community. Among features available, one key one that has helped adoption a lot is the ease with which MISP administrators can synchronize their data set with others’ MISP deployments, as well as many other data sets served in the STIX structured interchange format standard. In an effort to adopt some more tools common in today’s CTI ecosystem, I set out to host MISP using some of the serverless features offered via AWS. It turned out to be significantly easier than I anticipated, so here is my story.

Resources

Here is a rough list of resources and technologies that I used to get this up and running. I’ll provide a key list here, but these are also sprinkled around the article in context:

Purpose of this Article

This article is intended to walk through the set up of MISP using some of the AWS serverless features that exist. The goal is to communicate a template that works well as a development base for setting up a personal-use instance using these serverless and scalable features. AWS has a number of additional services that help facilitate production-environment deployments to support high-availability as well as better security. This article mainly focuses on the basics to getting up and running with a functional application, which naturally means that in some cases a more secure approach, that is more complex to set up, would be recommended for applications that are expected to serve as production environments for multiple users or teams - particularly those which have varying levels of trust amongst one another.

Use this guide to help set up a test environment, but if you wish to take it further, please engage the help of experts in AWS security or do further security research for AWS application development and management.

Setting Up the Database

First thing, I wanted to set up a database in AWS for hosting the data. The AWS RDS Aurora Serverless product offers two really beneficial services that make hosting a database much easier. First, the Aurora RDS service offers an AWS-optimized database that is still MySQL compatible at the network protocol level. This allows for a managed database that can run a lot faster than the more traditional EC2 instance-backed database servers. Additionally, the “serverless” variant of the product is a relatively new addition that allows for the database servers (the compute parts) to be put to rest after an admin-defined period of inactivity. This means that, when the app isn’t making database requests, the database infrastructure itself will be using 0 CPU resources, yet will be woken up and scaled to a defined minimum a short period after the first attempt to connect to it (with a small latency penalty for that time).

To do this, log into the AWS console, and choose the Services menu. On that menu, look for the Database service group, and choose the RDS option. This should send you to the Amazon RDS console panel, where you will be able to create and manage databases. In this view, click the Databases option from the left side-bar menu. Once there, click the Create Database button. This displays the Create database wizard. The first option is to choose the creation method - we will use the Standard method.

RDS Standard Creation Method

Under Engine options, choose the following:

  • Engine type: Amazon Aurora
  • Edition: Amazon Aurora with MySQL compatibility
  • Capacity type: Serverless
  • Version: I picked the newest version (MySQL 5.7) and it works for me

RDS Engine Options

Next is the Settings section, which allows you to define a name for the infrastructure in AWS, as well as access credentials. The misp-docker project doesn’t mandate any specific credentials, but does rely upon password-based authentication for database access. You may either choose to enter a password in here that you manually created, or select the Auto generate a password option to have AWS create one for you. If you do auto-generate a password, make sure you wait for it to be shown to you after the database is created, or you’ll have to force-reset it after creation. I created a database cluster named misp-blog-db, and a username of mispadmin.

RDS DB Settings

The next panel is the Capacity settings. This is where you really get a window into what the “serverless” Aurora has to offer. Since this will be “serverless”, you won’t have a bunch of “database instances” to manage and keep track of (if you’ve managed larger databases in RDS, you’ll know what I’m talking about). Instead, you tell Aurora how many “capacity units” (CU) you want to use, at a minimum, as well as what upper limit you want, which will help you control costs under load. The interface also provides a helpful hint of how much RAM is available for each choice of capacity units. The defaults of 1-64 seemed a little risky to me, for an experiment, so I picked the following instead:

  • Minimum Aurora capacity unit: 2 (4GB RAM)
  • Maximum Aurora capacity unit: 8 (16GB RAM)

Before leaving this section, there’s one more set of options that are initially folded/hidden from view. Click the Additional scaling configuration text to expand some additional options. In this section, I left the Force scaling the capacity to the specified values when the timeout is reached feature disabled, as I don’t expect to need this in an experimental deployment. The Pause compute capacity after consecutive minutes of inactivity feature, however, I enabled. This allows you to set a timeout whereby the cluster will be scaled down to zero compute units until connection attempts are made again. This will help ensure that you will not be charged for compute during times where the MISP aplication is idling. Additionally, this has the added benefit of making it more cost effective to serve content at higher performance levels (more CU’s) when you are using the application. A new setting is revealed after this box is checked, which allows you to modify the length of time that needs to pass before the database is “paused“ (reduced to zero compute units). I set this to 15 minutes, so that I could ensure that it would stay running and performant during my usage sessions.

The next section is the Connectivity section. This section allows you to decide which VPC to deploy the RDS instance to (so, what network segment will be considered its “home”), as well as what Network Security Group (NSG) that will govern its accessibility. For simplicity, I selected the Default VPC, as well as default in the Subnet group drop-down. Under VPC security group I selected the Create new option, and gave it a name of misp-blog-db-sg to correspond with the cluster name I gave at the beginning. Do make sure that you record this security group name for later. Later on we will also create a security group for the ECS task that executes the MISP web application, and we will come back to edit the misp-blog-db-sg so that it is accesible from (and only from) the web application security group. This type of least privileged access is a good practice to use in AWS, as it helps minimize unauthorized access from any other insecure workloads you may have hosted in AWS.

Finally, a really helpful feature that is exposed after unfolding the Additional configuration heading at the bottom of this panel is the Data API feature. Enabling this allows you to make queries against this database from the AWS console - which can help you diagnose database issues and perform simple maintenance steps from the AWS console, without having to spin up an EC2 instance or expose the database to the internet.

RDS Connectivity Settings

Finally, there’s also a last panel named Additional configuration (which you shouldn’t confuse for the sub-section of the prior panel that uses the same name). This panel allows you to modify settings related to data encryption, backup, as well as give a name for the initial MySQL database that will be created inside the cluster. MISP doesn’t dictate any specific database name, so feel free to use any name that you wish here. I chose the name misp, which appears to be the requirement for the Docker image:

  • Initial database name: misp
  • DB cluster parameter group: left as default (default.aurora-mysql5.7)
  • Backup retention period: 1 day (but feel free to choose more retention time, if desired)
  • Copy tags to snapshots: leave enabled, as this may be necessary for recovery
  • Encryption Master key: Leave as (default) aws/rds, unless you wish and know how to manage keys manually
  • Enable deletion protection: I also like to leave this enabled, as it will interrupt you before deleting a database, giving you a chance to stop if you’ve mistakenly started that process

RDS Additional Settings

Once all your settings have been locked in and reviewed, click Create database at the bottom-right of the form - if you chose to auto-generate a password above, don’t forget to keep the tab open and wait for the View credential details button to show up at the top-right of the next screen. This gives you your only opportunity to copy & save the auto-generated password created by AWS RDS.

RDS Created

The database will take awhile to create, so it is safe to continue through with this how-to, and setting up the MISP application in its container while the DB creation happens in the background. Before leaving this, click on the new database’s name to view the details for the new RDS database instance. Inside of this will be an Endpoint name that is a hostname to use later on when telling MISP what host to connect to for database access (the MYSQL_HOST parameter).

The endpoint name will look something like this:

mispblog.cluster-<random characters>.us-east-2.amazonaws.com

Create the Docker Container Image

In order to use the Elastic Container Registry to maintain your container, you’ll first need to use docker on a system of yours to create the container from the GitHub Repo. If you have a slower Internet connection and want to reduce your wait time in moving large amounts of data into and out of AWS, I recommend spinning up a quick AMI Limux EC2 instance to do the command-line work below. These are nicely set up already with the AWS CLI utility, as well as a consistent Linux environment that includes the yum package manager.

Make sure git and docker and docker-compose are all installed. On a yum based system, you’d run:

sudo yum install -y docker docker-compose git

For Debian/Ubuntu and other apt systems:

sudo apt install docker-ce docker-compose git

For Manjaro/Arch and other pacman systems:

sudo pacman -S docker docker-compose git

It is also handy to make sure that your user account is allowed to make docker calls, by adding your account to the docker group. In any of the above systems, this would be:

sudo usermod -a -G docker "${USER}"

Next, clone the repository, and enter the directory:

git clone "https://github.com/ckane/misp-docker.git" && cd "misp-docker"

The next command creates the image in your local docker repository. This command will take a long time to execute, as it spins up the container and performs all of the application-side installation and configuration steps. Upon completion, your local docker repository will contain a docker image named misp which is ready to be pushed into the AWS Elastic Container Registry (ECR):

docker-compose -f docker-compose-nodb.yml build

Push Container Image to ECR

Next, we will use the AWS Elastic Container Registry to host the image for our containers in AWS. The ECR service is similar to services like DockerHub, but just private for your AWS deployment, and already integrated nicely with the AWS container services.

Log into the AWS console, and choose the Elastic Container Registry choice from the Services menu, which will be under the Containers service family section. Once there, you’ll be presented with the UI for the registry service. This allows you to create & manage the Docker container images that you have registered, which can be used to deploy into the Elastic Container Service (ECS) or Elastic Kubernetes Service (EKS).

At top-right, click the Create repository button, and you’ll be presented with a set of panels that allows you to configure the new repository that will store the MISP image. I used the following options:

  • Visibility settings: Private
  • Repository name: misp-blog
  • Tag immutability: Disabled
  • Scan on push: Disabled
  • KMS encryption: Disabled

ECR Repository Creation

Once complete, a row will show up in the summary screen, which displays your repository name, URI, and a number of other attributes about it. Selecting it in the list will allow you to view the commands necessary on your dev environment to push a new image to the repository.

ECR List

Below are the commands I used for Linux. Make sure that you have AWS CLI installed on the system you will be using for Docker, and that you’ve already used aws configure to establish a session to your account. These instructions were lifted from those provided by the View push commands:

Pass an AWS authentication token to your local Docker client. Note that the region us-east-2 will differ based upon what AWS region you’re hosting your application in. Additionally, note that the placeholder below <numeric account id> is supposed to be populated by your numeric AWS account id, visible in the user-profile drop down in the upper-right of the AWS GUI:

aws ecr get-login-password --region "us-east-2" | \
  docker login --username AWS --password-stdin "<numeric account id>.dkr.ecr.us-east-2.amazonaws.com"

You should see output that looks sort of like below, explaining success:

WARNING! Your password will be stored unencrypted in /home/username/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

Note that the AWS instructions don’t anticipate the use of docker-compose so you won’t be able to follow them exactly for this project. The step to build the container image was already performed above, yielding a container image named misp. Next, you’ll want to apply a tag to that image such that it can be pushed to your ECR under the new name. Similar to above, you’ll need to use your own numeric account id:

docker tag "misp:latest" "<numeric account id>.dkr.ecr.us-east-2.amazonaws.com/misp-blog:latest"

Finally, push it from your local dev system to the ECR with the following command:

docker push "<numeric account id>.dkr.ecr.us-east-2.amazonaws.com/misp-blog:latest"

This command will take some time to complete, as large Docker image data is transferred into AWS’s cloud. Once the upload is completed, your MISP Docker image will be available for ECS and you can continue to the next section.

Create a MISP Task Definition

The Elastic Container Service (ECS) works by automating the creation, replication, and destruction of containers hosting an application. These applications are managed as Services, which are comprised of one or more Tasks that serve content or perform tasks. These are built from recipes called Task Definitions, which must be defined first, and then referenced by ECS services.

To set this up, go into the Elastic Container Service in the AWS service catalog, and choose the Task Definitions sub section. Click the button labeled Create New Task Definition, and this will start the task definition creation wizard. The first question will ask what type of task you’d like to create, called the Launch Type - for this project, choose FARGATE and click Next Step.

Next, you get to the Configure task and container definitions page, which provides you with an interface to customize the bulk of the service’s configuration. Below is a summary of the options I selected, with a short explanation about why I made the choices:

  • Task Definition Name: misp-blog-dev - A unique name within my AWS account for this application service
  • Task Role: None - Most secure is None, but in the future you may wish to expand this so customizations and plugins can integrate with AWS services
  • Task execution IAM role: ecsTaskExecutionRole - Follow AWS steps to create it if it doesn’t exist in your account yet. This is necessary for ECS to pull images and report monitoring data to AWS.
  • Task memory (GB): 4GB - Feel free to experiment with more or less RAM, to suit your needs.
  • Task CPU (vCPU): 0.5 vCPU - Feel free to experiment with more or less CPU, to suit your needs.

ECS Task Definition Configuration

The remaining configurations for App Mesh and FireLens can be left disabled unless you wish to use those advanced features. Additionally, there is no need to configure a volume as the MySQL database created will serve as the persistent store for the MISP environment. However, before exiting the page by clicking Create, next we will need to add a container.

ECS Add Container

Click the Add container button in the middle of the form, and a new form will pop up that can be populated with container-image specific details. This is where you will tell ECS to use the image created in the prior step. Below is what I used to populate this form, with some explanation. Where I skip over a parameter, simply go with the default choice:

  • Container name: misp-blog-dev - Simply used the same name as above, but this can be different if you want
  • Image: <numeric account id>.dkr.ecr.us-east-2.amazonaws.com/misp-blog:latest - this should be the same tag that was used in the docker push ... command earlier in the section setting up ECR.
  • Memory Limits (MiB): Soft limit / 4096 - Selected a 2GiB “soft limit” for utilization within the container, matching what was provisioned in the task definition.
  • Port mappings: 80 / tcp - MISP serves its content via TCP port 80.

ECS Add Container Standard settings

  • Environment variables: Under environment variables, in the ENVIRONMENT section, the following environment variables are used to configure the MISP application
  • MYSQL_HOST: mispblog.cluster-<random characters>.us-east-2.amazonaws.com - From RDS creation earlier
  • MYSQL_DATABASE: misp - The initial database name chosen in the RDS creation steps
  • MYSQL_USER: mispadmin - The database admin user created during RDS creation
  • MYSQL_PASSWORD: - Password from RDS creation*
  • MYSQL_ROOT_PASSWORD: - Leave empty as this is useful primarily for self-hosted MySQL environments.
  • MISP_ADMIN_EMAIL: Any email address you control. Will only be used if you configure the POSTFIX_RELAY_HOST below.
  • MISP_BASEURL: localhost - If you plan to give your deployment its own domain name, put that here.
  • POSTFIX_RELAY_HOST: - *If you wish to be able to send emails, configure this with an SMTP server name you can send mail to. Setting up AWS Simple Email Service (SES) may be helpful for you. *
  • TIMEZONE: America/New_York - The UNIX time zone identifier for your preferred time zone.
  • DATA_DIR: ./data - This is the default data directory set up in the container

ECS Add Container Environment Settings

Note for production configurations: It is strongly recommended to utilize the AWS KMS service to manage passwords, such as is used in MYSQL_PASSWORD above. The AWS ECS Task Definition allows for you to integrate with a “stored secret” that is safely contained in KMS, and not embedded in the task definition config.

With all this populated, click Add to add the container to the task definition, and you’ll be returned to the prior screen, which you can now click Create on to create the new task definition.

Create the MISP Cluster and Service

Under the Amazon ECS menu, choose the Clusters and click the Create Cluster button. This will bring up the wizard for creating a new cluster. This allows you to choose a cluster template from the available options. To utilize the Fargate for executing MISP, choose Networking only and proceed to Next Step.

ECS Networking Only

On the next page, you can configure the cluster (which is what AWS calls a family of containers that work together) with a unique name and some other options. I left the other options at their defaults and set the name of misp-blog, and clicked Create. You’ll get a launch status screen, and then click the View Cluster button when it becomes available.

Make sure you’re on the Services tab in the new Cluster View, and click Create to start creating a new service. A service is comprised of a task that is kept running by the cluster. If the task quits for (almost) any reason, the cluster will use the service definition to start a new task in its place. Also, in the future if you want to have a multi-headed application (where, say, 3 or more MISP applications use the same database, to support more users) the service also can be configured to allow that and will handle features like auto-scaling more tasks as demand requires, or stopping them as traffic dies down.

ECS Services Tab

For the Configure service view that comes up, I chose the Launch Type of FARGATE, the task definition of misp-blog-dev and the latest revision from the revision drop-down. Give it a unique service name, like misp and set the number of tasks to 1 (if you want more than one task, you’d want to set up a load balancer, which is outside the scope of this walkthrough). The rest of the options can be left as-is, so next click Next step.

ECS Configure Service

In the Next step you’ll get to choose the VPC - make sure you put this in the same VPC that the Aurora RDS instance is running in (you can verify this from the RDS service panel). Under the Subnets drop-down, you can select one or more subnets that you would like MISP to be deployed within. Unless you have some specific network architecture you’re designing, any of the subnets here are fine. You can even select more than one, and the MISP instance will spin up in one of them chosen at random. Similar to earlier with RDS, a new Security group will be created for your task. You will want to record this value and save it for later. Again, for a basic setup, the rest of the options can be left at their defaults, so click Next step.

ECS Configure Networking

The next page allows you to adjust the Auto scaling features of the service. I won’t go into these here, but if you aspire to host a heavier-utilized instance, these settings will come in handy for auto-creating more workers to handle more user traffic. To move on from here, click Next Step. You’ll be presented with a Review screen, which allows you to verify your choices before clicking Create Service.

Allow the MISP Security Group Access to the RDS Database

We aren’t done yet. Next, we need to grant access for the MISP tasks to talk to the Aurora RDS MySQL instance. This is where we want the two security groups that we have noted, to be handy. From the Services menu, click EC2 to bring up the EC2 service panel. On the left-hand sidebar, about mid-way down, is the Network & Security section with a link for Security Groups. Click that to bring up the list of Security Groups.

EC2 Network & Seecurity

Search for the Security Group name for your RDS instance - make sure its the RDS instance’s security group, not the one you just created for your ECS task. Click on its sg-* unique identifier, and this should bring up the detail view for that security group. In the lower section, there should be tabs for Inbound rules, Outboud rules, and Tags. Make sure Inbound rules is chosen, by clicking it, and then click the Edit inbound rules on the right-hand side of that panel. Click the Add rule button and a new row will appear. The left-most field is Type - use the drop-down to set this to MySQL/Aurora. Then, in the middle of the row, two fields comprise the Source entry: a source type, which is a drop-down and you want to set to Custom if that isn’t the default, and then the source address field. In this field, type the name of the Security Group you created recently in your ECS task. Add a short comment in the last field to help you remember what this entry is for, and then click Save rules.

EC2 Add Inbound

Once this is complete, your service should have access to the database. As the MISP application is starting up, it will keep retrying to contact the database address you provided for it until it finally connects successfully. So, if changing the security group takes awhile, the application will eventually recognize it finally has permission to access the dataase and begin setting up the initial default application.

If you browse back to the Service section of the ECS Cluster that was created earlier, you’ll have access to a Logs tab, which can show you the progress.

ECS Service Logs

Watch the logs, and after the following line is displayed, your application is up and running, and you should be able to proceed:

MISP Live

Before you call it a day, make sure to continue down the following section to log into MISP and perform some inition setup, such as resetting the default password. Don’t close the current browser tab quite yet.

Connecting To the New MISP Server

In the Logs view, above, you can see the right-hand column is a long hexadecimal value that is hyperlinked. Clicking on this will allow you to view the details for the running task that is managed by the misp-blog service you created. Alternatively, If you click on the Tasks tab, you’ll be able to see the MISP task that represents the running MISP application, and you can click on the task name in that view to get to the same Task details view. This view looks to contain information similar to the Review screen, from earlier when the service was being created. From here, look for the Public IP. This will be the public-facing IP address that was given to your MISP application. Do note, however, that this IP address wil change whenever the task is stopped and a new task started in its place.

MISP Task Details

Open a new tab in the web browser and go to this IP address (you might need to manually enter it with http:// in front to avoid some browsers trying to use HTTPS by default). You should be presented with the login screen for MISP. The initial default credentials are:

  • Email: admin@admin.test
  • Password: admin

MISP Login Screen

Upon logging in, you will be forced to change the password to something else. Do this immediately. The Confirm with your current password field should be populated with the admin password from above. Note that, as of this writing, MISP requires a 12+ character password. All of the user information is maintained in your RDS database, so this change will persist across restarts (and IP changes) of the MISP task.

MISP Change Password

Once the password is changed, you’ll remain logged in as admin@admin.test, and will be able to start using MISP. There are quite a few examples of MISP training and documentation out there.

Here is a good consolidated list that is actively maintained by the MISP project volunteers: