Thursday, July 28, 2016

Exposing a private Amazon RDS instance with iptables NAT rules

I needed to expose a private Amazon MySQL RDS instance to a 3rd party SaaS tool. I tried several approaches and finally found one that seemed to work pretty well.

I ended up creating a small EC2 instance in the same VPC as the RDS instance, and applied these iptables NAT/masquerading rules to it, mapping local port 3307 to port 3306 on the RDS instance, whose internal IP address is in this case

# cat

iptables -F
iptables -F -t nat
iptables -X
iptables -t nat -A PREROUTING -p tcp --dport 3307 -j DNAT --to
iptables -A FORWARD -p tcp -d --dport 3306 -j ACCEPT
iptables -t nat -A OUTPUT -p tcp -o lo --dport 3307 -j DNAT --to
iptables -t nat -A POSTROUTING  -j MASQUERADE

I also had to enable IP forwarding on the EC2 instance:

# sysctl net.ipv4.ip_forward
# sysctl -p

At this point, I was able to hit the external IP of the EC2 instance on port 3307, and get to the private RDS instance on port 3306. I was also able to attach the EC2 instance to an EC2 Security Group allowing the 3rd party SaaS tool IP addresses to access port 3307 on the EC2 instance.

My thanks to the people discussing a similar issue on this thread of LinuxQuestions. Without their discussion, I don't think I'd have been able to figure out a solution.

Wednesday, July 13, 2016

Using JMESPath queries with the AWS CLI

The AWS CLI, based on the boto3 Python library, is the recommended way of automating interactions with AWS. In this post I'll show some examples of more advanced AWS CLI usage using the query mechanism based on the JMESPath JSON query language.

Installing the AWS CLI tools is straightforward. On Ubuntu via apt-get:

# apt-get install awscli

Or via pip:

# apt-get install python-pip
# pip install awscli

The next step is to configure awscli by specifying the AWS Access Key ID and AWS Secret Access Key, as well as the default region and output format:

# aws configure
AWS Access Key ID: your-aws-access-key-id
AWS Secret Access Key: your-aws-secret-access-key
Default region name [us-west-2]: us-west-2

Default output format [None]: json

The configure command creates a ~/.aws directory containing two files: config and credentials.

You can specify more than one pair of AWS keys by creating profiles in these files. For example, in ~/.aws/credentials you can have:

[profile profile1]

[profile profile2]

In ~/.aws/config you can have:
[profile profile1]
region = us-west-2

[profile profile2]
region = us-east-1

You can specify a given profile when you run awscli:

# awscli --profile profile1

Let's assume you want to write a script using awscli that deletes EBS snapshots older than N days. Let's go through this one step at a time.

Here's how you can list all snapshots owned by you:

# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER --query "Snapshots[]"

Note the use of the --query option. It takes a parameter representing a JMESPath JSON query string. It's not trivial to figure out how to build these query strings, and I advise you to spend some time reading over the JMESPath tutorial and JMESPath examples.

In the example above, the query string is simply "Snapshots[]", which represents all the snapshots that are present in the AWS account associated with the profile profile1. The default output in our case is JSON, but you can specify --output text at the aws command line if you want to see each snapshot on its own line of text.

Let's assume that when you create the snapshots, you specify a description with contains PROD or STAGE for EBS volumes attached to production and stage EC2 instances respectively. If you want to only display snapshots containing the string PROD, you would do:

# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER--query "Snapshots[?contains(Description, \`PROD\`) == \`true\`]" --output text

The Snapshots[] array now contains a condition represented by the question mark ?. The condition uses the contains() function included in the JMESPath specification, and is applied against the Description field of each object in the Snapshots[] array, verifying that it contains the string PROD.  Note the use of backquotes surrounding the strings PROD and true in the condition. I spent some quality time troubleshooting my queries when I used single or double quotes with no avail. The backquotes also need to be escaped so that the shell doesn't interpret them as commands to be executed.

To restrict the PROD snapshots even further, to the ones older than say 7 days ago, you can do something like this:

TARGET_DATE=`date --date="$DAYS day ago" +%Y-%m-%d`

# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER--query "Snapshots[?contains(Description, \`PROD\`) == \`true\`]|[?StartTime < \`$TARGET_DATE\`]" --output text

Here I used the StartTime field of the objects in the Snapshots[] array and compared it against the target date. In this case, string comparison is good enough for the query to work.

In all the examples above, the aws command returned a subset of the Snapshots[] array and displayed all fields for each object in the array. If you wanted to display specific fields, let's say the ID, the start time and the description of each snapshot, you would run:

# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER--query "Snapshots[?contains(Description, \`PROD\`) == \`true\`]|[?StartTime < \`$TARGET_DATE\`].[SnapshotId,StartTime,Description]" --output text

To delete old snapshots, you can use the aws ec2 delete-snapshot command, which needs a snapshot ID as a parameter. You could use the command above to list only the SnapshotId for snapshots older than N days, then for each of these IDs, run something like this:

# aws ec2 delete-snapshot --profile profile1 --snapshot-id $id

All this is well and good when you run these commands interactively at the shell. However, I had no luck running them out of cron. The backquotes resulted in boto3 syntax errors. I had to do it the hard way, by listing all snapshots first, then going all in with sed and awk:

aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER --output=text --query "Snapshots[].[SnapshotId,StartTime,Description]"  > $TMP_SNAPS

TARGET_DATE=`date --date="$DAYS day ago" +%Y-%m-%d`

cat $TMP_SNAPS | grep PROD | sed 's/T[0-9][0-9]:[0-9][0-9]:[0-9][0-9].000Z//' | awk -v target_date="$TARGET_DATE" '{if ($2 < target_date){print}}' > $TMP_PROD_SNAPS


for sid in `awk '{print $1}' $TMP_PROD_SNAPS` ; do
echo Deleting PROD snapshot $sid
aws ec2 delete-snapshot --profile $PROFILE --region $REGION --snapshot-id $sid

Ugly, but it works out of cron. Hope it helps somebody out there.

July 14th 2016: I initially forgot to include this very good blog post from Joseph Lawson on advanced JMESPath usage with the AWS CLI.

Friday, July 01, 2016

More tips and tricks for running Gatling in Docker containers

This post is a continuation of my previous one on "Running Gatling tests in Docker containers via Jenkins". As I continued to set up Jenkins jobs to run Gatling tests, I found the need to separate those tests for different environments - development, staging and production. The initial example I showed contained a single setup, which is not suitable for multiple environments.

Here is my updated Gatling directory structure


Note that I created a separate directory under simulations for each environment (development, staging, production), each with its own simulation files.

I also created a data directory under user-files, because that is the default location for CSV files used by Gatling feeders.

Most importantly, I created a separate configuration directory (staging, production) under gatling/conf, each directory containing its own customized gatling.conf file. I started by copying the gatling-defaults.conf file from GitHub to gatling/conf/staging/gatling.conf and gatling/conf/production/gatling.conf respectively.

Here is what I customized in staging/gatling.conf:

mute = true # When set to true, don't ask for simulation name nor run description
simulations = user-files/simulations/staging

I customized production/gatling.conf in a similar way:

mute = true # When set to true, don't ask for simulation name nor run description
simulations = user-files/simulations/production

Setting mute to true is important because without it, running Gatling in a Docker container was segfaulting while waiting for user input for the simulation ID:

Select simulation id (default is 'gatlingsimulation'). Accepted characters are a-z, A-Z, 0-9, - and _ 
Exception in thread "main" java.lang.NullPointerException
at$Selector.loop$1(Selection.scala:127) at$Selector.askSimulationId(Selection.scala:135) at$Selector.selection(Selection.scala:50) at$.apply(Selection.scala:33) at at at$.start(Gatling.scala:57) at$.fromArgs(Gatling.scala:49) at$.main(Gatling.scala:43) at

The other customization was to point the simulations attribute to the specific staging or production sub-directories.

Since the CSV files containing URLs to be load tested are also environment-specific, I modified the Simulation.scala files to take this into account. I also added 2 JAVA_OPTS variables that can be passed at runtime for HTTP basic authentication. Here is the new Crawl object (compare with the one from my previous post):

object Crawl {
  val feeder = csv("staging-urls.csv").random

  val userName = System.getProperty("username")
  val userPass = System.getProperty("password")

  val crawl = exec(feed(feeder)
    .get("${loc}").basicAuth(userName, userPass)

One more thing is needed: to make Gatling use a specific configuration file instead of its default one, which is conf/gatling.conf. To do that, I set GATLING_CONF as an ENV variable in the Dockerfile, so it can be passed as a 'docker run' command line parameter. Here is the Dockerfile:

# Gatling is a highly capable load testing tool.
# Documentation:
# Cheat sheet:

FROM java:8-jdk-alpine

MAINTAINER Denis Vazhenin

# working directory for gatling

# gating version

# create directory for gatling install
RUN mkdir -p gatling

# install gatling
RUN apk add --update wget && \
  mkdir -p /tmp/downloads && \
  wget -q -O /tmp/downloads/gatling-$ \$GATLING_VERSION/gatling-charts-highcharts-bundle-$ && \
  mkdir -p /tmp/archive && cd /tmp/archive && \
  unzip /tmp/downloads/gatling-$ && \
  mv /tmp/archive/gatling-charts-highcharts-bundle-$GATLING_VERSION/* /opt/gatling/

# change context to gatling directory
WORKDIR  /opt/gatling

# set directories below to be mountable from host
VOLUME ["/opt/gatling/conf", "/opt/gatling/results", "/opt/gatling/user-files"]

# set environment variables
ENV PATH /opt/gatling/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV GATLING_HOME /opt/gatling
ENV GATLING_CONF /opt/gatling/conf


Finally, here is how I invoke 'docker run' to tie everything together:

docker run --rm -v ${WORKSPACE}/gatling/conf:/opt/gatling/conf -v ${WORKSPACE}/gatling/user-files:/opt/gatling/user-files -v ${WORKSPACE}/gatling/results:/opt/gatling/results -e GATLING_CONF="/opt/gatling/conf/staging" -e JAVA_OPTS="-Dusers=$USERS -Dduration=$DURATION -Dusername=myusername -Dpassword=mypass" /PATH/TO/DOCKER/REGISTRY/gatling

Note the GATLING_CONF parameter passed with -e with the value of /opt/gatling/conf/staging. Also note the username and password JAVA_OPTS parameters.

Happy load testing!

Tuesday, June 28, 2016

Running Gatling load tests in Docker containers via Jenkins

Gatling is a modern load testing tool written in Scala. As part of the Jenkins setup I am in charge of, I wanted to run load tests using Gatling against a collection of pages for a given website. Here are my notes on how I managed to do this.

Running Gatling as a Docker container locally

There is a Docker image already available on DockerHub, so you can simply pull down the image locally:

$ docker pull denvazh/gatling:2.2.2

Instructions on how to run a container based on this image are available on GitHub:

$ docker run -it --rm -v /home/core/gatling/conf:/opt/gatling/conf \
-v /home/core/gatling/user-files:/opt/gatling/user-files \
-v /home/core/gatling/results:/opt/gatling/results \

Based on these instructions, I created a local directory called gatling, and under it I created 3 sub-directories: conf, results and user-files. I left the conf and results directories empty, and under user-files I created a simulations directory containing a Gatling load test scenario written in Scala. I also created a file in the user-files directory called urls.csv, containing a header named loc and a URL per line for each page that I want to load test.

Assuming the current directory is gatling, here are examples of these files:

$ cat user-files/urls.csv

$ cat user-files/simulations/Simulation.scala

package my.gatling.simulation

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._

class GatlingSimulation extends Simulation {

  val httpConf = http
    .acceptEncodingHeader("gzip, deflate")
    .userAgentHeader("Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0")

  val scn1 = scenario("Scenario1")

  val userCount = Integer.getInteger("users", 1)
  val durationInSeconds  = java.lang.Long.getLong("duration", 10L)
    scn1.inject(rampUsers(userCount) over (durationInSeconds seconds))

object Crawl {

  val feeder = csv("/opt/gatling/user-files/urls.csv").random

  val crawl = exec(feed(feeder)

I won't go through the different ways of writing Gatling load tests scenarios here. There are good instructions on the Gatling website -- see the Quickstart and the Advanced Tutorial. What the scenario above does is it reads the file urls.csv and randomly picks a URL from it, then runs a load test against that URL.

I do want to point out 2 variables in the above script:

  val userCount = Integer.getInteger("users", 1)
  val durationInSeconds  = java.lang.Long.getLong("duration", 10L)

These variables specify the max number of users we want to ramp up to, and the duration of the ramp-up. They are used in the inject call:

scn1.inject(rampUsers(userCount) over (durationInSeconds seconds))

The special thing about these 2 variables is that they are read from JAVA_OPTS by Gatling. So if you have a -Dusers Java option and a -Dduration Java option, Gatling will know how to read them and how to set the userCount and durationInSeconds variables accordingly. This is a good thing, because it allows you to specify those numbers outside of Gatling, without hardcoding them in your simulation script. Here is more info on passing parameters via the command line to Gatling.

While pulling the Gatling docker image and running it is the simplest way to run Gatling, I prefer to understand what's going on in that image. I started off by getting the Dockerfile from GitHub:

$ cat Dockerfile

# Gatling is a highly capable load testing tool.
# Documentation:
# Cheat sheet:

FROM java:8-jdk-alpine

MAINTAINER Denis Vazhenin

# working directory for gatling

# gating version

# create directory for gatling install
RUN mkdir -p gatling

# install gatling
RUN apk add --update wget && \
  mkdir -p /tmp/downloads && \
  wget -q -O /tmp/downloads/gatling-$ \$GATLING_VERSION/gatling-charts-highcharts-bundle-$ && \
  mkdir -p /tmp/archive && cd /tmp/archive && \
  unzip /tmp/downloads/gatling-$ && \
  mv /tmp/archive/gatling-charts-highcharts-bundle-$GATLING_VERSION/* /opt/gatling/

# change context to gatling directory
WORKDIR  /opt/gatling

# set directories below to be mountable from host
VOLUME ["/opt/gatling/conf", "/opt/gatling/results", "/opt/gatling/user-files"]

# set environment variables
ENV PATH /opt/gatling/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV GATLING_HOME /opt/gatling


I then added a way to pass JAVA_OPTS via an environment variable. I added this line after the ENV GATLING_HOME line:


I dropped this Dockerfile in my gatling directory, then built a local Docker image off of it:

$ docker build -t gatling:local .

I  then invoked 'docker run' to launch a container based on this image, using the csv and simulation files from above. The current directory is still gatling.

$ docker run --rm -v `pwd`/conf:/opt/gatling/conf -v `pwd`/user-files:/opt/gatling/user-files -v `pwd`/results:/opt/gatling/results -e JAVA_OPTS="-Dusers=10 -Dduration=60" gatling:local -s MySimulationName

Note the -s flag which denotes a simulation name (which can be any string you want). If you don't specify this flag, the script which is the ENTRYPOINT in the container will wait for some user input and you will not be able to fully automate your load test.

Another thing to note is the use of JAVA_OPTS. In the example above, I pass -Dusers=10 and   -Dduration=60 as the two JAVA_OPTS parameters. The JAVA_OPTS variable itself is passed to 'docker run' via the -e option, which tells Docker to replace the default value for ENV JAVA_OPTS (which is "") with the value passed with -e.

Running Gatling as a Docker container from Jenkins

Once you have a working Gatling container locally, you can upload the Docker image built above to a private Docker registry. I used a private EC2 Container Registry (ECR).  

I also added the gatling directory and its sub-directories to a GitHub repository called devops.

In Jenkins, I created a new "Freestyle project" job with the following properties:

  • Parameterized build with 2 string parameters: USERS (default value 10) and DURATION in seconds (default value 60)
  • Git repository - add URL and credentials for the devops repository which contains the gatling files
  • An "Execute shell" build command similar to this one:
docker run --rm -v ${WORKSPACE}/gatling/conf:/opt/gatling/conf -v ${WORKSPACE}/gatling/user-files:/opt/gatling/user-files -v ${WORKSPACE}/gatling/results:/opt/gatling/results -e JAVA_OPTS="-Dusers=$USERS -Dduration=$DURATION"  /PATH/TO/DOCKER/REGISTRY/gatling -s MyLoadTest 

Note that we mount the gatling directories as Docker volumes, similarly to when we ran the Docker container locally, only this time we specify ${WORKSPACE} as the base directory. The 2 string parameters USERS and DURATION are passed as variables in JAVA_OPTS.

A nice thing about running Gatling via Jenkins is that the reports are available in the Workspace directory of the project. If you go to the Gatling project we created in Jenkins, click on Workspace, then on gatling, then results, you should see directories named gatlingsimulation-TIMESTAMP for each Gatling run. Each of these directories should have an index.html file, which will show you the Gatling report dashboard. Pretty neat.

Thursday, June 16, 2016

Running Jenkins jobs in Docker containers

One of my main tasks at work is to configure Jenkins to act as a hub for all the deployment and automated testing jobs we run. We use CloudBees Jenkins Enterprise, mostly for its Role-Based Access Control plugin, which allows us to create one Jenkins folder per project/application and establish fine grained access control to that folder for groups of users. We also make heavy use of the Jenkins Enterprise Pipeline features (which I think are also available these days in the open source version).

Our Jenkins infrastructure is composed of a master node and several executor nodes which can run jobs in parallel if needed.

One pattern that my colleague Will Wright and I have decided upon is to run all Jenkins jobs as Docker containers. This way, we only need to install Docker Engine on the master node and the executor nodes. No need to install any project-specific pre-requisites or dependencies on every Jenkins node. All of these dependencies and pre-reqs are instead packaged in the Docker containers. It's a simple but powerful idea, that has worked very well for us. One of the nice things about this pattern is that you can keep adding various types of automated tests. If it can run from the command line, then it can run in a Docker container, which means you can run it from Jenkins!

I have seen this pattern discussed in multiple places recently, for example in this blog post about "Using Docker for a more flexible Jenkins".

Here are some examples of Jenkins jobs that we create for a given project/application:
  • a deployment job that runs Capistrano in its own Docker container, against targets in various environments (development, staging, production); this is a Pipeline script written in Groovy, which can call other jobs below
  • a Web UI testing job that runs the Selenium Python WebDriver and drives Firefox in headless mode (see my previous post on how to do this with Docker)
  • a JavaScript syntax checking job that runs JSHint against the application's JS files
  • an SSL scanner/checker that runs SSLyze against the application endpoints
We also run other types of tasks, such as running an AWS CLI command to perform certain actions, for example to invalidate a CloudFront resource. I am going to show here how we create a Docker image for one of these jobs, how we test it locally, and how we then integrate it in Jenkins.

I'll use as an example a simple Docker image that installs the AWS CLI package and runs a command when the container is invoked via 'docker run'.

I assume you have a local version of Docker installed. If you are on a Mac, you can use Docker Toolbox, or, if you are lucky and got access to it, you can use the native Docker for Mac. In any case,  I will assume that you have a local directory called awscli with the following Dockerfile in it:

FROM ubuntu:14.04

MAINTAINER You Yourself <>

# disable interactive functions
ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
    apt-get install -y python-pip && \
    pip install awscli



As I mentioned, this simply installs the awscli Python package via pip, then runs a command given as an environment variable when you invoke 'docker run'. It also uses two other environment variables that contain the AWS access key ID and secret access key. You don't want to hardcode these secrets in the Dockerfile and have them end up on GitHub.

The next step is to build an image based on this Dockerfile. I'll call the image awscli and I'll tag it as local:

$ docker build -t awscli:local .

Then you can run a container based on this image. The command line looks a bit complicated because I am passing (via the -e switch) the 3 environment variables discussed above:

$ docker run --rm -e AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY -e AWS_COMMAND='aws cloudfront create-invalidation --distribution-id=abcdef --invalidation-batch Paths={Quantity=1,Items=[/media/*]},CallerReference=my-invalidation-123456' awscli:local

(where distribution-id needs to be the actual ID of your CloudFront distribution, and CallerReference needs to be unique per invalidation)

If all goes well, you should see the output of the 'aws cloudfront create-invalidation' command.

In our infrastructure, we have a special GitHub repository where we check in the various folders containing the Dockerfiles and any static files that need to be copied over to the Docker images. When we push the awscli directory to GitHub for example, we have a Jenkins job that will be notified of that commit and that will build the Docker image (similarly to how we did it locally with 'docker build'), then it will 'docker push' the image to a private AWS ECR repository we have.

Now let's assume we want to create a Jenkins job that will run this image as a container. First we define 2 secret credentials, specific to the Jenkins folder where we want to create the job (there are also global Jenkins credentials that can apply to all folders). These credentials are of type "Secret text" and contain the AWS access key ID and the AWS secret access key.

Then we create a new Jenkins job of type "Freestyle project" and call it cloudfront.invalidate. The build for this job is parameterized and contains 2 parameters: CF_ENVIRONMENT which is a drop-down containing the values "Staging" and "Production" referring to the CloudFront distribution we want to invalidate; and CF_RESOURCE, which is a text variable that needs to be set to the resource that needs to be invalidated (e.g. /media/*).

In the Build Environment section of the Jenkins job, we check "Use secret text(s) or file(s)" and add 2 Bindings, one for the first secret text credential containing the AWS access key ID, which we save in a variable called AWS_ACCESS_KEY_ID, and the other one for the second secret text credential containing the AWS secret access key, which we save in a variable called AWS_SECRET_ACCESS_KEY.

The Build section for this Jenkins job has a step of type "Execute shell" which uses the parameters and variables defined above and invokes 'docker run' using the path to the Docker image from our private ECR repository:


INVALIDATION_ID=jenkins-invalidation-`date +%Y%m%d%H%M%S`

COMMAND="aws cloudfront create-invalidation --distribution-id=$DISTRIBUTION_ID --invalidation-batch Paths={Quantity=1,Items=[$CF_RESOURCE]},CallerReference=$INVALIDATION_ID"


When this job is run, the Docker image gets pulled down from AWS ECR, then a container based on the image is run and then removed upon completion (that's what --rm does, so that no old containers are left around).

I'll write another post soon with some more examples of Jenkins jobs that we run as Docker containers to do Selenium test, JSHint testing and SSLyze scanning.

Thursday, May 26, 2016

Setting up AWS CloudFront for Magento

Here are some steps I jotted down for setting up AWS CloudFront as a CDN for the 3 asset directories that are used by Magento installations. I am assuming your Magento application servers are behind an ELB.

SSL certificate upload to AWS

Install aws command line utilities.

$ pip install awscli

Configure AWS credentials

Create IAM user and associate it with the IAMFullAccess policy. Run ‘aws configure’ and specify the user’s keys and the region.

Bring SSL key, certificate and intermediate certificate in current directory:

-rw-r--r-- 1 root root 4795 Apr 11 20:34 gd_bundle-g2-g1.crt
-rw-r--r-- 1 root root 1830 Apr 11 20:34
-rw------- 1 root root 1675 Apr 11 20:34

Run following script for installing wildcard SSL certificate to be used in staging CloudFront setup:

$ cat

aws iam upload-server-certificate --server-certificate-name WILDCARD_MYDOMAIN_COM_FOR_PROD_CF --certificate-body file:// --private-key file:// --certificate-chain file://gd_bundle-g2-g1.crt --path /cloudfront/prod/

After uploading the SSL certificates, they will be available in drop-downs when configuring CloudFront for SSL.

Apache Cache-Control headers setup
  • Add these directives (modifying max-age accordingly) in all Apache vhosts, both for port 80 and for port 443
 <FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">
        Header set Cache-Control "max-age=604800, public"

CloudFront setup
  • Origin: prod ELB (
  • Alternate domain name:\
  • Custom SSL client support: Only Clients that Support Server Name Indication (SNI)
  • Domain name:
  • Behaviors
    • /media/* /skin/* /js/*
    • Viewer protocol policy: HTTP and HTTPS
    • Allowed HTTP methods: GET, HEAD
    • Forward headers: None
    • Object caching: Use origin cache headers
    • Forward cookies: None
    • Forward query strings: Yes
    • Smooth streaming: No
    • Restrict viewer access: No
    • Compress objects automatically: No

DNS setup
  • is a CNAME pointing to the CloudFront domain name above

Magento setup

This depends on the version of Magento you are running (1.x or 2.x), but you want to look for settings for the Base Skin URL, Base Media URL and Base Javascript URL, which are usually under System->Configuration->General-Web. You need to set them to point to the domain name you set up as a CNAME to CloudFront.

Base Skin URL:
Base Media URL:
Base Javascript URL:

More in-depth Magento-specific instructions for integrating with CloudFront are available here.

Friday, April 15, 2016

LDAP server setup and client authentication

We recently bought at work a CloudBees Jenkins Enterprise license and I wanted to tie the user accounts to a directory service. I first tried to set up Jenkins authentication via the AWS Directory Service, hoping it will be pretty much like talking to an Active Directory server. That proved to be impossible to set up, at least for me. I also tried to have an LDAP proxy server talking to the AWS Directory Service and have Jenkins authenticate against the LDAP proxy. No dice. I ended up setting up a good old-fashioned LDAP server and managed to get Jenkins working with it. Here are some of my notes.

OpenLDAP server setup

I followed this excellent guide from Digital Ocean. The server was an Ubuntu 14.04 EC2 instance in my case. What follows in terms of the server setup is taken almost verbatim from the DO guide.

Set the hostname

# hostnamectl set-hostname my-ldap-server

Edit /etc/hosts and make sure this entry exists:

LOCAL_IP_ADDRESS my-ldap-server

(it makes a difference that the FQDN is the first entry in the line above!)

Make sure the following types of names are returned when you run hostname with different options:

# hostname

# hostname -f

# hostname -d

Install slapd

# apt-get install slapd ldap-utils
# dpkg-reconfigure slapd

(here you specify the LDAP admin password)

Install the SSL Components

# apt-get install gnutls-bin ssl-cert

Create the CA Template

# mkdir /etc/ssl/templates
# vi /etc/ssl/templates/ca_server.conf
# cat /etc/ssl/templates/ca_server.conf
cn = LDAP Server CA

Create the LDAP Service Template

# vi /etc/ssl/templates/ldap_server.conf
# cat /etc/ssl/templates/ldap_server.conf
organization = "My Company"
cn =
expiration_days = 3650

Create the CA Key and Certificate

# certtool -p --outfile /etc/ssl/private/ca_server.key
# certtool -s --load-privkey /etc/ssl/private/ca_server.key --template /etc/ssl/templates/ca_server.conf --outfile /etc/ssl/certs/ca_server.pem

Create the LDAP Service Key and Certificate

# certtool -p --sec-param high --outfile /etc/ssl/private/ldap_server.key
# certtool -c --load-privkey /etc/ssl/private/ldap_server.key --load-ca-certificate /etc/ssl/certs/ca_server.pem --load-ca-privkey /etc/ssl/private/ca_server.key --template /etc/ssl/templates/ldap_server.conf --outfile /etc/ssl/certs/ldap_server.pem

Give OpenLDAP Access to the LDAP Server Key

# usermod -aG ssl-cert openldap
# chown :ssl-cert /etc/ssl/private/ldap_server.key
# chmod 640 /etc/ssl/private/ldap_server.key

Configure OpenLDAP to Use the Certificate and Keys

IMPORTANT NOTE: in modern versions of slapd, configuring the server is not done via slapd.conf anymore. Instead, you put together ldif files and run LDAP client utilities such as ldapmodify against the local server. The Distinguished Name of the entity you want to modify in terms of configuration is generally dn: cn=config but it can also be the LDAP database dn: olcDatabase={1}hdb,cn=config.

# vi addcerts.ldif
# cat addcerts.ldif
dn: cn=config
changetype: modify
add: olcTLSCACertificateFile
olcTLSCACertificateFile: /etc/ssl/certs/ca_server.pem
add: olcTLSCertificateFile
olcTLSCertificateFile: /etc/ssl/certs/ldap_server.pem
add: olcTLSCertificateKeyFile
olcTLSCertificateKeyFile: /etc/ssl/private/ldap_server.key

# ldapmodify -H ldapi:// -Y EXTERNAL -f addcerts.ldif
# service slapd force-reload
# cp /etc/ssl/certs/ca_server.pem /etc/ldap/ca_certs.pem
# vi /etc/ldap/ldap.conf

* set TLS_CACERT to following:
TLS_CACERT /etc/ldap/ca_certs.pem

# ldapwhoami -H ldap:// -x -ZZ

Force Connections to Use TLS

Change olcSecurity attribute to include 'tls=1':

# vi forcetls.ldif
# cat forcetls.ldif
dn: olcDatabase={1}hdb,cn=config
changetype: modify
add: olcSecurity
olcSecurity: tls=1

# ldapmodify -H ldapi:// -Y EXTERNAL -f forcetls.ldif
# service slapd force-reload
# ldapsearch -H ldap:// -x -b "dc=mycompany,dc=com" -LLL dn
(shouldn’t work)

# ldapsearch -H ldap:// -x -b "dc=mycompany,dc=com" -LLL -Z dn
(should work)

Disallow anonymous bind

Create user binduser to be used for LDAP searches:

# vi binduser.ldif
# cat binduser.ldif
dn: cn=binduser,dc=mycompany,dc=com
objectClass: top
objectClass: account
objectClass: posixAccount
objectClass: shadowAccount
cn: binduser
uid: binduser
uidNumber: 2000
gidNumber: 200
homeDirectory: /home/binduser
loginShell: /bin/bash
gecos: suser
userPassword: {crypt}x
shadowLastChange: -1
shadowMax: -1
shadowWarning: -1

# ldapadd -x -W -D "cn=admin,dc=mycompany,dc=com" -Z -f binduser.ldif
Enter LDAP Password:
adding new entry "cn=binduser,dc=mycompany,dc=com"

Change olcDissalows attribute to include bind_anon:

# vi disallow_anon_bind.ldif
# cat disallow_anon_bind.ldif
dn: cn=config
changetype: modify
add: olcDisallows
olcDisallows: bind_anon

# ldapmodify -H ldapi:// -Y EXTERNAL -ZZ -f disallow_anon_bind.ldif
# service slapd force-reload

Also disable anonymous access to frontend:

# vi disable_anon_frontend.ldif
# cat disable_anon_frontend.ldif
dn: olcDatabase={-1}frontend,cn=config
changetype: modify
add: olcRequires
olcRequires: authc

# ldapmodify -H ldapi:// -Y EXTERNAL -f disable_anon_frontend.ldif
# service slapd force-reload

Create organizational units and users

Create helper scripts:

# cat


ldapadd -x -w adminpassword -D "cn=admin,dc=mycompany,dc=com" -Z -f $LDIF

# cat


ldapmodify -x -w adminpassword -D "cn=admin,dc=mycompany,dc=com" -Z -f $LDIF

# cat


ldappasswd -s $PASS -w adminpassword -D "cn=admin,dc=mycompany,dc=com" -x "uid=$USER,ou=users,dc=mycompany,dc=com" -Z

Create ‘mypeople’ organizational unit:

# cat add_ou_mypeople.ldif
dn: ou=mypeople,dc=mycompany,dc=com
objectclass: organizationalunit
ou: users
description: all users

# ./ add_ou_mypeople.ldif

Create 'groups' organizational unit:

# cat add_ou_groups.ldif
dn: ou=groups,dc=mycompany,dc=com
objectclass: organizationalunit
ou: groups
description: all groups

# ./ add_ou_groups.ldif

Create users (note the shadow attributes set to -1, which means they will be ignored):

# cat add_user_myuser.ldif
dn: uid=myuser,ou=mypeople,dc=mycompany,dc=com
objectClass: top
objectClass: account
objectClass: posixAccount
objectClass: shadowAccount
cn: myuser
uid: myuser
uidNumber: 2001
gidNumber: 201
homeDirectory: /home/myuser
loginShell: /bin/bash
gecos: myuser
userPassword: {crypt}x
shadowLastChange: -1
shadowMax: -1
shadowWarning: -1

# ./ add_user_myuser.ldif
# ./ myuser MYPASS

Enable LDAPS

In /etc/default/slapd set:

SLAPD_SERVICES="ldap:/// ldaps:/// ldapi:///"

Enable debugging

This was a life saver when it came to troubleshooting connection issues from clients such as Jenkins or other Linux boxes. To enable full debug output, set olcLogLevel to -1:

# cat enable_debugging.ldif
dn: cn=config
changetype: modify
add: olcLogLevel
olcLogLevel: -1

# ldapadd -H ldapi:// -Y EXTERNAL -f enable_debugging.ldif
# service slapd force-reload

Configuring Jenkins LDAP authentication

Verify LDAPS connectivity from Jenkins to LDAP server

In my case, the Jenkins server is in the same VPC and subnet as the LDAP server, so I added an /etc/hosts entry on the Jenkins box pointing to the FQDN of the LDAP server so it can hit its internal IP address:


I verified that port 636 (used by LDAPS) on the LDAP server is reachable from the Jenkins server:

# telnet 636
Connected to
Escape character is '^]'.

Set up LDAPS client on Jenkins server (StartTLSdoes not work w/ Jenkins LDAP plugin!)

# apt-get install ldap-utils

IMPORTANT: Copy over /etc/ssl/certs/ca_server.pem from LDAP server as /etc/ldap/ca_certs.pem on Jenkins server and then:

# vi /etc/ldap/ldap.conf
TLS_CACERT /etc/ldap/ca_certs.pem

Add LDAP certificates to Java keystore used by Jenkins

As user jenkins:
$ mkdir .keystore
$ cp /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/security/cacerts .keystore/
(you may need to customize the above line in terms of the path to the cacerts file -- it is the one under your JAVA_HOME)

$ keytool --keystore /var/lib/jenkins/.keystore/cacerts --import --alias --file /etc/ldap/ca_certs.pem
Enter keystore password: changeit
Owner: CN=LDAP Server CA
Issuer: CN=LDAP Server CA
Serial number: 570bddb0
Valid from: Mon Apr 11 17:24:00 UTC 2016 until: Tue Apr 11 17:24:00 UTC 2017
Certificate fingerprints:

Trust this certificate? [no]:  yes
Certificate was added to keystore

In /etc/default/jenkins, set JAVA_ARGS to:

As root, restart jenkins:

# service jenkins restart

Jenkins settings for LDAP plugin

This took me a while to get right. The trick was to set the rootDN to dc=mycompany, dc=com and the userSearchBase to ou=mypeople (or to whatever name you gave to your users' organizational unit). I also tried to get LDAP groups to work but wasn't very successful.

Here is the LDAP section in /var/lib/jenkins/config.xml:
 <securityRealm class="" plugin="ldap@1.11">
<groupSearchBase>ou=groups</groupSearchBase> <groupMembershipStrategy class=""> <filter>member={0}</filter> </groupMembershipStrategy>
   <userIdStrategy class="jenkins.model.IdStrategy$CaseInsensitive"/>
   <groupIdStrategy class="jenkins.model.IdStrategy$CaseInsensitive"/>


At this point, I was able to create users on the LDAP server and have them log in to Jenkins. With CloudBees Jenkins Enterprise, I was also able to use the Role-Based Access Control and Folder plugins in order to create project-specific folders and folder-specific groups specifying various roles. For example, a folder MyProjectNumber1 would have a Developers group defined inside it, as well as an Administrators group and a Readers group. These groups would be associated with fine-grained roles that only allow certain Jenkins operations for each group.

I tried to have these groups read by Jenkins from the LDAP server, but was unsuccessful. Instead, I had to populate the folder-specific groups in Jenkins with user names that were at least still defined in LDAP.  So that was half a win. Still waiting to see if I can define the groups in LDAP, but for now this is a workaround that works for me.

Allowing users to change their LDAP password

This was again a seemingly easy task but turned out to be pretty complicated. I set up another small EC2 instance to act as a jumpbox for users who want to change their LDAP password.

The jumpbox is in the same VPC and subnet as the LDAP server, so I added an /etc/hosts entry on the jumpbox pointing to the FQDN of the LDAP server so it can hit its internal IP address:


I verified that port 636 (used by LDAPS) on the LDAP server is reachable from the jumpbox:

# telnet 636
Connected to
Escape character is '^]'.

# apt-get install ldap-utils

IMPORTANT: Copy over /etc/ssl/certs/ca_server.pem from LDAP server as /etc/ldap/ca_certs.pem on the jumpbox and then:

# vi /etc/ldap/ldap.conf
TLS_CACERT /etc/ldap/ca_certs.pem

Next, I followed this LDAP Client Authentication guide from the Ubuntu documentation.

# apt-get install ldap-auth-client nscd

Here I had to answer the setup questions on LDAP server FQDN, admin DN and password, and bind user DN and password. 

# auth-client-config -t nss -p lac_ldap

I edited /etc/auth-client-config/profile.d/ldap-auth-config and set:

nss_passwd=passwd: ldap files
nss_group=group: ldap files
nss_shadow=shadow: ldap files
nss_netgroup=netgroup: nis

I edited /etc/ldap.conf and made sure the following entries were there:

base dc=mycompany,dc=com
uri ldaps://
binddn cn=binduser,mycompany,dc=com
rootbinddn cn=admin,mycompany,dc=com
port 636
ssl on
tls_cacertfile /etc/ldap/ca_certs.pem
tls_cacertdir /etc/ssl/certs

I allowed password-based ssh logins to the jumpbox by editing /etc/ssh/sshd_config and setting:

PasswordAuthentication yes

# service ssh restart

IMPORTANT: On the LDAP server, I had to allow users to change their own password by adding this ACL:

# cat set_userpassword_acl.ldif

dn: olcDatabase={1}hdb,cn=config
changetype: modify
add: olcAccess
olcAccess: {0}to attrs=userpassword by dn="cn=admin,dc=mycompany,dc=com" write by self write by anonymous auth by users none


# ldapmodify -H ldapi:// -Y EXTERNAL -f set_userpassword_acl.ldif

At this point, users were able to log in via ssh to the jumpbox using a pre-set LDAP password, and change their LDAP password by using the regular Unix 'passwd' command.

I am still fine-tuning the LDAP setup on all fronts: LDAP server, LDAP client jumpbox and Jenkis server. The setup I have so far allows me to have a single sign-on account for users to log in to Jenkins. Some of my next steps is to use the same user LDAP accounts  for authentication and access control into MySQL and other services.