0 Comments Posted in:

This is the third part of a series looking at how easy Docker makes it to explore and experiment with open source software. The previous two parts are available here

Today we're going to look at Elasticsearch, and this will give us the chance to see some of the capabilities of Docker Compose.

To follow along with the commands in this tutorial I recommend that you use Play with Docker which allows you to run all these commands in the browser.

Start a new container running Elasticsearch

If you just want to try out Elasticsearch running in a single node, then we can do that with the docker run command shown below.

We're exposing ports 9200 (for the REST API), and setting up a single node cluster (using an environment variable), from the official elasticsearch 6.4.2 image. I'm also showing how to set up a volume to store the index data in.

docker run -d -p 9200:9200 -e "discovery.type=single-node" \
-v esdata:/usr/share/elasticsearch/data \
docker.elastic.co/elasticsearch/elasticsearch:6.4.2

And all the Elasticsearch commands we run with curl will work just fine on this single container. But for this tutorial, I'm going to use a cluster created with docker-compose instead.

Use Docker Compose to create an Elasticsearch cluster

With docker-compose we can declare all the containers that make up an application in a YAML format. For each container we can also configure the environment variables that should be set, any volumes that are required, and define a network to allow the services to communicate with each other.

Here's the first version of our docker-compose.yml file. It defines a simple two-node cluster, and each node in the cluster has a volume so that our indexes can live independently of our containers, and survive upgrades (which we'll be doing later). Notice that we're using the version of elasticsearch tagged 6.4.1.

version: '2.2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.1
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet
  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.1
    container_name: elasticsearch2
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata2:/usr/share/elasticsearch/data
    networks:
      - esnet

volumes:
  esdata1:
    driver: local
  esdata2:
    driver: local

networks:
  esnet:

To download this file locally as docker-compose-v1.yml you can use the following command:

curl https://gist.githubusercontent.com/markheath/f246ec3aa5a3e7493991904e241a416a/raw/c4fa64575bc854e34a2506291bd14033caf5e9b6/docker-compose-v1.yml > docker-compose-v1.yml

And now we can use the docker-compose up command to start up the containers, and create all necessary resources like networks and volumes. We're using -d to run in the background just like we can with docker run

docker-compose -f docker-compose-v1.yml up -d

Check cluster health

We've exposed port 9200 on one of those containers, allowing us to query the cluster health with the following request:

curl http://localhost:9200/_cluster/health?pretty

Create an index

Now let's create an index called customer

curl -X PUT "localhost:9200/customer?pretty"

Add a new document

And let's add a document to that index:

curl -X PUT "localhost:9200/customer/_doc/1?pretty" \
-H 'Content-Type: application/json' -d'{"name": "Mark Heath" }'

By the way, if you're following along with PowerShell instead of bash you can use Invoke-RestMethod to accomplish the same thing.

Invoke-RestMethod -Method Put `
-Uri "http://localhost:9200/customer/_doc/1?pretty" `
-ContentType "application/json" -Body @'{"name": "Mark Heath" }'@

View documents in the index

There are lots of ways to query elasticsearch indexes and I recommend you check out the Elasticsearch 6.4 Getting Started Guide for more details. However, we can easily retrieve the documents in our existing customer index with:

curl localhost:9200/customer/_search?pretty

Upgrade the cluster to 6.4.2

Suppose we now want to upgrade the nodes in our cluster from Elasticsearch 6.4.2 (we were previously running 6.4.1). What we can do is update our YAML file with new container version numbers.

I have an updated YAML file available here, which you can download to use locally with

curl https://gist.githubusercontent.com/markheath/f246ec3aa5a3e7493991904e241a416a/raw/c4fa64575bc854e34a2506291bd14033caf5e9b6/docker-compose-v2.yml > docker-compose-v2.yml

Before we upgrade our cluster, take a look at the container ids that are currently running with docker ps. These containers are not going to be "upgraded" - they're going to be disposed, and new containers running 6.4.2 will be created. However, the data is safe, because it's stored in the volumes. The volumes won't be deleted, and will be attached to the new containers.

To perform the upgrade we can use the following command.

docker-compose -f docker-compose-v2.yml up -d

We should see it saying "recreating elasticsearch" and "recreating elasticsearch2" as it discards the old containers and creates new ones.

Now if we run docker ps again we'll see new container ids and new image versions.

Check our index is still present

To ensure that our index is still present we can search again and check our document is still present.

curl localhost:9200/customer/_search?pretty

Let's add another document into the index with:

curl -X PUT "localhost:9200/customer/_doc/2?pretty" -H 'Content-Type: application/json' -d'{"name": "Steph Heath"}'

Upgrade to a three node cluster

OK, let's take this to the next level. I've created a third version of my docker-compose YAML file that defines a third container, with its own volume. The YAML file is available here.

Something important to note is that I needed to set the discovery.zen.minimum_master_nodes=2 environment variable to avoid split brain problems.

You can download my example file with:

curl https://gist.githubusercontent.com/markheath/f246ec3aa5a3e7493991904e241a416a/raw/a2685d1bf0414acbc684572d00cd7c7c531d0496/docker-compose-v3.yml > docker-compose-v3.yml

And then we can upgrade our cluster from two to three nodes with

docker-compose -f docker-compose-v3.yml up -d

The change of environment variable means that we will recreate both elasticsearch and elasticsearch2, and of course the new elasticsearch3 container and its volume will get created.

We should check the cluster status and if all went well, we'll see a cluster size of three:

curl http://localhost:9200/_cluster/health?pretty

Let's check our data is still intact by retrieving a document by id from our index

curl -X GET "localhost:9200/customer/_doc/1?pretty"

Add Kibana and head plugin

While I was preparing this tutorial I came across a really nice article by Ruan Bekker who takes this one step further by adding a couple more containers to the docker-compose file for an instance of Kibana and the Elasticsearch Head plugin.

So here's the final docker-compose.yaml file we'll be working with:

version: '2.2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.2
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - discovery.zen.minimum_master_nodes=2
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet

  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.2
    container_name: elasticsearch2
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - discovery.zen.minimum_master_nodes=2
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata2:/usr/share/elasticsearch/data
    networks:
      - esnet

  elasticsearch3:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.2
    container_name: elasticsearch3
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - discovery.zen.minimum_master_nodes=2
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata3:/usr/share/elasticsearch/data
    networks:
      - esnet

  kibana:
    image: 'docker.elastic.co/kibana/kibana:6.4.2'
    container_name: kibana
    environment:
      SERVER_NAME: kibana.local
      ELASTICSEARCH_URL: http://elasticsearch:9200
    ports:
      - '5601:5601'
    networks:
      - esnet

  headPlugin:
    image: 'mobz/elasticsearch-head:5'
    container_name: head
    ports:
      - '9100:9100'
    networks:
      - esnet

volumes:
  esdata1:
    driver: local
  esdata2:
    driver: local
  esdata3:
    driver: local

networks:
  esnet:

You can download my YAML file with

curl https://gist.githubusercontent.com/markheath/f246ec3aa5a3e7493991904e241a416a/raw/a2685d1bf0414acbc684572d00cd7c7c531d0496/docker-compose-v4.yml > docker-compose-v4.yml

And now we can update our cluster again with

docker-compose -f docker-compose-v4.yml up -d

Try out Kibana

Once we've done this, we can visit the Kibana site by browsing to localhost:5601. If you were following along in "Play with Docker" then you'll see special links appear for each port that is exposed (9200, 9100 and 5601).

Play with Docker

If you click on the 5601 link, you'll be taken to an instance of Kibana. The first step will be to define an index pattern (e.g. customers*)

Kibana index

And then if you visit the discover tab, you'll see we can use Kibana to search the documents in our index:

Kibana search

Try out Elasticsearch head plugin

You can also visit localhost:9100 (or in Play with Docker, click the 9100 link) to use the Elasticsearch head plugin. This gives you a visualization of the cluster health.

Elasticsearch head plugin

Note that if you are using Play with Docker, you'll need to copy the port 9200 link and paste it into the Connect textbox to connect the head plugin to your Elasticsearch cluster.

Clean up

To stop and delete all the containers:

docker-compose -f docker-compose-v4.yml down

And if you want to delete the volumes as well (so all index data will be lost), add the -v flag:

docker-compose -f docker-compose-v4.yml down -v

Summary

In this tutorial we saw that not only is it really easy to get an instance of Elasticsearch running with Docker that we could use for experimenting with the API, but with Docker Compose we can define collections of containers that can communicate with one another and start them all easily with docker-compose up.

When we upgrade our YAML file, Docker Compose can intelligently decide which containers need to be replaced, and which can be left as they are.


0 Comments Posted in:

This is the second part of a series looking at how easy Docker makes it to explore and experiment with open source software. Last time we looked at Redis, and that gave us the opportunity to see the docker run and docker exec commands in action.

Today we're going to look at PostgreSQL which will give us an opportunity to see Docker volumes in action.

You can follow along with the commands in this tutorial if you have Docker installed. If you're running Docker for Windows put it in Linux mode. But another great option is Play with Docker which lets us run all these commands in the browser.

Start a new container running PostgreSQL

We'll use docker run to start a new container from the official postgres image with the name postgres1 and exposing port 5432 (the PostgreSQL default). We're running detached (-d) mode (so in the background).

But we're also going to mount a volume (with -v), which will be used to store the database we create. The volume name will be postgres-data, and Docker will automatically create it (just using storage on the Docker host's local disk) if a volume with this name doesn't already exist.

PostgreSQL stores its data in /var/lib/postgresql/data, so we're mounting our volume to that path.

docker run -d -p 5432:5432 -v postgres-data:/var/lib/postgresql/data `
           --name postgres1 postgres

Once we've done this we can check it's running with

docker ps

And view the log output with

docker logs postgres1

Create a database

We'll create a database and one easy way to do that is by using docker exec to launch an interactive shell running inside our postgres1 container, which has the PostgreSQL CLI tools installed. This saves us from needing to have any tools to connect to and manage PostgreSQL databases installed locally.

docker exec -it postgres1 sh

Inside that shell we can ask it to create a new database with the name mydb.

# createdb -U postgres mydb

And then let's launch the psql utility which is a CLI tool for PostgreSQL, connected to our mydb database:

# psql -U postgres mydb

Explore the database

Now inside psql, let's run some basic commands. \l lists the databases. We'll also ask for the database version, and the current date:

mydb=# \l
mydb=# select version();
mydb=# select current_date;

Now let's do something a bit more interesting. We'll create a table:

mydb=# CREATE TABLE people (id int, name varchar(80));
CREATE TABLE

Then we'll insert a row into the table:

mydb=# INSERT INTO people (id,name) VALUES (1, 'Mark');
INSERT 0 1

And finally, check it's there

mydb=# SELECT * FROM people;
 id | name 
----+------
  1 | Mark
(1 row)

Now we can quit from psql with \q and exit from our shell

mydb=# \q 
# exit

Of course our postgres1 container is still running.

Stop and restart the container

Let's prove that we don't lose the data if we stop and restart the container.

docker stop postgres1
docker start postgres1

And rather than connect again to this container, let's test from another linked container, using the same technique for linking containers we saw in our Redis demo.

docker run -it --rm --link postgres1:pg --name client1 postgres sh

Launch psql but connect to the other container (-h) which we've given the name pg in our link configuration:

# psql -U postgres -h pg mydb

Now from this client1 container we can access data in the database stored in the postgres1 container:

mydb=# SELECT * FROM people;
 id | name 
----+------
  1 | Mark
(1 row)

Now we can quit from psql and exit from our shell, which will remove the client1 container since we specified the --rm flag to auto-delete the container when the command it was running exits.

mydb=# \q 
# exit

Inspect the volume

We can find out information about the volume that we've created with docker volume inspect, including where on our local disk the data in that volume is being stored. Here's some typical output.

$ docker volume inspect postgres-data
[
    {
        "CreatedAt": "2018-09-03T19:50:23Z",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/postgres-data/_data",
        "Name": "postgres-data",
        "Options": null,
        "Scope": "local"
    }
]

And if we take a look inside the local folder on our Docker host, we can see all the data that has been stored in that volume.

$ ls /var/lib/docker/volumes/postgres-data/_data/
PG_VERSION            pg_multixact          pg_tblspc
base                  pg_notify             pg_twophase
global                pg_replslot           pg_wal
pg_commit_ts          pg_serial             pg_xact
pg_dynshmem           pg_snapshots          postgresql.auto.conf
pg_hba.conf           pg_stat               postgresql.conf
pg_ident.conf         pg_stat_tmp           postmaster.opts
pg_logical            pg_subtrans           postmaster.pid

Obviously a Docker volume doesn't need to be stored on local disk on the Docker host. In a production environment like Azure, you'd most likely mount an Azure file share as a volume.

Discard the container but keep the data

Let's stop and remove the postgres1 container with a single command (-f forces it to remove a running container). Because the data is stored in a volume, that is still safe.

docker rm -f postgres1

Attach an existing volume to a new container

Let's now start up a brand new container called postgres2 but attach the existing postgres-data volume that contains our database:

docker run -d -p 5432:5432 -v postgres-data:/var/lib/postgresql/data --name postgres2 postgres

Once it starts up, let's run a psql session inside it and check that the database, table and data are still all present and correct:

docker exec -it postgres2 sh
# psql -U postgres mydb
mydb=# SELECT * FROM people;
 id | name
----+-------
  1 | Mark
(1 row)

And exit out again:

mydb=# \q
# exit

Clean up everything

And now, let's really clean up. Not only will we remove the postgres2 container, but we'll then remove the postgres-data volume. So now the contents of the database are deleted as well.

docker rm -f postgres2
docker volume rm postgres-data

Summary

As you can see, not only is it easy to use Docker to explore PostgreSQL, we can also easily configure a volume allowing the lifetime of the data to be managed independently of the lifetime of the container. If we'd wanted to, we could also have connected directly to this PostgreSQL container on port 5432 and used it for some local development.

Next up, we'll explore running Elasticsearch in a container, which will give us an opportunity to see docker-compose in action.


0 Comments Posted in:

One of the great things about Docker is how you can quickly use it to try out applications without having to install it directly on your developer machine. No need to install runtimes like Java or erlang if you don't want to - the container encapsulates all dependencies, and leaves no trace behind when you delete it.

So over the next few days I plan to provide a few short tutorials showing off how to explore various bits of open source software with Docker, starting with Redis. The focus of these tutorials is to familiarize you with the Docker commands needed to create containers and execute custom commands within them, but hopefully you'll learn a few Redis basics along the way too.

Intro

This tutorial shows how you can use Docker to explore Redis. You can run the commands with Docker installed, or Docker for Windows in Linux mode. But you can also use Play with Docker to try this out.

Start a new container running Redis

Here we're giving it a name (redis1) and exposing port 6379 (the Redis default)

docker run -d -p 6379:6379 --name redis1 redis

Check it's running with

docker ps

And view the log output with

docker logs redis1

Run the Redis CLI in the container

We're going to start a new interactive session (-it) inside the running container, and use it to run redis-cli. We could run it directly, but for now, let's just start a shell with sh:

docker exec -it redis1 sh

And now we're attached to our container. Let's run redis-cli:

# redis-cli

Try out some basic Redis commands

If we send a "ping", should get "PONG" back:

127.0.0.1:6379> ping
PONG

Try out some more commands (set a key and increment a counter)

127.0.0.1:6379> set name mark
OK
127.0.0.1:6379> get name
"mark"
127.0.0.1:6379> incr counter
(integer) 1
127.0.0.1:6379> incr counter
(integer) 2
127.0.0.1:6379> get counter
"2"

And when we're done exit out of redis-cli and sh:

127.0.0.1:6379> exit
# exit

Connect from another linked container

Now let's start another container, called client1. We'll base it off the redis image but we're only using it to run the redis-cli so we'll just ask it to run sh in interactive mode -it. This means it won't be running Redis itself. We specify --rm so it will delete itself after the shell exits.

We link it to the redis1 container (which is still running), and it will be referred to from within this container simply as redis.

docker run -it --rm --link redis1:redis --name client1 redis sh

Now in this container let's start redis-cli and connect to redis1 which has the name redis:

# redis-cli -h redis
redis:6379>

And now let's issue some commands. Since we didn't stop the original redis1 container, the keys we created earlier should still be there:

redis:6379> get name
"mark"
redis:6379> get counter
"2"

And now exit out of redis-cli and sh:

redis:6379> exit
# exit

This has actually caused our client1 container to delete itself. So even if we ask to see all containers including stopped ones with:

docker ps -a

We should only see our single Docker container.

Clean up

Let's stop the redis1 container and then delete it.

docker stop redis1
docker rm redis1

We have also downloaded the redis image, which we will see in the list

docker image ls

It's only 83MB, so we may want to keep it for next time we want to run a Redis container, but if we're sure we don't need it anymore, we can remove the image to free up disk space and get us back to exactly where we started before:

docker image rm redis

Summary

As you can see, with just a few basic Docker commands we could run Redis in a container, and even connect to it from another container.

Next up, I'll show you how we can do the same with PostgreSQL, which will give us the opportunity to explore Docker volumes.