May 5, 2015

Using ceph-ansible to deploy and test a multi-node ceph cluster

Our team is presently working on building a Ceph Block Volume Plugin for Kubernetes. As such, I wanted a quick and easy way for everyone to be able to deploy a local Ceph Cluster in Virtual Machines so we can test the plugins ability to provision a Ceph Block Device and mount it to a given Docker Host (or Kubernetes Node) from a development environment or another virtual machine.

After spending a few days trying to find the most convenient solution, I settled on ceph-ansible as it uses a combination of vagrant (to provision the VMs) and ansible (to configure them) and the entire cluster is launched with literally one command (vagrant up). So here's how it works:

On the Developer's Machine:

1) Install Vagrant and install your Vagrant compatible Hypervisor of choice (I use VirtualBox as I find it has the broadest vagrant box support and I can't use KVM because I am on a Mac)

2) Install Ansible

3) Clone the ceph-ansible repository
  # git clone
  # cd ceph-ansible

4) Edit the  ceph-ansible/roles/ceph-common/defaults/main.yml and set the following values to “false" in the CEPH CONFIGURATION section.
  cephx_require_signatures: false
  cephx_cluster_require_signatures: false
  cephx_service_require_signatures: false

5) Deploy the Ceph Cluster
  # vagrant up

6) Check the Status of the Ceph Cluster you just deployed
  # vagrant ssh mon0 -c "sudo ceph -s"

7) Copy the ceph configuration file and ceph keyring to each server you plan to mount ceph block devices onto (such as a Fedora 21 server demonstrated in the diagram above).
  # vagrant ssh mon0
  # cd /etc/ceph/
  # sudo scp ceph.client.admin.keyring ceph.conf root@{IP of Fedora VM}:/etc/ceph/

Configuring the Ceph Client (Fedora 21 VM)

This section assumes that you have already provisioned another server to use as the means to create, format and mount a ceph block device onto. In the diagram above, this is the Fedora 21 VM.

1) Install the ceph client libraries
  # yum -y install ceph-common

2) Create this directory or you will see exceptions when using rbd commands
  # mkdir /var/run/ceph/

3) Disable and Stop firewalld
  # systemctl disable firewalld;  systemctl stop firewalld

4) Create a block device called "mydisk"
  # rbd create mydisk --size 4096

5) Map the block device from the server into your local block device list
  # rbd map mydisk --pool rbd --name client.admin

6) Verify that a new block device (rbd0) has been added 
  # ls -l /dev/rbd?
  brw-rw----. 1 root disk 252,  0 May  5 15:34 /dev/rbd0

7) Format the Block Device
  # sudo mkfs.ext4 -m0 /dev/rbd0 

8) Mount the Block Device for use
  # mkdir /mnt/mydisk
  # mount /dev/rbd0 /mnt/mydisk/

Feb 19, 2015

Enabling Docker Volumes and Kubernetes Pods to use NFS

Docker volumes allow you to mount a directory from your Host onto a path within your container. This is great if the data or path that you want to provide to your container is on your host, but what if what you really wanted to do is provide access to your container to a centralized storage location?

In my previous post, I described how one could create a network FUSE mount of a Distributed FileSystem onto the Host of your Docker Container or Kubernetes Pod. This provides a local directory to access your distributed filesystem. One then passes in that local directory as a Docker Volume or a HostDir in your Kubernetes Pod file and your containers have access to a Distributed FileSystem.

I spent some time today getting NFS (which is another common central storage scenario) working using the same Host mount model and I thought I'd share the process I used in case there were other folks interested in setting it up. The diagram below provides an overview of the configuration and the instructions follow after it.

Setting up the NFS Server (Fedora 21)

- Designate the server that that will be storing your data as the NFS Server, such as nfs-1.fed. Identify a path on that server that will be storing the data, such as /opt/data. In this example we'll later serve a file from this location called helloworld.html

- Install the required NFS Packages
# yum -y install nfs-utils

- Create the /etc/exports file which specifies which path is being shared over NFS
# vi /etc/exports
/opt/data *(rw,sync,no_root_squash)

- Start the appropriate services
# systemctl start rpcbind 
# systemctl start nfs-server 
# systemctl enable rpcbind 
# systemctl enable nfs-server

- Flush iptables
# iptables -F

Setting up the Docker Host (Fedora 21)

- Create the directory upon which to mount the NFS Share
# mkdir /mnt/nfs

- Flush iptables
# iptables -F

- Mount the NFS Share onto the local directory
# mount -t nfs nfs-1.fed:/opt/data /mnt/nfs 

- Run a directory listing and make sure that the local NFS mount is working
# ls -l /mnt/nfs

Using the NFS Share with Docker

- Launch a container and pass in the local NFS mount as the Docker volume

# docker run -v /mnt/nfs:/var/www/html/ php:5.6-apache
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using Set the 'ServerName' directive globally to suppress this message
[Thu Feb 19 00:06:11.833478 2015] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.10 (Debian) PHP/5.6.5 configured -- resuming normal operations
[Thu Feb 19 00:06:11.833601 2015] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND' - - [19/Feb/2015:00:06:31 +0000] "GET /hello.html HTTP/1.1" 200 282 "-" "curl/7.37.0" - - [19/Feb/2015:00:51:52 +0000] "GET /hello.html HTTP/1.1" 200 282 "-" "curl/7.37.0"

- Verifying that the volume mount worked
# curl
Hello World, this is being served by the NFS Share

Feb 9, 2015

Building Distributed Containerized Applications using Kubernetes and GlusterFS

This post is a follow-on from Building a Simple LAMP Application using Docker

The Kubernetes project originated from Google and is an Apache Licensed platform for managing clustered containerized applications. While the project page provides a lot more detail, in the interest of time, I thought I'd provide a quick summary. Kubernetes is a distributed Master/Worker architecture and calls its workers "Minions". Containerized applications are described and deployed by a Kubernetes Pod. A Pod typically contains one or more containers that are generally intended to reside on the same Minion. Kubernetes makes it easy to ensure that a certain amount of Pod Replicas exist via a runtime component called the ReplicationController. Lastly, Kubernetes ships with its own Load Balancer called a "Service" and this will round robin requests between the POD Replicas that are running in the cluster. 

Lets walk through an actual use case (explicit instructions follow later). If I had a PHP Docker Image and I wanted to deploy it in my Kubernetes Cluster, I would write and submit a JSON or YAML Pod file that describes the intended deployment configuration of my PHP Container. I would then write and submit a JSON or YAML ReplicationController that specifies that I want exactly 2 PHP Pods running at one time and then I would finish by writing and submitting a JSON or YAML Service file that species how I want my PHP POD Replicas load balanced. This use case is demonstrated in the diagram below. Note that on Minion 3 the PHP Pods are not running because I specified in the ReplicationController that I only want 2 PHP Pod Replicas running.

As you may have noticed, this really is a simple architecture. Now that we've covered how to deploy containerized runtimes, lets take a look at what options are available within Kubernetes to gives PODs access to data. At present, the following options are provided:

- The ephemeral storage capacity that is available within the container when it is launched

- EmptyDir, which is temporary scratch space for a container that is provided by the Host

- HostDir, which is a Host directory that you can mount onto a directory in the container.

- GCEPersistentDisk, which are block devices that are made available by the Google Compute Engine Block Service.

Given that using GCE Block Devices is really only something that you would consider if you were running in GCE, this only really leaves the HostDir option as a means to obtain durability for any kind of Kubernetes persistence on premise.

To explore how the HostDir option might be used, lets assume that you want to build the same load balanced, clustered PHP use case. One approach would be to copy the web content you want each PHP container to serve to the same local directory (/data) on each and every single Kubernetes Minion. One would then specify that directory as the HostDir parameter that is mounted onto /var/www/html in the container. This works well, but it swiftly becomes operationally onerous when you have to make updates to the web content as you now have to copy it out to every single minion in the cluster. In this scenario, it would be much easier if you could store the web content in one central place in the cluster and then provide a mount of that central place as the HostDir parameter.

One way to do this is to store the web content in a distributed file system and then mount the distributed file system onto each Minion. To demonstrate this example, we are going to use GlusterFS, which is a POSIX Compliant Distributed Filesystem. This means it looks just like your local ext4 or XFS filesystem to the applications that are using it. The diagram below displays how each Kubernetes Minion (and therefore Docker Host) has its own FUSE mount to the GlusterFS Distributed File System.

Great, so how do I do this?

1) Firstly, you're going to need a GlusterFS volume. If you don’t have one you can build one reasonably quickly by using this vagrant recipe from Jay Vyas.

2) You're going to need a working Kubernetes cluster. If you don't have one, you can follow this tutorial for how to set one up in Fedora.

3) On each Minion within the cluster, create a FUSE mount of the GlusterFS volume by running the following commands (This assumes gluster-1.rhs is the FQDN of a server in your Gluster Storage Pool provisioned by Vagrant and that your volume is called MyVolume):

# mkdir -p /mnt/glusterfs/
# yum install -y glusterfs-fuse
# mount -t glusterfs gluster-1.rhs:/MyVolume /mnt/glusterfs  

4) On a Kubernetes Minion, copy your web content to /mnt/glusterfs/php/ in the Distributed File System. If you just want a simple test you can create a helloworld.html in /mnt/glusterfs/php. Then shell into a different Kubernetes Minion and validate that you can see the file(s) you just created in /mnt/glusterfs/php. If you can see these files, it means that your Gluster volume is properly mounted.

5) Build and Submit a ReplicationController that produces a 2 Node PHP Farm that serves content from the distributed FileSystem by running the following commands on your Kubernetes Master:

# wget
# kubectl create -f php_controller_with_fuse_volume.json

6) Build and Submit the Load Balancer Service for the PHP Farm

# wget
# kubectl create -f php_service.json

7) Query the available services to obtain the IP that the Load Balancer Service is running on and submit a web request to test your setup.

# kubetctl get services
NAME                LABELS                                    SELECTOR            IP                  PORT
kubernetes          component=apiserver,provider=kubernetes   <none>           443
kubernetes-ro       component=apiserver,provider=kubernetes   <none>          80
php-master          name=php-master                           name=php_controller         80

# curl

Thanks to Bradley Childs, Huamin Chen, Mark Turansky, Jay Vyas and Tim St. Clair for their help in putting this solution together.

Jan 20, 2015

Building a simple LAMP Application using Docker

If you're looking to do this in a cluster with multiple instances, please check out this blog post too

As part of our team's effort to familiarize ourselves with Docker we wanted to start with a simple and well understood use case. To do so, we decided to build a small LAMP application on Fedora 21 Server (not F21 Workstation and not F21 Cloud). This helped us get to grips with the platform and in the event that it might help others, I thought it was worth sharing. Thanks to Erin Boyd, Jeff Vance, Scott Creeley and Brad Childs for their help in putting this together.

The application can be built in 4 relatively simple steps:

1) Set up your Docker Host environment
2) Launch a PHP Container
3) Launch a MySQL Container and create a Database
4) Write a simple PHP app that queries the database.

1. Setting up your environment

This tutorial offers a straightforward guide to setting up Fedora 21(F21) as your Docker Host. 

2. Configuring and launching the PHP Container

In order to serve the PHP page, we'll need a docker image that provides a PHP enabled Web Server. We can get this image from the docker hub by simply typing "docker pull php". You can then type "docker images" and you should now see the php image in your list of local images. 

One of the first things you need to understand about Docker is the relationship between containers and your data. The best practice is to store the data outside of your container and then mount the external store onto a directory within the container. This model allows you to have the PHP runtime inside a container but the actual PHP pages being served from an external location, such as a directory on the Host (F21) filesystem. 

To do this, you must select an EXISTING directory on F21 to host the PHP pages. In our case, we selected /opt/php/. Then you can simply launch a PHP container from the image and mount /opt/php onto /var/www/html using the -v parameter as follows: 

docker run php:5.6-apache --name RandomAppName -v /opt/php:/var/www/html/

The next step is to create a simple php files in /opt/php and verify that the page is being served by the container. When you launched the container, Docker will return a ContainerID. In order to obtain the IP address of your PHP container, run: 

docker inspect [ContainerID] | grep "IPAddress"

Next, launch a browser in F21 and use the IP of the PHP container in the URL, for example -

You should now be serving webpages! However, there's one more wrinkle. Unfortunately, the PHP image we're using does not contain any MySQL client libraries. To resolve this, we'll create a layered image that extends the PHP image that includes the MySQL client libraries. 

1. Stop the old container: 
docker stop [ContainerID]
2. Create a new directory, such as /opt/mynewphpimage
3. Create a new file in the directory called "Dockerfile" and add the contents below:
FROM php:5.6-apache
RUN docker-php-ext-install mysqli
4. Generate the image called "php-mysql" using the following command:
docker build -t php-mysql .

Finally, launch a new container with your new image that is mounted to the :
docker run -it --rm --name my-apache-php-app -v /opt/php:/var/www/html/ php-mysql

3. Configuring and launching the MySQL Container

Now that we have our PHP container all set up. The next step is to setup the MySQL container which means we'll need a MySQL Docker image. I used the tutum/mysql image and you can install it by running "docker pull tutum/mysql". You can then run "docker images" on your F21 Host and you should now see the tutum/mysql image in your list of local images.

MySQL will be set up the same way as the PHP Container was in that the actual database files will be persisted to /opt/mysql on the Fedora 21 Host and the container will only contain the MySQL runtime. This blog post describes how to do that, as well as create a small database using the container. You can verify the database files are in /opt/mysql after it has been created.

Lastly, you'll need to ensure that you create a user in MySQL that has access to the appropriate tables and database that you created, so that the PHP MySQLi client can use it to connect to them. I did this using the mysql command within the MySQL container as demonstrated below:

1. docker exec -it [ContainerIDOfMySQLContainer] bin/bash
2. mysql
3. mysql> CREATE USER 'myUserName'@'%' IDENTIFIED BY 'myPassword';

4. Write a simple PHP App that queries the database

Now that the runtimes are ready, we're going to write our application. To do so:

1. Create a "/opt/php/db.php" file on your F21 Host and place the following contents within it.

<html><head><title>PHP Test</title> </head> <body>

 $mysqli = new mysqli("", "myUserName", "myPassword", "us_states");
 if ($mysqli->connect_errno) {
    echo "Failed to connect to MySQL: (" . $mysqli->connect_errno . ") " . $mysqli->connect_error;

if ($result = $mysqli->query("SELECT * from states")) {

   while ($row = $result->fetch_assoc()) {
    echo " State: ".$row['state'];
    echo " Population: ".$row['population'];


} else {
echo "No result";

2. Launch a browser on your F21 Host and modify the URL to reflect your db.php. The PHP page should now reflect what is displayed below.

Jan 19, 2015

Creating and Persisting Databases outside your container using the Tutum/MySQL Docker Image

I'm presently hacking on Docker and I ran into a few issues getting the Tutum/MySQL Docker Image working on Fedora 21. This overview assumes that you've already gone through the basics to get Docker running on F21.

The issue I'm having is that I want to launch the MySQL container but persist the actual data (i.e. the System and Application Database) elsewhere. To do this we have to pass in a -v argument to the Docker run statement, which mounts your alternative host path to the /var/lib/mysql location in the container.

In order to do this one runs the following command to mount /opt/mysql on my F21 host server to /var/lib/mysql (which is where MySQL DBs live) in the container:

[root@localhost mysql]# docker run -d -v /opt/mysql:/var/lib/mysql tutum/mysql /bin/bash -c "/usr/bin/mysql_install_db"

However, if you check the logs, you'll notice that the attempt failed:

[root@localhost mysql]# docker logs 3224ede26db55241b320655da14bbf2737212fb3889b871ff5aa03307edf9ac8
chown: changing ownership of '/var/lib/mysql': Permission denied
Cannot change ownership of the database directories to the 'mysql'
user.  Check that you have the necessary permissions and try again.

To resolve this issue, you need to attach a --privileged directive to the docker run command as follows:

[root@localhost opt]# docker run --privileged -d -v /opt/mysql:/var/lib/mysql tutum/mysql /bin/bash -c "/usr/bin/mysql_install_db"

This will run the appropriate script in the container to create the system database and store it on your host machine and then return and release the container.  The next step is to relaunch the image with it mounted to the same system directory

[root@localhost opt]# docker run --privileged -d -p 3306:3306 -v /opt/mysql:/var/lib/mysql tutum/mysql

This should work. You can shell into the container and run the necessary SQL scripts or MySQL Admin commands to create any additional databases you wish. All Databases and their modifications will always be persisted outside of the container to the mounted directory that was specified.  For example:

[root@localhost opt]# docker exec -it aa9b0ddf6049 bash
root@aa9b0ddf6049:/# mysql

USE us_states;
INSERT INTO states (id, state, population) VALUES (NULL, ‘Alabama’, ‘4822023’);

root@aa9b0ddf6049:/# exit

[root@localhost opt]# docker stop [containerID]

In some cases I have got an exception when trying to start the MySQL docker image that states:

Cannot start container [container id]: (exit status 1)  

I find that I can get resolve this by simply restarting the docker service and trying again:

systemctl docker restart

Jan 28, 2013

Leveraging CapMetro Rail for SXSW Hotels and Commutes

It's inevitable that each year, all the hotels sell out within a certain radius of the SXSW location (the Austin Convention center). If you run into this problem, here's a little hack that might save the day.

Austin recently added a nice 2 car light rail option known as the Capital Metro Rail. This just so happens to stop right outside the Austin Convention Center and also has a greatly extended timetable during SXSW. There are nice (Westin) and decent (La Quinta) hotels within walking distance at several of the stops on the line.

This is a list of the stops on the line. Punch the address for a station into Google Maps and search nearby for hotels. The example I've provided below, using the Kramer station address, stops right next to The Domain complex which has several hotels within walking distance.

View Larger Map

Aug 14, 2012

Building an Iron Man Suit - Part 2

Given that I've had a ton of hits on the rather unsubstantive original post, I thought I'd post an update on what I've learned so far to save other folks the time I've spent figuring out how to approach this. Note: This post provides all the resources you need to build a full replica fiberglass Mark III Iron Man suit

As I mentioned in the previous post, there is an existing Iron Man Suit Builders community called Stark Industries Weapons, Data and Armor Technology (SIWDAT).  The site is largely driven by a gentleman named TMP (Timeless Movie Prop) who is a professional sculptor and prop builder. Its a great site and has a good forum where one can see some of the existing suits folks have built, but unfortunately has lost a lot of its content due to some forum crash in 2011. Users are slowly re-adding the content. I personally didn't find it super useful in how to build suits other than discovering what people use for full size blueprints for the various Iron Man Suits (War Machine, Mark I, Mark III, etc.), which is called Pepakura.

Pepakura is essentially a 3D Model comprised of a number of parts, whereby you can print out the parts and it tells you how to attach the tabs of the various parts together to recreate the 3D model. The cliff note version is that you print the templates on cardstock paper, Xacto Knife the templates out, glue the tabs on the templates together to build the model, resin the model, glue on fiberglass cloth, resin the cloth, apply a finishing product, sand it and paint it and you end up with amazing replica Iron Man Suits like the one to the right.

Pepkakura is actually pretty simple. It uses free software and you just need to find the templates, which are PDO files, and figure out what to do with them. If you watch all seven of the pepakura tutorials below, you'll learn everything you need to see how Pepakura is used to build an Iron Man Suit from start to finish. The link to the next tutorial appears at the end of each video. Plus, Stealth, their creator is pretty funny.
Another gentleman that goes by the handle Dancin_Fools, has one of the most popular and detailed Mark III pepakura suit designs built using 3D Studio Max.This is the link to the thread where you can see some of the models and the outcome.  You can download the templates (PDO) files directly here.

I'm actually attempting to build an aluminum suit since I'm having some fun exploring how far my son and I can get trying to build a real suit and not a costume. Aluminum is a much harder medium to work with than paper, so at this point, I've decided to first go/think through the process with paper to make sure I have the scale correctly identified. I'm currently working on the chest piece from the PDO files provided above.

Do not understimate the huge amount of time it takes to cut the pepakura templates out using the xacto knife, it is taking me days (in my spare time) to cut out just the chest peices. Someone should start a business selling pre-cut templates at a scale specified by the customer.

I was pretty encouraged to find that other people have actually built Aluminum suits. I have decided that Welding is going to really complicate things, so my current plan has changed to using steel rivets to join the pieces. Check out this really cool fully aluminum Mark VI suit below. The builders thread for this is available here. Granted, there's not a whole lot out there that I've found (yet) on how to build aluminum suits. I'll post more as I find it.

UPDATE: Samurai169 over at RPF has an awesome welded 20 gauge steel suit he is building.


The next step is to actually build your Arc Reactor replica. I've at least got this bit accomplished already. To begin with, here's a refresher of what an actual Mark I Arc Reactor looks like:

Now, unless you've got a 3 inch hole in your solar plexus, or you've already made the suit and it sits an inch or two off of your chest, you're not going to be able to wear a true replica. Instructables has a great section on how to build your own Iron Man Arc Reactor. I got all that I needed from Radioshack, Hobby Lobby and Lowes. The images below are some examples of Arc Reactors that can be built.


Lastly, if you want to start wiring the suit for functionality such as automating the face plate opening and closing, then the XRobots web site has a lot of tutorials and likely what you're looking for.

Also, dont forget about the ability to use the Arduino platform, as it provides a platform to start building out the suits motorized functions.

Another option is to use a 3D Printer as it possible to print some of the components in their entirety. For example, MakerBot has a 3D Printer that can print components the size of a loaf of bread. Some folks are using it to print entire helmets.

Happy Making !