From the BlogSubscribe Now

Blue/Green Deploys with Kubernetes and Amazon ELB

At Octoblu, we deploy very frequently and we’re tired of our users seeing the occasional blip when a new version is put into production.

Though we’re using Amazon Opsworks to more easily manage our infrastructure, our updates can take a while for dependencies to be installed before the service restarts – not a great experience.

Enter Kubernetes.

We knew that moving to an immutable infrastructure approach would help us deploy our apps, which range from extremely simple web services, to complex near-real-time messaging systems, quicker and easier.

Containerization is the future of app deployment, but managing and scaling a bunch of Docker instances, managing all the port mappings, is not a simple proposition.

Kubernetes simplified that part of our deployment strategy. However, we still had a problem, while Kubernetes is spinning up new versions of our docker instances, we could enter a state where old and new versions were in the mix. If we shut down the old before bringing up the new, we would also have a brief (sometimes not so brief) period of downtime.

Blue/Green Deploys

I first read about Blue/Green deploys in Martin Fowler’s excellent article BlueGreenDeployment, a simple, but powerful concept. We started to build out a way to do this in Kubernetes. After some complicated attempts, we came up with a simple idea: use Amazon ELBs as the router. Kubernetes handles the complexities of routing your request to the appropriate minion by listening to a given port on all minions, making ELB load balancing a piece of cake. Have the ELB listen on port 80 and 443, then route the request to the Kubernetes port on all minions.

Blue or Green?

The next problem was figuring out whether blue or green is currently active. Another simple idea, store a blue port and a green port as tags in the ELB and look at the current configuration of the ELB to see which one is currently live. No need to store the value somewhere that may not be accurate.

Putting it all together.

We currently use a combination of Travis CI and Amazon CodeDeploy to kick off the blue/green deploy process.

The following is part of a script that runs on our Trigger Service deploy. You can check out the code on GitHub if you want to see how it all works together.

I’ve added some annotation to help explain what is happening.


SCRIPT_DIR=`dirname $0`

export PATH=/usr/local/bin:$PATH
export AWS_DEFAULT_REGION=us-west-2

# Query ELB to get the blue port label
BLUE_PORT=`aws elb describe-tags --load-balancer-name triggers-octoblu-com | jq '.TagDescriptions[0].Tags[] | select(.Key == "blue") | .Value | tonumber'`

# Query ELB to get the green port label
GREEN_PORT=`aws elb describe-tags --load-balancer-name triggers-octoblu-com | jq '.TagDescriptions[0].Tags[] | select(.Key == "green") | .Value | tonumber'`

# Query ELB to figure out the current port
OLD_PORT=`aws elb describe-load-balancers --load-balancer-name triggers-octoblu-com | jq '.LoadBalancerDescriptions[0].ListenerDescriptions[0].Listener.InstancePort'`

# figure out if the new color is blue or green
if [ "${OLD_PORT}" == "${BLUE_PORT}" ]; then


# crazy template stuff, don't ask.
# Some people, when confronted with a problem,
# think "I know, I'll use regular expressions."
# Now they have two problems.
# -- jwz
perl -pe $REPLACE_REGEX $SCRIPT_DIR/triggers-service-blue-service.yaml.tmpl > $SCRIPT_DIR/triggers-service-blue-service.yaml
perl -pe $REPLACE_REGEX $SCRIPT_DIR/triggers-service-green-service.yaml.tmpl > $SCRIPT_DIR/triggers-service-green-service.yaml

# Always create both services
kubectl delete -f $SCRIPT_DIR/triggers-service-${NEW_COLOR}-service.yaml
kubectl create -f $SCRIPT_DIR/triggers-service-${NEW_COLOR}-service.yaml

# destroy the old version of the new color
kubectl stop rc -lname=triggers-service-${NEW_COLOR}
kubectl delete rc -lname=triggers-service-${NEW_COLOR}
kubectl delete pods -lname=triggers-service-${NEW_COLOR}
kubectl create -f $SCRIPT_DIR/triggers-service-${NEW_COLOR}-controller.yaml

# wait for Kubernetes to bring up the instances properly
while [ "$x" -lt 20 -a -z "$KUBE_STATUS" ]; do
   sleep 10
   echo "Checking kubectl status, attempt ${x}..."
   KUBE_STATUS=`kubectl get pod -o json -lname=triggers-service-${NEW_COLOR} | jq ".items[][\"triggers-service-${NEW_COLOR}\"].ready" | uniq | grep true`

if [ -z "$KUBE_STATUS" ]; then
  echo "triggers-service-${NEW_COLOR} is not ready, giving up."
  exit 1

# remove the port mappings on the ELB
aws elb delete-load-balancer-listeners --load-balancer-name triggers-octoblu-com --load-balancer-ports 80
aws elb delete-load-balancer-listeners --load-balancer-name triggers-octoblu-com --load-balancer-ports 443

# create new port mappings
aws elb create-load-balancer-listeners --load-balancer-name triggers-octoblu-com --listeners Protocol=HTTP,LoadBalancerPort=80,InstanceProtocol=HTTP,InstancePort=${NEW_PORT}
aws elb create-load-balancer-listeners --load-balancer-name triggers-octoblu-com --listeners Protocol=HTTPS,LoadBalancerPort=443,InstanceProtocol=HTTP,InstancePort=${NEW_PORT},SSLCertificateId=arn:aws:iam::822069890720:server-certificate/

# reconfigure the health check
aws elb configure-health-check --load-balancer-name triggers-octoblu-com --health-check Target=HTTP:${NEW_PORT}/healthcheck,Interval=30,Timeout=5,UnhealthyThreshold=2,HealthyThreshold=2

Oops happens!

Sometimes Peter makes a mistake. We have to quickly rollback to a prior version. If it is the off-cluster, rollback is as simple as re-mapping the ELB to forward to the old ports. Sometimes Peter tries to fix his mistake with a new deploy and now we have a real mess.

Because this happened more than once, we created oops. Oops allows us to instantly rollback to the off cluster, simply by executing oops-rollback, or quickly re-deploy a previous version oops-deploy git-commit.

We add an .oopsrc to all our apps that looks something like this:

"elb-name": "triggers-octoblu-com",
"application-name": "triggers-service",
"deployment-group": "master",
"s3-bucket": "octoblu-deploy"

oops list will show us all available deployments.

We are always looking for ways to get better results, if you have some suggestions, let us know.

Programming Philosophy

A few months back I started an internal weekly mailing list at Octoblu sharing my views on Programming Philosophy. I want to share those ideas a little more broadly and get some new perspectives.

Programming is a deeply creative and philosophical work, unfortunately we don’t share our beliefs widely enough.

You can join the Facebook Group, or I just launched a newsletter that anyone can subscribe to. You can subscribe to Programming Philosophy, or check out the Programming Philosophy Archive.

Twenty-Something Theses of Autonomy

I believe in a radically different organization than what exists in the world today. In order to build the new economy (and thus a new world) our ideas of how an organization works must be challenged (“You can’t make an omelete [sic] without nuking the existing social order“). A keystone of this “new way” is Autonomy. In order to get the best results, Freedom is essential. I have begun the process of capturing my theory in my “Twenty-Something Theses of Autonomy.” This list will evolve as I expand on each of these Theses, however, I want to begin the improvement process now by starting a discussion.

Do you see anything obvious missing? What has your experience taught you? Let’s talk.

Twenty-Something Theses of Autonomy

  • Customer Delight Cannot Exceed Worker Delight
  • Fully Engaged + Fully Present = Fully Human
  • Humans Own Outcomes
  • Creativity Seeks Free Spirits
  • Nonlinear Innovation Needs Creativity
  • Innovation Breeds Failure Breeds Innovation
  • Community Improves Results (and Expedites Failure)
  • Fear is the Org Killer
  • Telling Triples Turnover
  • Demanding Delivers Dummies
  • Teams Solve Difficult Problems
  • Autonomy Trumps Hegemony
  • Ivory Towers Are For Wizards (and Look Where That Got Saruman)
  • Only Gamblers Pick Winners
  • Diversity Wins
  • The Best Ideas are at the Market
  • Heterogenous Systems Increase Effectiveness (over time)
  • Simple is Better
  • Maximize Laziness
  • Effort is Expensive
  • Results > Effort
  • Only Results Matter
  • Adults Come to Work
  • Team = Product
  • (Team + Product)^n = Organization
  • Leaders Don’t Manage
  • Results Cover a Multitude of Sins

Gangplank on Arizona Horizon

I appeared on Arizona Horizon on PBS on Wednesday. You can watch the video here.

Hacking The Future of Humanity at TEDxLivermore

On June 8th, 2013 I spoke at TEDxLivermore about Hacking the Future of Humanity. The videos were recently published and now you can watch it for yourself.

From the Youtube Video:

Jade Meskill believes we have a choice: we can continue to condemn our children to a life of mediocrity through our apathy….or we can deliver them a future made of magnificence. Delving into the true hacker ethic, Jade poses the question: “How do you hack the system to leave our kids a magnificent future?”

The core principals of hacking: openness, sharing, decentralization of authority and free access to information, are shared from the perspective of someone who began a hackers’ life before becoming a teen. Jade also shares his core belief that the most important aspect of the hacker mentality is world improvement. According to Jade, the original definition of the word ‘hacker,’ (before it meant invasion of privacy or people taking over your computer), bespoke of people who were insanely curious about the new world of technology, of communication, and of computing power. Hackers were pioneers in a new frontier, building the basis of what fuels our modern, tech-driven world.

Jade Meskill works in the highly dynamic area of culture change, team improvement and organizational transformation. In this talk he discusses how his early education through Montessori school was a perfect match for supporting his penchant for creative exploration. He attributes the Montessori explorative learning style as a basis for his success as a serial entrepreneur and co-founder of Gangplank, the best collaborative workspace for creative people and innovative companies in Metro Phoenix. Gangplank is an unusual shared workspace that, instead of charging a fee, demands your energy, your time, and your passion for true co-creation. It is as much a set of values and cultural beliefs as it is a shared work environment, and it is growing, with venues in various cities in the US and Canada.

Learn more about Gangplank at

TEDxLivermore is organized by people from Livermore California who believe in the power of ideas to change the world. The mutual dependence of education and innovation means schools, industry and the broader community can join together to build an ecosystem where innovation + education can cross-fertilize and thrive.

In the spirit of ideas worth spreading, TEDx is a program of local, self-organized events that bring people together to share a TED-like experience. At a TEDx event, TEDTalks video and live speakers combine to spark deep discussion and connection in a small group. These local, self-organized events are branded TEDx, where x = independently organized TED event. The TED Conference provides general guidance for the TEDx program, but individual TEDx events are self-organized.* (*Subject to certain rules and regulations)

Is anyone opposed … ?

Would anyone be opposed to … ?

Are you opposed to … ?

Or my personal favorite

As long as no one is opposed …

I hear this question, under many different guises, throughout the day. I cringe every time I hear it, especially when it comes out of my own mouth. The question sets up a challenge: “Do you dare prevent me from … ?” Sometimes people will take you up on the challenge. If you are in a position of authority, many times they will not.

Why is this a bad thing?

The challenge will trigger a fight or flight response, increasing stress levels, dramatically decreasing the ability to create the best idea.

When this challenge is issued, the desired response is silence. Silence is not agreement. Silence can be a seed of bitterness and resentment that will bear fruit later.

By setting up agreement through inaction, you have also accepted the full consequences of this decision. If things go wrong, and the inevitably do, don’t expect any support from “the unopposed.” Just because they weren’t opposed, doesn’t mean they support it.

What to do

Break the cycle of opposition. When I hear the Opposed Question, I pause, and state that it is time to make a decision. Turn the question around to ask “Will you support … ?” and get a response from each individual regarding their support. You should not take action until support is unanimous, and the best ideas have been discovered.

This is going to take forever!

Getting unanimous support for taking action is extremely difficult. Luckily, there is help readily available. I have found the Decider from the Core Protocols to be an extremely helpful way to make unanimous decisions (that are biased toward results) that support the best ideas. Try it with your team!

So as long as no one is opposed, I’m going to publish this now…

My Coaching / Facilitation Toolkit

I was asked recently to document what’s I use for my toolkit when I’m at a client engagement. I’ve experimented with many different options and materials over the last year and so I’ll share with you what I’m currently using.

Before I show you the list, I want to share my philosophy that your toolkit needs to be simple, yet flexible. You never know what might happen during the course of a meeting you are facilitating, therefore you need to be prepared for anything. I’ve found that having an assortment of simple, yet highly effective basic materials gives me the flexibility I need to pull off almost anything.

Let’s begin:


Toolbox(es) – I have found using a common toolbox to be a great container for all this stuff. Bonus: it’s tons of fun to carry around and get strange looks from people. A toolbox appears very unusual to creative workers and helps encourage diversity of thought.

I use a 20″ toolbox as my main kit, and a 12.5″ toolbox as my grab-and-go kit.

Both my toolkits

Let’s begin with the small toolbox.

I use this one to facilitate meetings for up to 8 people, like simple retrospectives, or other meetings where I don’t need the full capabilities I have available to me.

12.5" toolkit (link is to a similar, but yellower toolbox)


12.5" toolkit full of stuff


Here are it’s contents:

Dry erase markers in both standard (Red, Green, Blue, Black) and other pastel colors. I walk into unknown meeting situations all the time. Are there markers? Do they still work? How badly damaged are they? Keeping my own set with me ensures I won't be scrambling.

Fine Point Sharpies

Planning Poker Cards

1" Painters Tape (sticks well enough without taking paint off the walls) Good for taping up papers, making grids, timelines

3x3 Post it notes - your most vital creative tool

Poker Chips - Used for coin games, placeholders, other creative ideas. I use these a lot to play simple coin puzzles while waiting for everyone to show up to meetings.

3x5 index cards (blank) - a most essential tool

Drafting Dots - useful for sticking and moving things on a whiteboard or on the wall without tearing the paper when removed

6 sided dice - just for fun and games


The Large Toolbox – I use this for more complex meetings, usually involving some highly creative activities. Enough for 20 people.

Large 20" Toolbox

Large Toolbox - Full of Fun!

The contents:

Star Post Its - Fun, fun fun

1" Painters Tape

1 - 2 boxes of Fine Point Sharpies

Sharpies of Many Colors - Black and White only is boring.

4x6, 3x5 index cards of various kinds

Scotch Tape - Need

Stickers - Everyone loves getting a sticker for doing a good job! Oh, also dot voting.

3x3 Post it notes of many colors - keep things visually interesting

Brightly colored 3x5 index cards - Liven things up!

Pastel 4x6 index cards

Paper Clips - Build fun things, clip things together, the possibilities are endless!

Super Sticky colored Post Its

Foam Sticky Shapes - Unleash creativity

Pipe Cleaners - Build shapes, also helps people focus who need to keep their hands busy during meetings

Rubber bands - shoot people, bind things together


Bonus Material:


Pencil box with Crayola Markers, Scissors, sharpies, pens -- useful for storyboard activities and other drawing fun

Bucket of tennis balls - useful for the Ball Point Game

Misc Games - useful for estimation, taking a break, or other made up fun

Lunch – A Crucial Coaching Technique

Make time to go to lunch with the people you are serving.

I’ve learned far more about the organization, the people I’m working with, and the challenges that lie ahead by going to lunch with someone almost every single day I’m on site.

I believe that lunch is one of the least respected and valued times in American culture. It is a critical time of day to step back and release from the focus of our daily work. It is an ideal opportunity to restore sustenance not merely to our bodies, but our minds as well. I’ve forged or solidified many relationships over a simple lunch. The more you can talk, laugh, cry, and share life with those around you, the more human you become, the more influence you will be permitted to have.

Lunch is a powerfully simple tool to leave behind lasting change.

P.S. This is doubly important for those who are in a consultant role. There are many barriers to overcome when you are from the outside, especially with those who do not understand what you are doing here. Lunch is that safe opportunity to share a meal to reveal yourself as the amazing person you are. (without making it inappropriate, formal, or just plain weird, like a dinner with someone unfamiliar can be)

Agile Portfolio Management Game

Lately I have been doing a lot of work with organizations on Portfolio Management. I needed to create a fun game to give an introduction to the challenges of an Agile Portfolio without being overwhelming.

In April 2011, I attended the Agile Game Incubator at Agile 2011 and utilized the PLAID technique that Mike and Don taught us. This framework was extremely helpful for focusing me on my objectives and building a solid game. If you haven’t checked out and used some of the amazing games, you must check it out.

My goal was to create something meaningful, yet playful, that had a team building element included. I created some fun personas to the game to let people role play and laugh a lot. I’ve used this game successfully with clients and also with my fellow coaches at the Scrum Coaches Retreat 2011.

Here is an introduction to the game that I created and you can find a link to download the facilitator’s kit at the bottom of this post. If you have any improvements or suggestions, please comment.


The management team of Zebco, Inc. has too many projects, too few resources. This game teaches teams how to prioritize the people and resources they have available and try to maximize their effectiveness.

Some randomness is introduced by the dice roll. This helps simulate reality when unexpected things happen and makes the game more fun.

This game is sized for a team of 3-7 people. For larger groups, split them into smaller teams of 3-7 people and provide them with extra copies of all materials.

Lead Objectives

  • Learn how delicate the balance of projects can be
  • Learn how to manage limited resources
  • Learn to deal with conflict

How Not To Have A Fierce Conversation

I have been reading the book Fierce Conversations. It has helped me to have a much more direct approach to difficult conversations, especially the ones that you really want to put off because they are so uncomfortable.

Recently at Integrum we have been undergoing a lot of deep, radical change, more about that in a later post. This has required me to have some very honest conversations with my entire team, much of it unpleasant. I don’t like confrontation; I don’t like to tell people that they need to move on (as in: time to work somewhere else). I’ve had to do that many times in the last few months. Reading this book has completely changed my approach. I don’t have it down to a science yet, and it’s not my first inclination, but I’m working hard to improve myself and my communication style.

Interestingly enough, today I got a letter in the mail: no return address, unfamiliar handwriting on the front. Anthrax jokes ensued. What was inside was almost as dangerous.

Jade everyone really likes and respects you. No one is denying you have a great business model with Gangplank and Integrum. But lately a day doesn’t go by without hearing something negative you have said about someone else. It seems your ego is getting in the way of your business. It hardly seems that the founder of Gangplank should be bad mouthing people constantly. We know you are smart, but that doesn’t mean everyone else is stupid. The foundation you are seeking to build is exactly the foundation you are destroying.

After the initial shock wore off, I started thinking… Wow, thank you anonymous person (that’s sarcasm by the way, one of my greatest gifts (that’s double sarcasm, Inception sarcasm)). It’s great that you feel that way, but now what? How do I ask questions? How do I get real feedback? Am I really doing this? This kind of Anonymous complaining is not helpful at all. It saddens me that whoever wrote this thought they were being courageous by sending this letter, but in reality, revealed the opposite. Engaging in a direct conversation addressing your issues with someone is one of the bravest things you can ever do.

Derek Neighbors recently wrote a post called You Can’t Handle The Truth, where he talks about dealing with people’s criticism of you. He states you have two choices Ignore the Criticism, or Fix the Problem. The challenge with Anonymous feedback is it’s impossible to know what to do… you can only look inside your own mind. It’s very difficult to be completely honest with yourself. How can you not be influenced by your own perceptions?

No man was ever so much deceived by another as by himself.
– Fulke Greville

So how do I deal with something like this? The honest truth is I don’t let things like this bother me too much. I try every day to be the best person I know how to be. Somedays I am a failure, but the path to being human is about the grace we extend to each other every new day, and challenging each other to rise to new heights. We are losing this skill as a race; too afraid to challenge, too afraid to speak up, too afraid to sign your own name. That’s why I’m publishing this here, radical transparency is scary, I really don’t want everyone to know that someone thinks these kinds of things about me. What will my mother say?

Here’s to pushing myself and to a better tomorrow!

The human race is a race of cowards; and I am not only marching in that procession but carrying a banner.
– Mark Twain