IT 2.0 Next Generation IT Infrastructures Fri, 21 Jul 2017 11:57:00 +0000 en-US hourly 1 15181877 Yelb, yet another sample app Fri, 21 Jul 2017 11:57:00 +0000 Another pet project I have spent cycles on as of late is an open source sample application called Yelb (thanks to my partner in crime chief developer Andrea Siviero for initiating me to the mysteries of Angular2).

This is the link to the Yelb repo on GitHub.

I am trying to be fairly verbose in . . . → Read More: Yelb, yet another sample app]]> Another pet project I have spent cycles on as of late is an open source sample application called Yelb (thanks to my partner in crime chief developer Andrea Siviero for initiating me to the mysteries of Angular2).

This is the link to the Yelb repo on GitHub.

I am trying to be fairly verbose in the README files in the repo so I am not going to repeat myself here. Someone said GitHub repos are the new blog posts in the DevOps and cloud era. I couldn’t agree more.

For the records, Yelb looks like this (as of today). The skeleton of this interface was literally stolen (with permission) from a sample application the Clarity team developed.

When deployed with direct Internet access it should allow people to vote and (thanks to Angular2 and the Clarity based UI) you will see graphs dynamically changing. In addition to that, Yelb also track the number of page views as well as the application layer container hostname serving the request.

I thought this was a good mix of features to be able to demo an app to an audience while inspecting what was going on in the app architecture (e.g. watching the container name serving the request changing when multiple containers are deployed in the app server layer).

Good for using it to demo Docker at a conference, good for using it as the basis to build a new deployment YAML for the 37th container orchestration framework we will see next week.

This is the architecture of the application (as of today).

Check on the GitHub repo for more (up to date) information about Yelb.

If you are into the container space I think it helps a lot owning something that you can bring from personal development (from scratch) to production. You have got to see all the problems a dev sees by taking his/her own app into production using containers and frameworks of sort.

While you are more than welcome to use Yelb for your own demos and tests (at your own peril), I truly suggest you build your own Yelb.

Not to mention the amount of things you learn as you go through these exercises. I am going to embarrass myself here by saying I didn’t even know Angular was not server side and that I didn’t know how the mechanics of the Angular compiling process worked.  Stack Overflow is such an awesome resource when you are into these things.


]]> 0 974
Project Harbor makes an entry into Rancher! Thu, 20 Jul 2017 09:03:45 +0000 This article was originally posted on the VMware Cloud Native corporate blog. I am re-posting here for the convenience of the readers of my personal blog.

Early this year I challenged myself with a pet project to create a Rancher catalog entry for Project Harbor (a VMware-started open sourced enterprise container registry).

. . . → Read More: Project Harbor makes an entry into Rancher!]]> This article was originally posted on the VMware Cloud Native corporate blog. I am re-posting here for the convenience of the readers of my personal blog.

Early this year I challenged myself with a pet project to create a Rancher catalog entry for Project Harbor (a VMware-started open sourced enterprise container registry).

This is something that I have been working on, off and on in my spare time. I originally envisioned this to be a private catalog entry. In other words, an entry that a Rancher administrator could add as a private catalog to her/his own instance.

I am happy to report that, a few weeks ago, Rancher decided to include the Project Harbor entry into the Rancher community catalog. The community catalog is a catalog curated by Rancher but populated by the community of partners and users.

The specific entry for Project Harbor is located here.

As a Rancher administrator, you don’t have to do anything to configure it other than enabling visibility of the Rancher community catalog. Once you have that option set, every Rancher user out there can point to Project Harbor and deploy it in their Cattle environments.

This is how the view of the Rancher community catalog looks like today:

Note that, as of today, this catalog entry only works with Rancher Cattle environments (depending on community interest support could be expanded to Rancher Kubernetes and Rancher Swarm environments as well).

Originally, this catalog entry had a couple of deployment models for Harbor (standalone and distributed). The last version of this catalog entry has only one model and depending on the parameters you select Harbor will be deployed on a single Docker host in the Cattle environment or it will be distributed across the hosts in a distributed fashion.

The README of the catalog entry will explain the various options and parameters available to you.

If you are interested in understanding the genesis of this pet project and all the technical details you have to consider to build such a catalog entry for Harbor, I suggest you read the original blog post that includes lots of technical insides about this implementation (including challenges, trade-offs, and limitations). Note that, at the time of this writing, the Rancher community catalog entry for Project Harbor will instantiate the OSS version of Harbor 1.1.1.

Last but not least, mind that the Rancher community catalog is supported by the community. The Project Harbor catalog entry follows the same support pattern so, if you have any issue with this catalog entry, please file an issue on the GitHub project.


]]> 0 970
Hashidays 2017 – London: a personal report Thu, 22 Jun 2017 10:06:55 +0000 I have lately started a tradition of copying/pasting reports of events I attend for the community to be able to read them. As always, they are organized as a mix of (personal) thoughts that, as such, are always questionable …. as well as raw notes that I took during the keynotes and breakout . . . → Read More: Hashidays 2017 – London: a personal report]]> I have lately started a tradition of copying/pasting reports of events I attend for the community to be able to read them. As always, they are organized as a mix of (personal) thoughts that, as such, are always questionable …. as well as raw notes that I took during the keynotes and breakout sessions.

You can find/read previous reports at these links:

Note some of these reports have private comments meant to be internal considerations to be shared with my team. These comments are removed before posting the blog publicly and replaced with <Redacted comments>.

Have a good read. Hopefully, you will find this small “give back” to the community helpful.


Massimo Re Ferré, CNA BU – VMware

Hashidays London report.  

London – June 12th 2017

Executive summary and general comments

This was in general a good full day event. Gut feeling is that the audience was fairly technical, which mapped well the spirit of the event (and HashiCorp in general).

There had been nuggets of marketing messages spread primarily by Mitchell H (e.g. “provision, secure, connect and run any infrastructure for any application”) but these messages seemed a little bit artificial and bolted on. HashiCorp remains (to me) a very engineering focused organization where the products market themselves in an (apparently) growing and loyal community of users.

There were very few mentions of Docker and Kubernetes compared to other similar conferences. While this may be due to my personal bias (I tend to attend more containers-focused conferences as of late), I found interesting that there were more time spent talking about HashiCorp view on Serverless than containers and Docker.

The HashiCorp approach to intercept the container trend seems interesting. Nomad seems to be the product they are pushing as a counter answer for the like of Docker Swarm / Docker EE and Kubernetes. Yet Nomad seems to be a general-purpose scheduler which (almost incidentally) supports Docker containers. However, a lot of the advanced networking and storage workflows available in Kubernetes and in the Docker Swarm/EE stack aren’t apparently available in Nomad.

One of the biggest tenet of HashiCorp’s strategy is, obviously, multi-cloud. They tend to compete with some specific technologies available from specific cloud providers (that only work in said cloud) so the notion of having cloud agnostic technologies that work seamlessly across different public clouds is something they leverage (a ton).

Terraform seemed to be the special product in terms of highlights and number of sessions. Packer, Vagrant were hardly mentioned outside of the keynote with Vault, Nomad and Consul sharing almost equally the remaining of the time available.

In terms of backend services and infrastructures they tend to work with (or their customers tend to end up on) I will say that the event was 100% centered around public cloud. <Redacted comments>.

All examples, talks, demos, customers’ scenarios etc. etc. were focused on public cloud consumption. If I have to guess a share of “sentiment” I’d say AWS gets a good 70% with GCP another 20% and Azure 10%. These are not hard data, just gut feelings.

The monetization strategy for HashiCorp remains (IMO) an interesting challenge. A lot (all?) of the talks from customers were based on scenarios where they were using standard open source components. Some of them specifically proud themselves for having built everything using free open source software. There was a mention that at some point this specific customer would have bought Enterprise licenses but the way it was phrased let me think this was to be done as a give-back to HashiCorp (to which they owe a lot) rather than specific technical needs for the Enterprise version of the software.

Having that said there is no doubt HashiCorp is doing amazing things technologically and their work is super well respected.

In the next few sections there are some raw notes I took during the various speeches throughout the day.

Opening Keynote (Mitchell Hashimoto)  

The HashiCorp User Group in London is the largest (1300 people) in the world.

HashiCorp strategy is to … Provision (Vagrant, Packer, Terraform), secure (Vault), connect (Consul) and run (Nomad) any infrastructure for any application.

In the last few years, lots of Enterprise features found their way into many of the above products.

The theme for Consul has been “easing the management of Consul at scale”. The family of Autopilot features is an example of that (set of features that allows Consul to self-manage itself). Some of such features are only available in the Enterprise version of Consul.

The theme for Vault has been to broaden the feature set. Replication across data centers is one such feature (achieved via log shipping).

Nomad is being adopted by largest companies first (very different pattern compared to the other HashiCorp tools). The focus recently has been on solving some interesting problems that surface with these large organizations. One such advancement is Dispatch (HashiCorp’s interpretation of Serverless). You can now also run Spark jobs on Nomad.

The theme for Terraform has been to improve platforms support. To achieve this HashiCorp is splitting the Terraform core product from the providers (managing the community of contributors is going to be easier with this model). Terraform will download providers dynamically but they will be developed and distributed separately from the Terraform product code. In the next version, you can also version the providers and require a specific version in a specific Terraform plan. “Terraform init” will download the providers.

Mitchell brings up the example of the DigitalOcean firewall feature. They didn’t know it was coming but 6 hours after the DO announcement they did receive a PR from DO that implemented all the firewall features in the DO provider (these situations are way easier to manage when community members are contributing to provider modules if these modules are not part of the core Terraform code base).

Modern Secret Management with Vault (Jeff Mitchell) 

Vault is not just an encrypted key/value store. For example, generating and managing certificates is something that Vault is proving to be very good at.

One of the key Vault features is that it provides multiple (security related) services fronted with a single API and consistent authn/authz/audit model.

Jeff talks about the concept of “Secure Introduction” (i.e. how you enable a client/consumer with a security key in the first place). There is no one size fits all. It varies and depends on your situation, infrastructure you use, what you trust and don’t trust etc. etc. This also varies if you are using bare metal, VMs, containers, public cloud, etc. as every one of these models has its own facilities to enable “secure introduction”.

Jeff then talks about a few scenarios where you could leverage Vault to secure client to app communication, app to app communication, app to DB communications and how to encrypt databases.

Going multi-cloud with Terraform and Nomad (Paddy Foran) 

Message of the session focuses on multi-cloud. Some of the reasons to choose multi-cloud are resiliency and to consume cloud-specific features (which I read as counter-intuitive to the idea of multi-cloud?).

Terraform provisions infrastructure. Terraform is declarative, graph-based (it will sort out dependencies), predictable and API agnostic.

Nomad schedules apps on infrastructure. Nomad is declarative, scalable, predictable and infrastructure agnostic.

Paddy is showing a demo of Terraform / Nomad across AWS and GCP. Paddy explains how you can use output of the AWS plan and use them as inputs for the GCP plan and vice versa. This is useful when you need to setup VPN connections between two different clouds and you want to avoid lots of manual configurations (which may be error prone).

Paddy then customizes the standard example.nomad task to deploy on the “datacenters” he created with Terraform (on AWS and GCP). This will instantiate a Redis Docker image.

The closing remark of the session is that agnostic tools should be the foundation for multi-cloud.

Running Consul at Massive Scale (James Phillips) 

James goes through some fundamental capabilities of Consul (DNS, monitoring, K/V store, etc.).

He then talks about how they have been able to solve scaling problems using a Gossip Protocol.

It was a very good and technical session arguably targeted to existing Consul users/customers that wanted to fine tune their Consul deployments at scale.

Nomad and Next-generation Application Architectures  (Armon Adgar)

Armon starts to define the role of the scheduler (broadly).

There are a couple of roles that HashiCorp took in mind when building Nomad: developers (or Nomad consumers) and infrastructure teams (or Nomad operators).

Similarly, to Terraform, Nomad is declarative (not imperative). Nomad will know how to do things without you needing to tell it.

The goal for Nomad was never to build an end-to-end platform but rather to build a tool that would do the scheduling and bring in other HashiCorp (or third party) tools to compose a platform. This after all has always been the HashiCorp spirit of building a single tool that solves a particular problem.

Monolith applications have intrinsic application complexity. Micro-services applications have intrinsic operational complexity. Frameworks has helped with monoliths much like schedulers are helping now with micro-services.

Schedulers introduce abstractions that helps with service composition.

Armon talks about the “Dispatch” jobs in Nomad (HashiCorp’s FaaS).

Evolving Your Infrastructure with Terraform (Nicki Watt)

Nicki is the CTO @ OpenCredo.

There is no right or wrong way of doing things with Terraform. It really depends on your situation and scenario.

The first example Nicki talks about is a customer that has used Terraform to deploy infrastructure on AWS to setup Kubernetes.

She walks through the various stages of maturity that customers find themselves in. They usually start with hard coded values inside a single configuration file. Then they start using variables and applying them to parametrized configuration files.

Customers then move onto pattern where you usually have a main terraform configuration file which is composed with reusable and composable modules.

Each module should have very clearly identified inputs and outputs.

The next phase is nested modules (base modules embedded into logical modules).

The last phase is to treat subcomponents of the setup (i.e. Core Infra, RDS, K8s cluster) as totally independent modules. This way you manage these components independently hence limiting the possibility of making a change (e.g. in a variable) that can affect the entire setup.

Now that you moved to this “distributed” stage of independent components and modules, you need to orchestrate what needs to be run first etc. Different people solve this problem in different ways (from README files that guide you through what you need to manually do all the way to DIY orchestration tools going through some off-the-shelf tools such as Jenkins).

This was really an awesome session! Very practical and very down on earth!

Operational Maturity with HashiCorp (James Rasell and Iain Gray)

This is a customer talk.

Elsevier has an AWS first. They have roughly 40 development teams (each with 2-3 AWS accounts and each account has 1-6 VPCs).

Very hard to manage manually at this scale. Elsevier has established a practice inside the company to streamline and optimize this infrastructure deployments (they call this practice “operational maturity”). This is the charter of the Core Engineering team.

The “operational maturity” team has 5 pillars:

  • Infrastructure governance (base infrastructure consistency across all accounts). They have achieved this via a modular Terraform approach (essentially a catalog of company standard TF modules developers re-use).
  • Release deployment governance
  • Configuration management (everything is under source control)
  • Security governance (“AMI bakery” that produces secured AMIs and make it available to developers)
  • Health monitoring

They chose Terraform because:

– it had a low barrier to entry

– it was cloud agnostic

– codified with version control

Elsevier suggests that in the future they may want to use Terraform Enterprise. Which underlines the difficulties of monetizing open source software. They are apparently extracting a great deal of value from Terraform but HashiCorp is making 0 out of it.

Code to Current Account: a Tour of the Monzo Infrastructure (Simon Vans-Colina and Matt Heath)

Enough said. They are running (almost) entirely on free software (with the exception of a small system that allows communications among banks). I assume this implies they are not using any HashiCorp Enterprise pay-for products.

Monzo went through some technology “trial and fail” such as:

  • from Mesos to Kubernetes
  • from RabbitMQ to Linkerd
  • from AWS Cloud Formation to Terraform

They, right now, have roughly 250 services. They all communicate with each other over http.

They use Linkerd for inter-services communication. Matt suggests that Linkerd integrates with Consul (if you use Consul).

They found they had to integrate with some banking systems (e.g. faster payments) via on-prem infrastructure (Matt: “these services do not provide an API, they rather provide a physical fiber connection”). They appear to be using on-prem capacity mostly as a proxy into AWS.

Terraform Introduction training

The day after the event I attended the one day “Terraform Introduction” training. This was a mix of lecture and practical exercises. The mix was fair and overall the training wasn’t bad (albeit some of the lecture was very basic and redundant with what I already knew about Terraform).

The practical side of it guides you through deploying instances on AWS, using modules, variables and Terraform Enterprise towards the end.

I would advise to take this specific training only if you are very new to Terraform given that it assumes you know nothing. If you already used Terraform in one way or another it may be too basic for you.



]]> 0 963
Kubecon 2017 – Berlin: a personal report Mon, 03 Apr 2017 13:48:58 +0000 Following the establishing best practices of ‘open sourcing’ my trip reports of conferences I attend, I am copying and pasting hereafter my raw comments related to the recent Kubecon EMEA 2017 trip.

I have done something similar for Dockercon 2016 and Serverlessconf 2016 last year and given the feedbacks I had received, this is something . . . → Read More: Kubecon 2017 – Berlin: a personal report]]> Following the establishing best practices of ‘open sourcing’ my trip reports of conferences I attend, I am copying and pasting hereafter my raw comments related to the recent Kubecon EMEA 2017 trip.

I have done something similar for Dockercon 2016 and Serverlessconf 2016 last year and given the feedbacks I had received, this is something worthwhile apparently.

As always:

  1. these reports contains some facts (hopefully I did get those right) plus personal opinions and interpretations.
  2. If you are looking for a properly written piece of art, this is not it.
  3. these are, for the most part, raw notes (sometimes written laying on the floor during those oversubscribed sessions at the conference)

Take it or leave it.


Massimo Re Ferre’ – Technical Product Manager – CNA Business Unit @ VMware

Kubecon Europe 2017 – Report

Executive Summary

In general, the event was what I largely expect. We are in the early (but consistent) stages of Kubernetes adoption. The ball is still (mainly) in the hands of geeks (gut feeling: more devs than ops). While there are pockets of efforts on the Internet to help un-initiated to get started with Kubernetes, the truth is that there is still a steep learning curve you have to go through pretty much solo. Kubecon 2017 Europe was an example of this approach: you don’t go there to learn Kubernetes from scratch (e.g. there are no 101 introductory sessions). You go there because you know Kubernetes (already) and you want to move the needle listening to someone else’s (advanced) experiences. In this sense Dockercon is (vastly) different compared to Kubecon. The former appears to be more of a “VMworld minime” at this point, while the latter is still more of a “meetup on steroids”.

All in all, the enthusiasm and the tail winds behind the project are very clear. While the jury is still out re who is going to win in this space, the odds are high that Kubernetes will be a key driving force and a technology that is going to stick around. Among all the projects at that level of the stack, Kubernetes is clearly the one with the most mind-share.

These are some additional (personal) core takeaways from the conference:

  • K8s appears to be extremely successful with startups and small organizations as well as in pockets of Enterprises. The technology has not been industrialized to the point where it has become a strategic choice (not yet at least). Because of this, the prominent deployment model seems to be “you deploy it, you own it, you consume it”. Hence RBAC, multi-tenancy and security haven’t been major concerns. We are at a stage though where, in large Enterprises, these teams that own the deployment are seeking for IT help and support in running Kubernetes for them.
  • The cloud native landscape is becoming messier and messier. The CNCF Landscape slide is making a disservice to the cloud native beginners. It doesn’t serve any other purpose than officially underline the complexity of this domain. While I am probably missing something about the strategy here, I am puzzled how the CNCF is creating category A and category B projects by listing hundreds of projects in the landscape but only selecting a small subset to be part of the CNCF.
  • This is a total gut feeling (I have no data to back this up) but up until 18/24 months ago I would have said the container orchestration/management battle was among Kubernetes, Mesos and Docker. Fast forward to these days, it is my impression that Mesos is fading out a bit. These days the industry seems to be consolidating around two major centers of gravity: one is Kubernetes and its ecosystem of distributions, the other being Docker (Inc.) and their half proprietary stack (Swarm and UCP). More precisely there seems to be a consensus that Docker is a better fit and getting traction for those projects that cannot handle the Kubernetes complexity (and consider K8s being a bazooka to shoot a fly) while Kubernetes is a better fit and getting traction for those projects that can absorb the Kubernetes complexity (and probably requires some of its advanced features). In this context Mesos seems to be in search of its own differentiating niche (possibly around big data?).
  • The open source and public cloud trends are so pervasive in this domain of the industry that it is also changing some of the competitive and positioning dynamics. While in the last 10/15 years the ‘lock-in’ argument was around ‘proprietary software’open source software’. Right now, the ‘lock-in’ argument seems to be around ‘proprietary public cloud services’ Vs. ‘open source software’. Proprietary software doesn’t even seem to be considered a contender in this domain. Instead, its evil role has been assumed by the ‘proprietary cloud services’. According to the CNCF, the only way you can fight this next level of lock-in is through (open source) software that you have the freedom to instantiate on-prem or off-prem at your will (basically de-coupling the added-value services from the infrastructure). This concept was particularly clear in Alexis Richardson’s keynote.


The Expo was pretty standard and what you’d expect to see. Dominant areas of ecosystem seem to be:

  • Kubernetes setup / lifecycle (to testify that this is a hot/challenging area)
  • Networking
  • Monitoring

My feeling is that storage seems to be “under represented” (especially considering the interest/buzz around stateful workloads). There were not a lot of startups representing this sub-domain.

Monitoring, on the other hand, seems to be ‘a thing’. Sematext and Sysdig (to name a few) have interesting offerings and solutions in this area. ‘We have a SaaS version and an on-prem version if you want it’ is the standard delivery model for these tools. Apparently.

One thing that stood out to me was Microsoft’s low profile at the conference (particularly compared to their presence at, say, Dockercon). There shouldn’t be a reason why they wouldn’t want to go big on Kubernetes (too).

Keynote (Wednesday)

There are 1500 attendees circa at the conference. Given the polls during the various breakout sessions, the majority seem to be devs with a minority of ops (of course boundaries are a bit blurry in this new world).

The keynote opens up with the news (not news) that Containerd is joining the CNCF. RKT makes the CNCF too. Patrick C and Brandon P get on stage briefly to talk about, respectively, Containerd and RKT.

Aparna Sinha (PM at Google) gets on stage to talk about K8s 1.6 (just released). She talks about the areas of improvement (namely 5000 hosts support, RBAC, dynamic storage provisioning). One of the new (?) features in the scheduler allows for “taint” / “toleration” which may be useful to segment specific worker nodes for specific namespaces e.g. dedicated nodes to tenants (this needs additional research).

Apparently RBAC has been contributed largely by Red Hat, something I have found interesting given the fact that this is an area where they try to differentiate with OpenShift.

Etcd version 3 gets a mention as having quite a quite big role in the K8s scalability enhancements (note: some customers I have been talking to are a bit concerned about how to [safely] migrate from etcd version 2 to etcd version 3).

Aparna then talks about disks. She suggests to leverage claims to decouple the K8s admin role (infrastructure aware) from the K8s user role (infrastructure agnostic). Dynamic storage provisioning is available out of the box and it supports a set of back end infrastructure (GCE, AWS, Azure, vSphere, Cinder). She finally alludes to some network policies capabilities being cooked up for next version.

I will say that tracking where all (old and new) features sit on the spectrum of experimental, beta, supported (?) is not always very easy. Sometimes a “new” features is being talked about just to find out that it has moved from one stage (e.g. experimental) to the next (i.e. beta).

Clayton Coleman from Red Hat talks about K8s security. Interestingly enough when he polls about how many people stand up and consume their own Kubernetes cluster a VAST majority of users raise their hand (assumption: very few are running a centralized or few centralized K8s instances that users access in multi-tenant mode). This is understandable given the fact that RBAC has just made into the platform. Clayton mention that security in these “personal” environments isn’t as important but as K8s will start to be deployed and managed by a central organization for users to consume it, a clear definition of roles and proper access control will be of paramount importance. As a side note, with 1.6 cluster-up doesn’t enable RBAC by default but Kubeadm does.

Brandon Philips from CoreOS is back on stage to demonstrate how you can leverage a Docker registry to not only push/pull Docker images but entire Helm apps (cool demo). Brandon suggests the standard and specs for doing this is currently being investigated and defined (hint: this is certainly an area that project Harbor should explore and possibly endorse).

Keynote (Thursday)

Alexis Richardson does a good job at defining what Cloud Native is and the associated patterns.

CNCF is “project first” (that is, they prefer to put forward actual projects than just focus on abstract standards -> they want to aggregate people around code, not standards).

Bold claim that all projects in the CNCF are interoperable.

Alexis stresses on the concept of “cloud lock-in” (Vs generic vendor lock-in). He is implying that there are more people that are going to AWS/Azure/GCP consuming higher level services (OSS operationalized by the CSP) compared to the number of people that are using and being locked in by proprietary software.

Huawei talks about their internal use case. They are running 1M (one million!) VMs. They are on a path to reduce those VMs by introducing containers.

Joe Beda (CTO at Heptio) gets on stage. He talks about how to grow the user base 10x. Joe claims that K8s contributors are more concerned about theoretical distributed model problems than they are with solving simpler practical problems (quote: “for most users out there the structures/objects we had 2 or 3 versions ago are enough to solve the majority of the problems people have. We kept adding additional constructs/objects that are innovative but didn’t move the needle in user adoption”).

Joe makes an Interesting comment about finding a good balance between solving products problems in the upstream project Vs solving them by wrapping specific features in K8s distributions (a practice he described as “building a business around the fact that K8s sucks”).

Kelsey Hightower talks about Cluster Federation. Cluster Federation is about federating different K8s clusters. The Federation API control plane is a special K8s client that coordinates dealing with multiple clusters.


These are some notes I took while attending breakout sessions. In some sessions, I could physically not step in (sometimes rooms were completely full). I skipped some of the breakouts as I opted to spend more time at the expo.


This session was presented by Docker (of course).

Containerd was born in 2015 to control/manage runC.

New in project government (but the code is “old”). It’s a core container runtime on top of which you could build a platform (Docker, K8s, Mesos, etc.)

The K8s integration will look like:

Kubelet –> CRI shim –> containerd –> containers

No (opinionated) networking support, no volumes support, no build support, no logging management support etc. etc.

Containerd uses gRPC and exposes gRPC APIs.

There is the expectation that you interact with containerd through the gRPC APIs (hence via a platform above). There is a containerd API that is NOT expected to be a viable way for a standard user to deal with containerd. That is to say… containerd will not have a fully featured/supported CLI. It’s code to be used/integrated into higher level code (e.g. Kubernetes, Docker etc.).

gRPC and container metrics are exposed via Prometheus end-point.

Full Windows support is in plan (not yet into the repo as of today).

Speaker (Justin Cormack) mentions VMware a couple of times as an example of an implementation that can replace containerd with a different runtime (i.e. VIC Engine).

Happy to report that my Containerd blog post was fairly accurate (albeit it did not go into much details):

Kubernetes Day 2 (Cluster Operations)

Presented by Brandon Philips (CoreOS CTO). Brandon’s session are always very dense and useful. Never a dull moment with him on stage.

This session covered some best practices to manage Kubernetes clusters. What stood out for me in this preso was the mechanism Tectonic uses to manage the deployment: fundamentally CoreOS deploys Kubernetes components as containers and let Kubernetes manage those containers (basically letting Kubernetes manage itself). This way Tectonic can take advantage of K8s own features from keeping the control plane up and running all the way to rolling upgrades of the API/scheduler.


This session was presented by two of the engineers responsible for the project. The session was pretty full and roughly 80% of attendees claimed to be using K8s in production (wow). Helm is a package manager for Kubernetes. Helm Charts are logical units of K8s resources + variables (note to self: research the differences between “OpenShift applications” and “Helm charts” <- they appear to be the same thing [or at least are being defined similarly]).

There is a mention about which is a front-end UI to monocular (details here:

The aim of the session was to seed the audience with new use cases for Helm that aspire to go beyond the mere packaging and interactive setup of a multi-container app.


The attendance was low. The event population being skewed towards developers tend to greatly penalize sessions that are skewed towards solutions aimed to solve primarily Ops problems.

Their value prop (at the very high level) is similar to vSphere Integrated Containers, or Intel Clear Containers for that matter: run Docker images as virtual machines (as opposed to containers). Hyper proud themselves to be hypervisor agnostic.

They claim a sub-second start-time (similar to Clear Path). Note: while the VIC value prop isn’t purely about being able to start containerVMs fast, tuning for that characteristic will help (half-joke: I would be more comfortable running VIC in production than showing a live demo of it at a meetup).

The most notable thing about Hyper is that it’s CRI compliant and it naturally fits/integrate into the K8s model as a pluggable container engine.

]]> 4 959
Docker Containerd Explained in Plain Words Thu, 16 Mar 2017 15:15:24 +0000 This article was originally posted on the VMware Cloud Native corporate blog. I am re-posting here for the convenience of the readers of my personal blog.

I have been frequently asked “what’s [Docker] Containerd?” The short answer I gave may be of benefit for the larger community so I am turning this into a short blog post. I . . . → Read More: Docker Containerd Explained in Plain Words]]> This article was originally posted on the VMware Cloud Native corporate blog. I am re-posting here for the convenience of the readers of my personal blog.

I have been frequently asked “what’s [Docker] Containerd?” The short answer I gave may be of benefit for the larger community so I am turning this into a short blog post. I hope the condensed format is useful.

Background: What’s the Problem

Docker started a technology (Docker Engine) that allows you to package and run your application in a Linux container on a single host. Linux containers have been around for decades.

Docker made them consumable for the masses. At this point a couple of things happened in parallel:

  • The ecosystem started to flourish and more open source projects started to leverage Docker (Engine) as a core building block to run containerized applications at scale on a distributed infrastructure (think Kubernetes).
  • Docker Inc. (the company) started working on solving similar problems and decided to embed into the core Docker (Engine) technologies that would help solve the problems of running containerized applications at scale on a distributed infrastructure (think Docker Swarm Mode)

This created a dynamic where Docker and Kubernetes started building similar solutions – one is building solutions to solve problems on a core building block that happens to embed another solution (created by the latter) to solve the same problems and it is creating friction in the industry.

What Happened Next?

Purists and open source advocates are advocating that, by doing so, Docker Inc. is bloating the core building block with additional bugs and instability to make space for code that isn’t needed (when third party solutions are being used). Third party vendors are claiming that Docker Inc. is creating an artificial funnel and path with commercial interest. The general fear is rooted in (1) using Docker (Engine) for free results in (2) enabling Swarm Mode for free leads to (3) buying Docker Data Center. The industry has started to bifurcate to either forking Docker (Engine) or building a completely separate container runtime.

Enter Containerd

Docker has announced Containerd (, an open-source project that the industry can use as a common container run-time to build added value on top (e.g. container orchestration, etc.)

Containerd is a daemon that runs on Linux and Windows, and it can be used to manage the container lifecycle including tasks such as image transfer, container execution, some storage and networking functions. With Containerd, Docker has evolved again to below:

There are many questions such as how Containerd will be packaged, how current Docker Engine will be re-packaged, etc. that have yet to be answered.  I will write another blog to follow up, and look forward to hearing your thoughts.


]]> 3 952
Automating vSphere Integrated Containers Deployments Tue, 10 Jan 2017 15:04:30 +0000 In the second half of 2016 I had an opportunity to focus (almost) exclusively on vSphere Integrated Containers all the way through its GA in early December.

I had been working on VIC Engine and project Bonneville since 2015 but in H2 2016 I focused pretty much full-time on that and I had the pleasure . . . → Read More: Automating vSphere Integrated Containers Deployments]]> In the second half of 2016 I had an opportunity to focus (almost) exclusively on vSphere Integrated Containers all the way through its GA in early December.

I had been working on VIC Engine and project Bonneville since 2015 but in H2 2016 I focused pretty much full-time on that and I had the pleasure to be involved in the discussions that led vSphere Integrated Containers to be what it is today.

As part of my job I found myself in situations where I had to tear up and down environments for test and other various purposes. Because of this I had been using a script (with many rough edges) to properly setup, against a given vSphere environment, the three VIC components (i.e. VIC Engine, Harbor and Admiral).

I figured I’d share that script publicly at this point as it may benefit (somehow) the community. This isn’t only for being able to use it as-is (it could be adapted and enhanced to fit your specific needs) but also because it may show you how each component is being setup and installed (you can read through the script and see how the mechanic works).

The script is called and it is available at this GitHub repo.

This solution is nothing more than the codified version of the user workflows we created for the alpha and beta phases of the VIC product (where users were given a step by step flow to setup an end-to-end VIC stack).

The purpose of this script was (also) to validate whether the documented steps where technically valid.

Using code to perform documentation quality checks, if you will. Mh…

In case this script will evolve over time I am going to document what it does and what its major limitations are on GitHub so you can always read the latest (Vs printing a description in stone on a static blog post while the code, potentially, evolves).

Please move to the GitHub repo of the project if you want to read more about it (and if you want to give it a try).



]]> 0 949
VMware Harbor as a Rancher catalog entry Mon, 09 Jan 2017 10:23:51 +0000 In the last few months I have been looking at Rancher (an open source container management product) as a way to learn more about Docker containers and understand better the ecosystem around them.

One of the things that appealed me about Rancher is the notion of an extensible catalog of application . . . → Read More: VMware Harbor as a Rancher catalog entry]]> In the last few months I have been looking at Rancher (an open source container management product) as a way to learn more about Docker containers and understand better the ecosystem around them.

One of the things that appealed me about Rancher is the notion of an extensible catalog of application and infrastructure services. There is an “official” catalog tier called Library (maintained and built by Rancher the company), there is a “community” catalog tier called Community (maintained by Rancher the company but built and supported by the Rancher community) and then there is a “private” tier (where you can add your own private catalog entries that you own and maintain).

While Rancher supports users to connect to cloud based registries, I noticed there is only one container registry package in the Community catalog that one can deploy and manage (that is Convoy Registry). I thought this was a great opportunity (and a great learning exercise) to add VMware Harbor (an Enterprise class container registry) as a private catalog item option.

If you know Rancher, something like this. If you will.

What you will see and read below should be considered, literally, the proof of a concept that, in its current shape and form, can’t be used as-is for production purposes. Having that said, if there is enough appetite for this, I am willing to work more on it and refine some of the rough edges that exist today.


Before we dive into the topic, I want to thank Raul Sanchez from Rancher that has (patiently) answered all of my questions (and fixed some of my broken yaml). Without his help this blog post would have been much shorter. Oh I also want to thank my wife and my dog (although I don’t have a dog), because everyone does that.

An escalation of difficulties learning opportunities

It became immediately very clear that there were different set of tasks I needed to accomplish to be able to achieve the initial goal. One task was, usually, dependent on the other.

In order to be able to create a Rancher catalog entry, you need to be able to instantiate your application from an application definition file (when using the default Cattle scheduler that would be a standard Docker Compose file) as well as a Rancher Compose file. You can’t really run any shell script or stuff like that as part of a Rancher catalog entry.

If you explore how Harbor gets installed on a Docker host (via the documented “on-line installer”), it’s not really compatible with the Rancher catalog model.

With the standard on-line installer, you have to download the Harbor on-line tar.gz installer file, you have to explode it, you have to set your configuration settings in the harbor.cfg file, you have to run a “prepare” script that takes the harbor.cfg file as an input and create configuration files and ENVIRONMENT variables files for you to THEN, eventually, run the Docker Compose file passing the config files and ENVIRONMENT variable files as volumes and directives of Docker Compose (note some of these steps are buried under the main install script but that’s what happens behind the scenes). At this point the Docker Compose file actually grabs the docker images off of Docker Hub and instantiate Harbor based on the configuration inputs.

At the end, what started as a simple “PoC” project, turned into three “sub-projects”:

  1. Dockerizing (is this a word?) the Harbor on-line installer so that you can include the “preparation” procedure as part of the Docker Compose and pass input parameters as variables to Docker Compose (instead of editing the harbor.cfg file manually and perform the whole preparation circus)
  2. Rancherizing (this is not definitely a word!) the dockerized Harbor on-line installer and create a Rancher private catalog entry that mimic the typical single-host Harbor setup
  3. As a bonus: rancherizing the dockerized Harbor on-line installer and create a Rancher private catalog entry that allows Harbor to be installed on a distributed cluster of Docker hosts

Note that, while I had to create a dockerized Harbor on-line installer to fit the Rancher catalog model, you can also use it for other use cases where you need to automatically stand up Harbor but you don’t have a way to do that manually and interactively (Rancher being one of those use cases).

In the next few sections, I am going to cover in more specific details what I have done to implement these sub-projects.

Sub-project #1: Dockerizing the Harbor on-line installer

At the time of this writing, Harbor 0.5.0 can be installed either using an OVA or through an installer. The installer can be on-line (images get dynamically pulled from Docker Hub) or off-line (images are shipped as part of the installer and loaded locally).

We are going to focus on the on-line installer.

As we have alluded already, once you download the on-line installer, you have to “prepare” your Harbor installation by tweaking the parameters of the harbor.cfg file that ships with a template inside the install package.

The resulting set of configurations are then passed as an inputs to the Docker Compose file (via local directories mapped as “volumes” and via the “env_file” directive).

Wouldn’t it be easier/better if you could pass the Harbor setup parameters directly to the Docker Compose file without having to go through the “preparation” process?

Enter harbor-setupwrapper.

Harbor-setupwrapper is a Harbor installer package that includes a new docker image which (more or less) implements, in a docker container, the “preparation” process. This container accepts, as inputs, Harbor configuration parameters as environment variables. Last but not least the container runs a script that launches the preparation routines (this is all self-contained inside the container).

The Dockerfile for this image and the script that kicks off the preparation routines that ships with it are worth a thousand words.

If you will, what the does, is very similar to what the does for the standard Harbor on-line installer.

We now have a new Docker Compose file which is largely based on the original Docker Compose file that ships with the official on-line installer. You can now “compose up” this new Docker Compose file passing the parameters that you would have otherwise tweaked in the harbor.cfg file.

This is a graphical representation of what I have done:

Remember this is just a PoC. Mind there are caveats!

  • I have only really tested this with the HARBORHOSTNAME and HARBOR_ADMIN_PASSWORD variables. Other variables should work but I haven’t tested them
  • There will definitely be scenarios where this will break. For example, I haven’t implemented a way to create certificates if you choose to use a secure connection (https). This would need to be implemented as additional logic inside (hint: do not try to enable https because weird things may happen)
  • The original on-line installer is (really) meant to be run on a single Docker host. The approach I have implemented in this new install mechanism honors that model and assumes the same constraints
  • Because of the above, I didn’t even try to deploy this compose file on a distributed Swarm cluster. BTW, in the transition from “legacy Swarm” to “Swarm mode” Docker Compose doesn’t seem to have gained compatibility with the latter and given I didn’t want to waste too much time with the former, I have just opted to not test it in a Swarm environment
  • More caveats that I haven’t thought about (but that certainly may exist!)

Making available the configuration files from the wrapper (generated by the script) to the application containers was the easy piece. I have achieved that with the “volumes_from” directive where the application containers would grab their relevant configuration files directly from the wrapper container.

What proved to be more challenging was figuring out a way to pass the ENVIRONMENT variables (that are, again, in various files on the wrapper container) to the application containers. I could not use the “env_file” directive in compose because the value of the directive refers to files that are visible to the system where the compose is run from (whereas in my case those files were inside the wrapper container). Long story short, I ended up tweaking the ENTRYPOINT of the application containers to point to a script that would, first, load those environment variables and, then, it would start the original script or command that was in the original ENTRYPOINT. If you are curious, you can check all the entrypoint*.sh files on the harbor-setupwrapper GitHub repo.

If you want to play with this and setup Harbor using this new mechanism, all you need to do is cloning the harbor-setupwrapper repo and “up” the Docker Compose file you find in the harbor-setupwrapper directory. However, before you launch it, you will have to export the HARBORHOSTNAME and the HARBOR_ADMIN_PASSWORD variables. This is the equivalent of tweaking the harbor.cfg file in the original installer. If you forget to export these variables, Docker Compose will show this:

root@harbor:~/harbor-setupwrapper# docker-compose up -d
WARNING: The HARBORHOSTNAME variable is not set. Defaulting to a blank string.
WARNING: The HARBOR_ADMIN_PASSWORD variable is not set. Defaulting to a blank string.
Creating network "harborsetupwrapper_default" with the default driver

At a minimum the HARBORHOSTNAME variable needs to be set and it needs to be set to the IP address or FQDN of the host you are installing it on (otherwise the setup will not be functional for reasons that I am going to explain later in the post). If you do no set the HARBOR_ADMIN_PASSWORD variable you will have to use the default Harbor password (Harbor12345).

What you want to do is this:

root@harbor:~/harbor-setupwrapper# export HARBORHOSTNAME=
root@harbor:~/harbor-setupwrapper# export HARBOR_ADMIN_PASSWORD=MySecretPassword
root@harbor:~/harbor-setupwrapper# docker-compose up -d
Creating network "harborsetupwrapper_default" with the default driver
Creating harbor-log

Hint: if you keep bringing up and down Harbor instances on the same host and you intend to start them from scratch, please remember to get rid of the /data directory on the host (because that is where the state of the instance is saved, and new instances will inherit that state if the instances find that directory).

Sub-project #2: Creating the Rancher catalog entry for the single-host deployment

Now that we have a general purpose dockerized Harbor installer that we can bring up by just doing a “compose up”, we can turn our attention to the second sub-project. That is, creating the structure of the Rancher catalog entry.

I thought this was the easy part. After all, this was pretty much about re-using the new docker-compose.yml file we discussed above, in the context of Rancher. Right? Well…

I have learned (the hard way) that the devil is always in the details and, particularly in the context of containers, a tweak here to “fix” a particular problem often means opening up a can of worms somewhere else.

I will probably report hereafter more than what you would need/want to read to be able to consume this package but I am doing so to share my pains in the hope they may help you in other circumstances (I am also documenting my findings otherwise I will forget in 2-week time).

First and foremost, in Rancher, you can only do “volume_from” within the context of a sidekick. I have originally played with adding “io.rancher.sidekicks: harbor-setupwrapper” to every container in compose. However, I suddenly found out that this would create a harbor-setupwrapper helper container for every container that this was declared a sidekick of. Albeit this sounded ok to start with, I eventually figured that running multiple instances of the preparation scripts for a single Harbor deployment may lead to various configuration inconsistencies (e.g. tokens signed with untrusted keys, etc.).

I had to revert back to a strategy where I only had one instance of the harbor-setupwrapper container (that would generate consistently all the configuration files in one go) and I have accomplished that by making it the main container with all the other application containers being sidekicks of it. Practically, I just added “io.rancher.sidekicks: registry, ui, jobservice, mysql, proxy” as a label of the harbor-setupwrapper container. Warning: I did not tell Raul this and this may horrify him (or any other Rancher expert). But it works, so bear with me.

As usual, we fixed a problem by opening up another. Name resolution with sidekick containers doesn’t really work the way you expect it to work so I had to put in place another workaround (if you are interested you can read the problem and the fix here).

There were another couple of problems I had to resolve in the process of creating a Rancher catalog entry:

  • Harbor requires that the variable “harborhostname” is set to the exact value that the user will use to connect to that harbor instance.
  • All Harbor containers needs to be deployed on a single host which is more likely going to be one of the hosts of a (Cattle) cluster of many hosts.

Again, in a move that may horrify Rancher experts, I have configured the Docker Compose file to schedule all containers on the host that has a “harbor-host=true” label.

This allowed me to make sure all containers get deployed on the same host (and, more importantly, have some degree of control over which one that is). In fact, given that I know which host my containers are going to land on, I can choose wisely the variable “harborhostname”. That could be the host IP address or the host FQDN.

Last but not least the Docker Compose file will publish ports 80 and 443 of the proxy container on the host (it goes without saying that those ports need to be free on that host, otherwise the deployment will fail). Again, perhaps not a great best practice but something basic and easy that works.

Note: remember that state is saved in the /data directory of the host so if you intend to bring up and down instances of Harbor for test purposes, you need to be aware that state is kept there across multiple deployments. This is far from how you’d run a true cloud native application but it is how Harbor (0.5.0) is architected and I am just honoring the original operational model in the Rancherization scenario for single host.

The following picture shows the details and relationships of the various components in a single-host deployment:

This picture shows the actual deployment in action in my lab:

As a recap, at the high level, the current status of the Harbor private catalog entry for Rancher for single-host deployment is as follows:

  • It only works with the Cattle scheduler
    • Building Harbor catalog versions for Swarm and K8s based off the Cattle version should be relatively trivial (last famous words)
  • This catalog entry inherits all of the limitations of the dockerized on-line installer described above (e.g. it doesn’t support https etc)
  • The Docker hosts where you pull/push images need to have the “–insecure-registry” flag set on the Docker daemon (because we can only fire up Harbor with http access)
  • One of the hosts will have to have the “harbor-host=true” label for the docker-compose to work and schedule containers properly
  • The host with the “harbor-host=true” label will have to have ports 80 and 443 available

You can locate the deliverable for this sub-project inside my overall Rancher catalog extension repo.

You can watch the single-host deployment in action in this short video:

Sub-project #3: Creating the Rancher catalog entry for the distributed deployment

This is the part where I have learned the most about the challenges of operationalizing distributed applications. It was definitely a fun useful ride.

While Harbor is shipped as a containerized application, there are some aspects of it that do not make it an ideal candidate for applying cloud native application operational best practices. It doesn’t really adhere to The Twelve-Factor App methodology.

First and foremost, the Harbor installer has been built with the assumption that all 6 containers are run on a “well-known” single host. Let me give you a couple of examples that may underline some of these challenges. We may have already mentioned some of those:

  • The Harbor package comes with an embedded syslog server that the Docker daemon talks/logs to. If you look at the original Docker Compose file, you will notice that all application containers log to implying and assuming that the syslog is running on the same host of all other containers
  • You have to enter (as a setup parameter) the exact harbor hostname that users will use to connect to the registry server. Ideally, in a cloud native context, an app should be able to work with any given IP / FQDN that has been associated with it. As a last resort you should have an option to set (post-setup) the proper IP/FQDN endpoint that the app should be using. With Harbor 0.5.0 you have to know (upfront) what that IP/FQDN is before launching the setup (making things a bit more difficult to operationalize in a dynamic, self-service and distributed environment). That is to say: if your harbor service will happen to be exposed to users as “” you have (had) to enter that string as the FQDN at deployment time (without even possibly knowing which hosts the containers are going to be deployed on)
  • As part of the assumption that Harbor runs on a single well-known host, the product saves its own state on local directories on the host it is deployed onto. This is accomplished by means of various directory mappings in the containers configuration

The goal for this sub-project is to make Harbor run distributed on a Cattle cluster and no longer on a well-known host.

In order to do that the log image gets instantiated on every node of the cluster (requirement: each node has to have the label “harbor-log=true”). A more elegant solution would be to have a separate syslog server to point to (thus getting rid completely of the log service in Docker Compose).

Also, given we don’t know which host the proxy server is going to end up on (and given that in this scenario we wanted to implement a low touch experience in terms of service discovery) we have implemented the Harbor distributed model by leveraging Traefik (as explained in this blog post by Raul). If you are familiar with Docker, what Traefik does is (somewhat) similar to the “HTTP Routing Mesh” out-of-the-box experience that Docker provides with Swarm Mode. Please note that the proxy container ports (80 and 443) do not get exposed on the host and Traefik is the only way to expose the service to the outside world (in this particular distributed implementation).

The overall idea is that your DNS can resolve the IP of where Traefik runs and then Traefik “automagically” adds the hostname you have entered at Harbor setup time to its configuration. Check Raul’s blog post for more information on the setup concepts.

Storage management has been another interesting exercise. In a distributed environment you can’t let containers store data on the server they happen to run on at any given point in time.

If the container is restarted on another host (due to a failure or due to an upgrade) it needs to get access to the same set of data. Not to mention if other containers (that may happen to run on different hosts) need to access the same set of data.

To overcome this problem, I opted to use the generic NFS service Rancher provides. This turned out to be useful, flexible and handy because it allows you to pre-provision all the volumes required (in which case they persist across re-instantiation of the Harbor catalog entry) or you can let Docker Compose create them automatically at instantiation time (in which case they will be removed when the Harbor instance gets brought down). Please note that, to horrify the purists, all volumes are mapped to all the application containers (except the log and proxy container which don’t require volumes). There is vast room for optimization here (as not all volumes need to be mapped to all containers) but I figured I’ll leave it like that for now.

This approach leaves all the hosts stateless as there is no volume directories mapping in Docker Compose (all volumes are named volumes that live on the NFS share).

The following picture shows the details and relationships of the various components in a distributed deployment:

This picture shows the actual deployment in action in my lab:

As a recap, at the high level, the current status of the Harbor private catalog entry for Rancher for distributed deployment is as follows:

  • It only works with the Cattle scheduler
    • Building Harbor catalog versions for Swarm and K8s based off the Cattle version should be relatively trivial (last famous words)
  • This catalog entry inherits all of the limitations of the dockerized on-line installer described above (e.g. it doesn’t support https etc)
  • The Docker hosts where you pull/push images need to have the “–insecure-registry” flag set on the Docker daemon (because we can only fire up Harbor with http access)
  • All of the hosts in the cluster will have to have the “harbor-log=true” label for the docker-compose to work and schedule the log image properly
  • The Traefik service (found in the Community catalog) needs to be up and running for being able to access Harbor from the outside. This has been tested exposing port 80 (note the Traefik default is 8080)
  • The NFS service (found in the Library catalog) needs to be up and running and properly configured to connect to an NFS share. The Docker Compose file has been parametrized to use, potentially, other drivers but the only one I have tested is “rancher-nfs

You can locate the deliverable for this sub-project inside my overall Rancher catalog extension repo.

You can watch a distributed deployment in action in this short video:

Overall challenges and issues

Throughout the ­project I came across a few challenges. I am going to mention some of them here mostly for future reference by keeping a personal track of those.

  • Even small things can turn into large holes. Sometimes it’s a cascading problem (i.e. to do A you need to do B but doing B requires you to do C). A good example was the Rancher sidekick requirement to be able to perform a “volume_from“. That basically broke entirely name resolution (see the single-host section for more info re what the problem was)
  • Containers coming up “all green” doesn’t mean your app is up and running (properly). There were situations where containers were starting ok with no errors but I couldn’t login into harbor (due to certificates mismatches generated by running multiple instances of the install wrapper). There were situations where I could login but couldn’t push images. And there were situations where I could push images but the UI couldn’t show them (due to the registry container not being able to resolve the ui container name because of the name resolution issues with sidekicks)
  • Debugging containers in a distributed environment is hard. At some point I came across what seemed to be a random issue to later find out that the problem was due to a particular container getting scheduled (randomly) on a specific host that was mis-configured. Fixing the issue was easy once this was root-caused. Root-causing was hard
  • Knowledge of the application internal is of paramount importance when packaging it to run in containers (and, most importantly, to orchestrate its deployment). One of the reasons for which I left all named volumes connected to all containers in the distributed scenario is because I am not 100% sure which container reads/writes from/to which volume. Plus, a thousands of other things for which not knowing the application makes its packaging difficult (particularly for debugging purposes when something doesn’t work properly). All in all, this enforces my idea that containers (and their orchestration) is more akin to how you package and run an application Vs how you manage an infrastructure
  • While container orchestration is all about automation and repeatable outcomes, it’s also a little bit like “hand-made elf art“. There was an issue at some point where (randomly) the proxy container would only show the nginx welcome page (and not the Harbor user interface). Eventually I figured that restarting that container (after the deployment) fixed the problem. I thought this was due to some sort of start up sequence. I tried leveraging the “depends_on” directive to make the proxy container start “towards the end” of the compose up but that didn’t work. It seems to work consistently now by leveraging the “external_links” directive (which, in theory, shouldn’t be required AFAIK). All in all being able to properly orchestrate the start up of containers is still very much a working in progress and a fine art (apparently since 2014)
  • Managing infrastructure (and services) to run containerized applications is hard. During my brief stint into leveraging something simple like the basic Rancher NFS service, I came across myself a few issues that I had to workaround using different levels of software, different deployment mechanisms, etc etc. Upgrades from one version of the infrastructure to another are also very critical
  • Another NFS related issue I came across was that volumes don’t get purged properly on the NFS share when the stack is brought down. In the Rancher UI the volumes seem to be gone but, looking directly at the NFS share, some of them (a random number) seem to be left over in the form of left over directories. I didn’t dig too deep into why that was the case


As I alluded to this is a rough integration that, needless to say, can be perfected (euphemism). This was primarily an awesome learning exercise and future enhancements (e.g. integration with the Kubernetes scheduler in Rancher, enabling the https protocol, etc. etc.) will allow me to stretch it even further (and possibly making it more useful).

An exercise like this is also very useful to practice some of the challenges that you could come across in the context of distributed systems with dynamic provisioning consumed via self-service. Sure this was not super complex but being able to get your hands dirty with these challenges will help you better understand the opportunities that exist to solve them.

As a side note, going deep into these experiments allows you to appreciate the difference between PaaS and CaaS (or, more generally, between a structured approach Vs an unstructured approach, if you will).

With PaaS a lot of the design decisions (particularly around scheduling, Load Balancing, name resolution etc) are solved for you out of the box. However, it may be the case that hammering your application to make it fit a distributed operational model may not work at all or it may work in a too opinionated and limited way. With an unstructured approach (like the one discussed in this blog post) there is a lot more work that needs to be done to deploy an application (and there is a lot more “sausage making” that you get expose to) but it can definitely be tailored to your specific needs.


]]> 4 946
Dockercon 2016 – Seattle: a personal report Wed, 29 Jun 2016 12:34:14 +0000 A few weeks ago I shared on my blog a cleaned up conference report from Serverlessconf 2016.

Given the relatively good success of the experiment (some good feedback at “no cost” for me – copy and paste FTW!) I decided to take another shot.

This time I am sharing . . . → Read More: Dockercon 2016 – Seattle: a personal report]]> A few weeks ago I shared on my blog a cleaned up conference report from Serverlessconf 2016.

Given the relatively good success of the experiment (some good feedback at “no cost” for me – copy and paste FTW!) I decided to take another shot.

This time I am sharing a report from Dockercon 2016 (in Seattle). This has been similarly polished by deleting comments that were not supposed to be “public”. You will see those comments being replaced by <redacted comments>.

As always, these reports are supposed to be very informal and surely includes some very rough notes. You have been warned.

What you will read below this line is the beginning of my original report (and informal notes).



Executive Summary

These are, in summary, my core takeaways from the conference.

It’s all about Ops (and not just about microservices)

So far everything “Docker” has done, it has primarily catered to a developer audience.

But there were two very high level and emerging trends at Dockercon 2016.

  • Trend #1: running containers in production

This was a very pervasive message from both Docker Inc. as well as all ecosystem vendors. It’s almost like if they realized that to monetize their technology they (also) need to start talking to Ops (on top of evangelizing Devs).

  • Trend #2: how to containerize legacy applications

This was another pervasive trend at the show. The cynic in me is suggesting that this is also due to a monetization strategy. It feels like there are too many fish (vendors/startups) in such a (still) little pond (microservices) that if you can turn the pond into a sea they can all swim and prosper better.


Is it me or was Docker Cloud missing in action?

Completely unrelated, but at least as interesting, it is that Docker Cloud had very little coverage during the event. Docker acquired Tutum a while back and a few months ago they renamed their offering to Docker Cloud. Docker Cloud is supposed to be (at least according to the message given at Dockercon 2015 in Barcelona) the public cloud version (managed by Docker Inc.) of a Docker Datacenter instantiation on-prem (managed by the customer). Last year I pointed out that the two stacks (Docker Cloud and Docker Datacenter) were so vastly different that making them converge into a single stack would have been a daunting exercise. This year apparently Docker chose to not talk about it: it was not featured prominently in keynotes and did not have a dedicated breakout session (unless I missed it?). It is as if Docker decided to step back from it and focus on being a pure software company instead on being a cloud services provider (i.e. “if you want a Docker stack in the public cloud go get the bits and instantiate them on AWS or Azure”).


Technology focus (and then lack thereof)

Another interesting aspect of Dockercon 2016 is that a lot of technology focus has been put into the OSS version of the bits (i.e. Docker Engine 1.12) during the day 1 keynote. The technology focus on Docker Datacenter (the closed source “pay for” bits) during the day 2 keynote was not that big of a splash. They focused instead more on getting customers on stage and more on educating on the journey to production rather than announcing 20 new features of Docker UCP. Whether that is because Docker Inc. did not have 20 new features in Docker UCP or because they thought the approach they took was a better one, I don’t know.


Bare metal, Containers, VMs, etc

The architectural confusion re the role of bare metal, VMs, and containers reigns supreme. There is a lot of pre-constituted love, hate or otherwise bias towards which role each of these technology plays that, many times, it is hard to get involved in a meaningful and genuine architectural discussion without politics and bias getting in the way. All in all, the jury is still out re how this will pan out. Recurring patterns will only start to emerge in the next few years.



During the closing keynote on Tuesday there were a number of interesting demos, one of which was about how to use Docker for Serverless patterns. What was interesting to me was not so much about the technical details but how Docker felt the need to rush and get back the crown of “this is the coolest thing around”. Admittedly the “Serverless movement” has started to steal some momentum from Docker lately by burying (Docker) containers under a set of brand new consumption patterns that would NOT expose a native Docker experience. I’d speculate that Docker felt a little bit frightened by this and is starting to climb back up to remain at the “top of the stack”.  The Serverless community made an interesting counter argument.


Docker and the competition

Docker is closing some of the gaps they had compared to other container management players and projects (Kubernetes, Rancher, Mesos, etc). Docker has a huge asset in Docker Engine OSS and with the new 1.12 release they have made a lot of good improvements. I am only puzzled that they keep adding juice to the open source (and free) bits (Vs to UCP) thus posing a potential risk to their overall monetization strategy. But then again when you fight against the like of Rancher and Kubernetes that make available (so far) all they do “for free”, it’s hard to take a different approach. On a different note, the anti-PaaS attitude continues strong by pointing out that the approach is too strict, too opinionated and doesn’t leave the freedom to the developers to make the choices the developers may want to make.

Interesting times ahead for sure.


Docker and VMware

<redacted comments>



The floor was well attended and the (main) messages seem to be aligned to those of Docker itself. The gut feeling is that there was slightly less developers love and more Ops love (i.e. running containers in production and containerize traditional applications).


Monday Keynote (rough notes)

Dockercon SF 2014 had 500 attendees, Dockercon SF 2015 had 2000 attendees, Dockercon Seattle 2016 had 4000 attendees. Good progression.

Ben Golub (Docker Inc. CEO) is on stage.

He suggests that 2900+ people are contributing to Docker (the OSS project).

Docker Hub has 460K images and has seen 4.1B pulls to date.

Ben claims 40-75% of Enterprise companies are using Docker in production. I call this BS (albeit one has to ask what production means).

Solomon Hykes (Docker CTO) gets on stage and starts with an inspirational speech re how docker enables everyone to change the world building code etc.

Solomon suggests that the best tools for developers:

  • Get out of the way (they enable but you need to forget about them)
  • Adapt to you (this sounds like a comment made against opinionated PaaS vendors)
  • Make powerful things in a simple way

Interesting demo involving fixing live a bug locally (on Docker for Mac) in the voting app and deploying the app again in the staging environment (building / deploying automatically triggered by a GitHub push).

CTO of Splice is on stage (this is a good session but a “standard talk” re why Splice loves Docker). Surely the value here is not so much content but rather having a testimonial talking about how they use Docker for real.

Orchestration is the next topic. Solomon claims the technology problem is solved but the problem is that you need an “army of experts”.

Interesting parallel: orchestration today is like containers 3 years ago: solved problem but hard to use.

Docker 1.12 includes lots of built-in orchestration features out of the box.

  • Swarm mode: docker engine can detect other engines and form a swarm with a decentralized architecture. Note this seems to be a feature of the daemon Vs implemented with containers (as it used to).
  • Cryptographic node identity: every node in the swarm is identified by a unique key. This is out of the box and completely transparent.
  • Docker Service API: dedicated/powerful APIs for Swarm management.
  • Built-in Routing Mesh: overlay networking, load balancing, DNS service discovery (long overdue).

Now Docker 1.12 exposes “docker create service” and “docker service scale” commands which sound very similar to the concept of Kubernetes Pods (in theory).

In addition, the routing mesh allows EACH docker host to expose a map of the container port on every host.

Last but not least, you can scale the containers and the built-in load balancer will help load balance the n instances of the container.

The “service” will enforce that the the right number of containers running on the swarm is enforced (the demo involves shutting down a node and all containers gets restarted on a different node to honor the committed number of containers running).

The speaker uses the word “migrated” referring to containers restarted on other nodes but (I think) what really happens is that another stateless container is started on the new host. There is no data persistency out-of-the-box (for now at least) that I can see.

Again, this is all very Kubernetes-ish to the point that it feels like the original comment of “needing an army of experts to do proper orchestration” is addressed to Kubernetes itself (my speculation).

Zenly (@zenlyapp) is on stage.

They claim that public cloud is expensive (at their scale) and they decided to move from (GCP) to on-prem (on bare-metal). They don’t mention why bare-metal (albeit their use case is so well defined that their “single app” focus does not need the flexibility that virtualization can provide in an otherwise more complex environment).

Solomon introduces the DAB (Distributed Application Bundle) as a way to package the containers comprising your distributed application as a bundled “app” (docker-compose bundle).

And here we are back full circle to distributing an app as a 500GB zip file (I am just kidding – sort of).

Docker for AWS and Docker for Azure are announced.  This is a native Docker experience integrated with AWS / Azure (e.g. as you expose a port with a docker run the AWS ELB gets configured to expose that port). From the demo this Docker for AWS looks like a CloudFormation template.

The demo involves the IT person to deploy a 100 nodes swarm cluster with the integration and showing his familiarity with all AWS terminology when it comes to Ops (keys, disks, instances, load balancers etc).

They copy the voting app DAB file to an AWS instance (docker host) and they “docker deploy instavote” from that EC2 instance.

All in all, the first day was completely devoted to the progresses being made in the Docker OSS domain. In general, this was very well received even though someone was wondering if this is how you actually run OSS projects:


Tuesday Keynote (rough notes)

Yesterday was all about democratizing docker for developers and individuals.

Today it is all about democratizing docker for Enterprises.

Ben Golub is back on stage and he talks about the fallacies of “Bimodal IT”, “PaaS”, “Private IaaS”.

Docker isn’t about a revolution but rather an evolution.

“PaaS is too high, IaaS is to low, CaaS is just about right”.

He introduces Docker Datacenter.

Docker won’t impose infrastructure models nor will impose higher level management frameworks etc.

Very entertaining demo on Docker Datacenter covering RBAC, security vulnerability scanning, fixing issues and rolling updates.

Only thing is… all is being talked about / demoed is often around stateless applications / services. Data persistency in these environments is still the elephant in the room.

Ben gives a shout out to HP for shipping servers with Docker pre-loaded-whatever. No one (I talked to) seems to understand what this means in practice.

Now Mark Russinovich (MS Azure CTO) is on stage showing that Docker Datacenter is featured in the Market place.

Despite a semi-failed demo Mark killed it showing a number of things (Docker Datacenter on Azure with nodes on Azure Stack on-prem with the app connecting to a MSSQL instance running on-prem on Ubuntu).

Surely MS wanted to show the muscles and a departure from the “it must be Windows only” approach.

The only thing that stood out for me is that they have decided NOT to show anything that relates to running containers on Windows (2016). That was an interesting choice.

Mark is done and now the CTO of ADP is on stage talking about their experience. He claims that every company is becoming a software company to compete more effectively.

In an infinite race among competitors “speed” is more important than “position” (i.e. if you are faster, in the long run you will overtake your competition (no matter what).

Coding faster isn’t enough. You need to ship faster.

He makes an awesome parallel with the F1 race circuit: cars cost $30M and pilots are celebrities BUT the people at the pits (IT?) are what determine whether the pilot will win or lose depending on whether the pit stop is going to be 1.5 seconds or 5 seconds. THAT can make (and will make) the difference.

This keynote was all about “running Docker in production”. Solomon made sure the message was well received with a tweet:


Breakout Sessions (rough notes)

What follows is a brief summary of some of the breakout sessions I attended.


Containers & Cloud & VMs: Oh my

The content seems to be a superset of a blog post Docker published weeks ago:

Quote: “People think about containers as lightwave virtualization and that’s so wrong”.

Some people go bare metal. Some people go virtualized. How did they decide? It depends.

So many variables:

  • Performance
  • Security
  • Scalability
  • Existing skillsets
  • Costs
  • Flexibility
  • Etc

 If you can’t fill up an entire server with containers go with VMs

If you are using mixed workloads you may want to consider VMs.

If your apps are latency sensitive, consider bare metal.

If you are looking for DR scenarios, consider go with VMs (vSphere is called out specifically).

If you need Pools / Quotas consider VMs.

If you want to run 1 container per docker host, consider running in VMs.

Interesting comment: dockerizing monolithic applications isn’t a best practice but gives you portability for gen 2 applications.

Overall the session content was a lot of common sense (which needs to be reiterated anyway).

The closing is interesting: “I wish I could give you a precise answer to the question but reality is that we are in the early days and patters / best practices aren’t yet defined.”

I couldn’t agree more with the above view from the speaker. There is no clear pattern emerging and everyone is doing what they <think> it’s the right thing to do.

Answering a question from the audience the speaker claims that security being weaker in containers Vs VMs is just FUD.



The session was interesting. You can feel there are some serious talents at Joyent but they seem to be very opinionated re how things should be done (perhaps not a bad thing).

In this session the topic was database persistency and I walked in convinced they would end up talking about docker volumes. Ironically, in the end, they said docker volumes were a big no no and their suggested approaches were database technologies that could self-heal by leveraging native sharding and/or replication support on ephemeral containers.

They were one of the few vendors that did not buy into the “you can dockerize your monolith” new approach Docker and others showed at the conference.


Docker in the Enterprise

This session was delivered by ex-VMware Banjot C (now PM for Docker Datacenter).

Introduction to the value proposition of CaaS (Build / Ship / Run).

Docker Datacenter is Docker’s implementation of a CaaS service.

Goldman Sachs, GSA and ADP are brought up as customers using docker.

Enterprise can start their journey either with monolith applications or microservices applications.


  1. Containerizing monolith is useful for build/test CI and or better Capex/Opex than VMs.
  2. Containerizing monolith and then decompose it into microservices
  3. Containerizing microservices

The value prop for #1 is that you can now truly port your application everywhere.

Other customers’ numbers:

Swisscom went from 400 MongoDB VMs to 400 containers in 20 VMs.

This session was more of an educational session on how to approach the Docker journey for production. There was little information being shared re Docker Datacenter (the product).

Apcera: Moving legacy applications to containers

This session was interesting (in its own way) as it was a VERY Ops oriented talk re how you profile your legacy app and try to containerize it. Kudos for never mentioning Apcera throughout the talk.



Interesting session on orchestration (“promise based theory” etc) that concludes with a Chef Habitat demo.

<redacted comments>

Docker for Ops  

Breakout session on running docker in production.

A large part of the session is common sense CaaS / Docker Datacenter concepts (some of which have been covered in the keynote).

The message of “control” is explained in further details. RBAC is described.

Also the process of IT controlling the image life cycle is described. IT uploads “core” images into the local registry and Devs consume those core images and build their apps on top of them (yet uploading to the local registry).

UCP is the piece that actually controls the run-time and controls who can run and see what.

DTR is the piece that determines the authenticity of images and also determines who owns what and who gets access to what.

AWS: Deploying Docker Datacenter on AWS

In this session the AWS solutions engineer shows how to deploy Docker Datacenter on AWS.

Interesting: UCP nodes and DTR nodes leverage the auto-recovery EC2 feature (think vSphere HA) because they are “somewhat stateful” and so the recovery is easier if the instances recover with their original disk / IP address / personality etc.

That was an interesting point that made me think. If not even Docker can design/build from scratch a truly distributed brand-new application, it tells a lot about the challenges of “applications modernization”.

]]> 2 926
Serverlessconf 2016 – New York City: a personal report Tue, 07 Jun 2016 09:53:05 +0000 Warning: social media experiment ahead.

Two weeks ago I attended Serverlessconf in NYC. I’d like to thank the organization (Stan and the crew) for the tremendous job (and for inviting me to attend).

I originally wanted to write a (proper) blog post but then I figured that:

I have already written much of what . . . → Read More: Serverlessconf 2016 – New York City: a personal report]]>
Warning: social media experiment ahead.

Two weeks ago I attended Serverlessconf in NYC. I’d like to thank the organization (Stan and the crew) for the tremendous job (and for inviting me to attend).

I originally wanted to write a (proper) blog post but then I figured that:

  • I have already written much of what I wanted to say in an “internal report” that I shared with my team
  • I don’t really have time these days to write a (proper) blog post
  • The “internal report” doesn’t really have much of “internal” / confidential stuff anyway (just a few comments).

So why not sharing a copy/paste of that report?

I did indeed remove (see <redacted comments> throughout the post) some of the commentary I made in the original document because those were not meant to be public comments.

Below you will find some high level conclusions (first) and then some personal notes on a few sessions.

This is my way of giving back (a little bit) to the community I serve. I hope you will find it (somewhat) useful.

What you will read below this line is the beginning of my original report (and informal notes).


Executive summary and general comments

Event website:

Industry state of the art (according to what I see)

This event left me with the impression (or the confirmation) that there are two paces and speeds at which people are moving.

There is the so called “legacy” pace. This is often characterized by the notion of VMs and virtualization. This market is typically on-prem, owned by VMware and where the majority of workloads (as of today) are running. Very steady.

The second “industry block” is the “new stuff” and this is a truly moving target. #Serverless is yet another model that we are seeing emerging in the last few years. We have moved from Cloud (i.e. IaaS) to opinionated PaaS, to un-opinionated PaaS, to DIY Containers, to CaaS (Containers as a Service) to now #Serverless. There is no way this is going to be the end of it as it’s a frenetic moving target and in every iteration more and more people will be left behind.

This time around was all about the DevOps people being “industry dinosaurs”. So if you are a DevOps persona, know you are legacy already. You could feel this first hand on Twitter where DevOps experts where trying to minimize the impact (and viability) of Serverless while the Serverless proponents where all:

Where / when this will stop is hard to predict at this point.

What’s Serverless anyway?

The debate re what #Serverless is still on and it ranges from “oh well but we were doing these stuff 5 years ago but we called it something else” all the way to “this is futuristic stuff”.

I think it’s fair to look at AWS to define this market (as they have basically been the first to talk about these stuff) and so I’d suggest you read the “Keynote Day 1 (AWS)” section below carefully.

IMHO I see Serverless as some sort of “PaaS on steroids” (or should I say “PaaS on a diet”?)

I see Serverless being different compared to PaaS in two (major) areas:

  1. The unit of deployment. In PaaS you deploy an “application” (which could be as small as you want) whereas in Serverless you deploy a “function” (of an application) which is small by definition.
  2. The consumption pattern. PaaS is often pitched in the context of an application with an interface (called either by the user or another application). While this is one of the legit patterns for Serverless as well, the other pattern that is specific to Serverless is that the function is being triggered by external data-driven events (e.g. “run this function when this happens in the data”). This is the pattern I was focusing describing when I made a parallel between Lambda and Stored Procedures.

An interesting description of one of the Serverless vendors when I challenged him with “how isn’t this PaaS anyway?”:

It is sort-of like a PaaS. I usually avoid the term PaaS because (1) people might associate it with a traditional PaaS, where apps have a three-tier architecture with a DB offering already chosen for you and (2) PaaS usually includes more services around the execution bit.

<my prod> has a microservice architecture and is focused on the execution (the business logic) part of cloud apps. DB / persistence and other services are decoupled and no opinion is forced in that area”.

Serverless patterns

As suggested in the section above there are two emerging and re-occurring patterns that industry pundits are seeing WRT Serverless:

  • Write a function (or a set of those) and expose them via an API (in the AWS world this would be AWS Lambda + AWS API Gateway) that a user / program can interact with.
  • Write a function that is triggered by a data-driven event.


“Composable applications” (for lack of a better term)

This is not strictly related to Serverless (as described by industry pundits) but it’s interesting to see how much these new applications are being developed on top of Internet delivered laser focused services that provide application components that wouldn’t differentiate software products. Examples of these are Auth0 and Firebase (they were front and center during the entire conference, to the point that you could feel a good chunk of the audience would favorite them Vs equivalent AWS services such as Cognito and DynamoDB).

Basically your entire application consists for 80% of a mix/collage of third party web services that deliver specific value in a specific domain (e.g. authentication, data persistency, etc) and 20% of the “business logic” which defines what your application does. The key point is that only this 20% will differentiate you so that you can outsource commodity / generic services that are undifferentiating.

This pattern was very clearly described during the “10X Product Development” breakout session. See below. IMO the best session of the event.

This doesn’t have much to do with the concept of Serverless (as framed during the event) but nonetheless it was front and center for the whole 3 days (including the workshop – see below for more info on the workshop).

DevOps Vs NoOps

This has been debated throughout the conference. While the average consensus was that “with Serverless the need for Ops disappears” was strong, there were a few sessions during the event that underlined the importance of Ops (and DevOps).

It wasn’t however clear what the world of Ops would be in the Serverless world (as painted). The level of abstraction is so high in this model that other than having developers writing code/functions there is little left to be done. Surely there are challenges (e.g. designing proper application architectures) but typically all of these challenges are Dev challenges and not so much Ops challenges.

Monetization strategies

There were no such discussions during the conference. I will note though that all the players in this arena seems to be providing web services (where a tiered monetization strategy seem to be a viable option and somewhat accepted as a valid go to market). IMO the mood around software (in these contexts) is still very much 1) it must be free and 2) it must be open source.

Kelsey Hightower summarized it pretty well:

On the topic of (services) monetization, it is important to take note of the AWS march towards self-destroying and cannibalizing their lucrative revenue streams (arguably a strategy that is working very well for them).

Similarly to how they cannibalized (part of) S3 when they introduced Glacier, they are now cannibalizing the very lucrative EC2 product for a much cheaper compute alternative. There is no doubt that they are doing this for long term longevity and a stronger “lock-in” with their customers (it will be easier to leave AWS if you use EC2 than if you use Lambda). Having that said there are few vendors around that are taking better long term decision trading off short term revenue streams. Kudos to AWS in this sense.

Implications for Docker

This was another interesting aspect of the conference. Clearly Docker was being marginalized to a mere packaging format that everyone (except AWS?) is using in the backend as a mechanism to run the function code BUT it’s never ever exposed to the end-user. This is great news for Docker (the OSS project) but it’s disastrous news for Docker (Inc. the company).

I myself added to the cynic approach of the conference and I did shoot a couple of “bombs” (one of which was picked up by not less than Salomon in person):

<Redacted comments> 

Containers are more and more marginalized to being a run-time detail.

<Redacted comments> 


Serverlessconf was a full 2-day event fronted by a hands-on workshop (the day before).

I attended the workshop and I found it quite interesting. This was an instructor led lab whose documentation is still available on Github: (

[Note: the repository has been taken down after the workshop]

I do suggest that, anyone interested in the topic, goes through this hands-on workshop. Physical presence was a nice-to-have but not a strict requirement. Documentation is well organized and DYI is certainly possible.

The content is based on Lambda but it’s a good way to experiment first hand with some of the #serverless logic and patterns. Eveything suggested in the workshop could probably easily be achieved / applied to other #serverless implementations such as Azure Functions and Google Cloud Functions.

The workshop also includes some interesting “composability” patterns such as using external web services to off-load and simplify your own code (some services that have been leveraged include Auth0 and Firebase).

Other than the content of the workshop itself,  it was interesting to see some of the real-life challenges associated to consuming and compose third party services. For example the workshop was based on some Firebase consumption models associated to default accounts behaviours that literally changed overnight (before the workshop) so people creating new Firebase accounts for the workshop would experience a drift of behaviour from the documentation.

This underlines the challenges for these cloud services to honour backward compatibility contracts as well the challenges for consumers to deal with the fact that <things may change on the fly> and they need their code to cope with it.

All in all it was an interesting workshop.


<What follows are informal notes I took during some of the most interesting sessions>

Keynote Day 1 (AWS)

Tim Wagner (AWS GM for Lambda and API Gateway) starts by physically smashing a few (fake) servers on stage. That sets the tone.

Serverless allows “democratizing access to scale”. Not everyone is an expert in distributed systems hence Serverless represents the OS of the giant computer that the cloud represents.

DevOps is described as “the punch card of 2016”. The speaker was alluding to the ecosystem of skills required to do DevOps (and punch cards in the early days) Vs the transparent experience of #Serverless. There surely was also the intent of downplaying/denigrating DevOps as people pushing Serverless are trying to claim DevOps is not needed / irrelevant.

The speaker hints: “isn’t this like PaaS? No: PaaS is too curated, too intrusive but more than anything… it uses the wrong unit of scale (monolithic applications).”

I am wondering what Pivotal will have to say re this.

A hint from Andrew Clay Shafer:

Speaker makes the point that “the less in serverless is really server software, not server hardware”.

The Serverless manifesto is shown.

How do you run a 1-hour task?

Old school: 1 x 60 minutes task

New school: 60 x 1 minute tasks

He stresses on the “you don’t pay for idle” concept leveraging on the fact that it’s known that majority of compute capacity (in cloud or on-prem) is often idle (and you pay for that). This is indeed a known problem for wrongly sized EC2 instances that sit idle most of the time.

It is amazing to see how they have 0 problems cannibalizing EC2 (arguably what makes most of their revenue right now).

Lambda uses containers to run functions. They figured the challenging of scheduling, packaging (fast) user’s code etc.

In an effort to push Serverless and get more vendors on the same page, the speaker announces Flourish which is an effort to create an industry wide way to describe the app model and a way to group those different functions into a real application.

He points out this is not to go back to the “monolith” (single functions can still be updated independently etc).

More on Flourish here:

The speaker makes four predictions:

  • All data will stream (driven by speed requirements)
  • Natural language interfaces will permeate man-machine interaction
  • Serverless is going to confer economic advantages for the organizations that are embracing it (thus allowing those organizations to be differentiated)
  • Code stays in the cloud (Vs on the laptop and moved to the cloud eventually)

10X Product Development

This talk was also super interesting.

The narrative was about “how do you make your devs 10x more productive”.

He shows and then it shows the architecture: basically it’s a composition of a series of on-line services (firebase, algolia, auth0, cloudinary, etc).

It took 4 months, 2 developers, 13000 lines of code to build it.

95% of developer efficiency (developer efficiency is <how much time you worked on “business” code> Vs. <how much time you worked on non differentiating code>)

Second project:

Similar architecture. They use the same online services as before plus things like DocRaptor, Auth0 Webtasks.

Interesting comment: ironically, now the biggest chunk of code you are writing is front-end code (not back-end code, because MOST of the backend stuff are being delegated to external on-line services). There is going to be some glue that needs to happen between these services (and here you can use Serverless functions aka Lambda or even implement some of the logic in the front-end code if need be).

The speaker then talks about why they do not use AWS. AWS is about back-end processing, which they have largely outsourced (note: they outsourced to laser focused startups solving a single domain problem Vs to the counterpart AWS services).

It’s amazing how in this event startups like Auth0 and Firebase were seen as “the new cool thing” Vs there were seeds of AWS being started to be seen as the 800 pounds gorilla that doesn’t pay attention to users etc.

They need Auth0 etc more than they need “a place to execute code” (i.e. Lambda). And according to the speaker, Auth0 Webtasks are way more than enough if they need a Lambda-like solution.

AWS Serverless is complicated: to achieve a similar online service experience you need to collate 3 or 4 or 5 different AWS services.

The “Serverless Framework” ( is, in the opinion of the speaker, testament of how limited AWS Lambda is (if you need external libraries for a good experience than it means Lambda is too limited).


They started as a no SQL database. Then they added Authentication and “Hosting” capabilities.

It recently become a Suite of 50-ish features/services.

Firebase is positioned by Google as a “Backend as a Service” (between App Engine [PaaS] and Google Apps [SaaS]).

Now they show a demo that involves uploading a picture to Object Storage via Firebase and Firebase then runs a check against the Cloud Vision APIs and they render the picture on the website with the results from the Vision APIs.

(this was similar to the demo AWS did for Lambda at re-invent 2 years ago where they took a picture of the audience, upload it to S3 and a Lamnda function would create a thumbnail of that picture that landed on another S3 bucket).

“This is how you resize an image in a bucket” is becoming the new “This is how you can run WordPress in a Docker container”.


This talk was delivered by Charity Majors, one of the early employees of Parse (acquired and later shut down by Facebook).

Major was one of the few people talking about the importance of Ops. Her talk was also centered around the process people should go through to assess what services you could outsource and what services you should keep control of (surely the Parse experience taught her one thing or two).

Charity is starting to collect her thoughts in some blog posts if you are interested:


]]> 6 916
AWS Lambda: a few years of advancement and we are back to stored procedures Mon, 04 Apr 2016 16:38:00 +0000 Serverless computing is the new buzzword.

AWS describes Lambda (their implementation of Serverless) as the way how you’ll do things post containers.

Go figure how behind you are if you are head down learning Docker thinking it’s “the next big thing”. Sorry.

In order not to look too legacy, I decided to push on GitHub . . . → Read More: AWS Lambda: a few years of advancement and we are back to stored procedures]]> Serverless computing is the new buzzword.

AWS describes Lambda (their implementation of Serverless) as the way how you’ll do things post containers.

Go figure how behind you are if you are head down learning Docker thinking it’s “the next big thing”. Sorry.

In order not to look too legacy, I decided to push on GitHub a small experiment I built last year: that is a super short and simple Python program (that can be run as a Lambda function) that I had used to record (in a DynamoDB table) the status of the vCloud Air login service.

My use case was fairly simple: I did want to have a historical record of the up-time of the vCA login service (which was at that time experiencing some glitches I wanted to track) for trending analysis.

The code to make that happen was fairly trivial but having a VM (running that code) that saved data in a data base (running in the same or in a separate VM) seemed to be the traditional bazooka to shoot a fly considering the requirements.

Enter Lambda.

I had been to the latest two AWS re:Invent events and Lambda (which intrigues me considerably) was always front and center.

I am not sentimentally as involved as Ant Stanley but I, for one, love the idea and the principles behind Lambda. BTW this is a picture of Ant’s latest tattoo:

What’s interesting about Lambda (to me at least) isn’t so much about the fact that “you can focus on your code, upload just the bits, and let the back-end figure out how to run it for you”.

That is, to me, the traditional PaaS value proposition.

What intrigued me about Lambda is the fact that it’s “an extension of your data”. This changes the landscape quite dramatically and substantially.

In traditional PaaS world the code is the indisputable protagonist (oh, damn, and you also happen to need a persistent data service to store those transactions BTW).

With Lambda the data is the indisputable protagonist (oh and you also happen to attach code to it to build some logic around data).

A few years of advancement and we are back to stored procedures.

In all seriousness the Lambda example I have authored for my own testing is the exact opposite of an event driven approach (as described above). At best it’s a way to avoid the bazooka but, for what I wanted to achieve, a PaaS approach would have been more applicable (perhaps).

There also have been a lot of discussions as of late re the risk of being locked-in by abusing Serverless architectures (like Lambda).

There is some truth to it. This, however, isn’t due (too much) by how you write the code from a syntax perspective: while coding my Python program I noticed that when the function is run in the context of Lambda, the platform expects to pass a couple of parameters to the function (“event” and “context”). I had to tweak my original code to include those two inputs (even though I make no use of them in my program).

I am no Lambda expert but my take is that this isn’t a mere copy/paste of your own code into Lambda functions. Some tweaking may be required.

But this is still nothing compared to the level of complexity that will occur when you disintegrate your logic and attach it to data and/or events. At that point re-assembling your code/logic (that has been scattered all over the places) will be a gigantic effort.

So, IMO, the lock-in will not be a function of how different the syntax in your code will be Vs. running it on a platform you control (probably minimally different) but rather in how scattered and interleaved with other services your code will be (at scale).

Perhaps there will be a new meaning for “spaghetti code”?

I am not trying to say you should avoid using Lambda.

I also, for one, think that this whole lock-in thing is just rubbish.

There are so many people that waste so many energies in trying to avoid lock-in that ends up doing nothing: as a matter of fact, standing still is a way to avoid lock-in.


P.S. Yes, I know that it’s called “Serverless” but it doesn’t mean “there are no servers involved”. Are we really discussing this?

]]> 7 905