A Comparison of Serverless Frameworks for Kubernetes

The term Serverless has become synonymous with AWS Lambda. Decoupling from AWS has two benefits; it avoids lock in and improves flexibility.

The misnomer Serverless, is a set of techniques and technologies that abstract away the underlying hardware completely. Obviously these functions still run on “servers” somewhere, but the point is we don’t care. Developers only need to provide code as a function. Functions are then used or consumed via an API, usually REST, but also through message bus technologies (Kafka, Kinesis, Nats, SQS, etc.).

This provides a comparison and recommendation for a Serverless framework for the Kubernetes platform.

Edits:

• v1 02/09/18
• v2 02/09/18: Major edits to OpenFaas info thanks to @alexellisuk, Richard Gee, @kenfdev. Removed Serverless from comparison table thanks to @sebgoa.
• v3 10/09/18: Edits regarding Fn project thanks to @delabassee.

Comparison Table

Below is a table providing a high-level comparison of the different Serverless frameworks for k8s. They are segmented by popularity, stability, tooling, technology and useability.

The problem with opinions like popularity is that it can be hard to quantify. We have to resort to a proxy metic. For example, I’m really interested how many people use this framework on a day to day basis. But it is hard to find those numbers so we use a proxy metric like Github stars or Google searches. These are represented as <desired_metric (proxy_metric)>. See here for more information about project popularity.

Hence, please note that many of these metrics are gross estimates. Individual frameworks may appear better or worse than they really are. Read with a holistic view of all the evidence and features.

Comparison/Framework	OpenFaas	OpenWhisk	Kubeless	Fission	IronFunctions	Fn
Popularity (Github Stars)
Popularity (Relative google trends - 100 means most popular)	37	58	24	N/A (conflicts with fission)	N/A (conflicts with serverless)	3
Popularity (Total StackOverflow.com posts)	21	359	15	2	5	9
Stability (Contributors)
Stability (Contributors > 10 Commits)	10	33	7	6	9	19
Stability (Corporate Backer)	VMWare (1)	IBM (Apache Foundation) (3)	Bitnami	Platform9	https://iron.io	Oracle
Stability (Started)	Dec 2016	Feb 2016	Nov 2016	Aug 2016	Feb 2016	May 2016
Stability (Development Language)	Go	Scala	Go	Go	Go	Go
Tooling (Packaging mechanism)	Docker container	Docker container	Docker container	Docker container	Docker	Docker
Tooling (Deployment functions on k8s)	Manifest with custom yaml	Manifest with custom yaml	Manifest	Manifest with custom yaml	Proprietary	Proprietary
Tooling (Deployment via Serverless?)	Yes (WIP)	Yes	Yes	No	No	Yes
Technology (Underlying technology)	Alertmanager / Prometheus, Nats	CouchDB, Kafka, Nginx, Redis, Zookeeper	None (optional Nats or Kafka)	fluentd (optional Nats)	Postgres, Redis	DB (sqlite3, PostgreSQL, MySQL), MQ (Bolt, Redis), Prometheus
Usabillity (Worked out of the box?)	Yes	Yes	No (serverless plugin failed)	Yes (but annoyances)	Didn’t try	Didn’t try
Usability (Quality of documentation)	Good	Good	Average (not well organised)	Poor (not well organised, strange navigation and missing documentation)	Average (not well organised)	Good
Useability (Slack channel?)	Yes but through email	Yes	Yes (#kubeless channel of the k8s slack: http://slack.k8s.io)	Yes	Yes	Yes

1. Team of developers from VMWare work full time on OpenFaas.
2. Serverless included due to the recommendation for SDK.
3. Used in IBM’s Cloud Functions offering

Serverless Framework Recommendation

Using this table as a comparison, I can recommend:

• Using the Serverless framework as a software development kit (SDK)
• Using OpenFaas or OpenWhisk as the orchestrator of functions on k8s.
• OpenFaas is mature, easy to use and scalable, but when compared to OpenWhisk has fewer active developers on the core project (according to my definition of active developers) and is less popular (according to my choice of popularity metric).
• OpenWhisk is mature, backed by many active developers, popular, but is complex, written in Scala and backed by IBM/Apache (may be a good thing or a bad thing depending on your opinion)

The resulting technology stack looks like:

The additional recommendation of the Serverless framework allows developers to choose whether they want to deploy to lambda or to a serverless platform deployed on k8s. This helps migration if there are already functions on lambda.

Framework Notes

OpenWhisk

OpenWhisk is a mature Serverless framework supported by the Apache foundation and backed by IBM. OpenWhisk forms the basis of IBM’s Cloud Functions service. The main committers are IBM employees.

There are many underlying components, which adds to the complexity. It leverages CouchDB, Kafka, Nginx, Redis and Zookeeper. The upside is that the developers have clearly focused on scalability and resiliency. The downside is that Developers and Operators would require a working knowledge of these tools. Another downside is that they are duplicating features that are present in a orchestrator like Kubernetes (e.g. autoscaling). Functions are ultimately bundled into Docker containers that run alongside the framework.

OpenWhisk can be installed using a Helm chart but unfortunately requires some manual intervention. Applications can be deployed using the CLI or with the Serverless framework. Prometheus metrics are exported out of the box.

OpenFaas

OpenFaas is a popular (not quite as popular as OpenWhisk according to my estimates of popularity), easy to use Serverless framework. It is not as popular as OpenWhisk and commits are made on an individual basis. VMWare employs a team to work full time on OpenFaas, in addition to the great work performed by individual contributors in their spare time. A company, OpenFaas Ltd., has been incorporated in the UK, although it is unclear how this company relates to the project.

The architecture of OpenFaas is relatively simple. The API Gateway can be invoked synchronously or asynchronously via Kafka, SNS, CloudEvents, CRON and other triggers. Asynchronous invocations are handled by NATS Streaming. The autoscaling is performed using Prometheus and Prometheus Alertmanager; but thankfully it appears that this can be swapped out to use Kubernetes’ HorizontalPodAutoscaler.

Fully supported Kubernetes installer available via Helm or kubectl including an Operator allowing use of CRDs i.e. kubectl get functions. There is also a Kubernetes operator that is WIP, but I found that this worked well.

Applications can be deployed using the CLI or with the Serverless framework (a WIP). A “Function Store” is also provided, a curated list of functions to use with OpenFaas. Prometheus metrics are exported out of the box.

OpenFaaS ® is a registered trademark within the U.K.

Kubeless

I was really excited about Kubeless, the function-native framework for Kubernetes. It works by adding the idea of a “function” as a custom resource definition (CRD) in Kubernetes. Along with some clever code, this means that it turns Kubernetes into a function machine, with no add-on complexities like a messaging bus like the other frameworks.

I love the ability to manage functions like standard Kubernetes objects which means that all the usual Kubernetes goodies work out of the box (Helm, Ark, etc.).

Interaction is through the standard kubectl so no extra tooling and it has Serverless support built in.

It sounds perfect!

But unfortunately it just isn’t mature enough to be used in production yet. The community isn’t large enough, the documentation isn’t up to scratch (have to rely on blog posts) and there were bugs with the Serverless support which meant it didn’t work on EKS.

Given the positive fundamentals, I am sure that this will become the defacto Kubernetes serverless framework within the next six months.

Fission

Fission was interesting because it sits somewhere between Kubeless and OpenWhisk. It’s heavily dependent on Kubernetes features, but it’s not completely integrated. Then benefit of this approach is that it can take advantage of things that Kubernetes is good at, like autoscaling, but does something different to get better performance when needed. For example, it has a quite sophisticated cold-start pooling mechanism.

It is backed by Platform9 and can be installed via Helm. It uses Influxdb to handle state and provides FluentD for logging out of the box. It uses Nats for it’s message bus and Redis for caching. As you can see, other frameworks don’t provide caching and logging out of the box, even though it would be reasonably simple to add it.

Fission has a really nice extra called Fission Workflows. this is a tool that allows developers to compose functions. A la functional programming. This is a really interesting direction and I’m keen to see what can be done with it.

However, there are very few users (only two stack overflow questions - does that mean it’s easy to use?) and very few serious contributors - only 6 people have more than 10 commits. There was a few annoyances, but that’s mainly due to lack of users and developers. And the documentation was poorly organised. This made it hard to figure out how the framework worked. I also didn’t like how the code was responsible for templating and starting pods itself; this is likely to cause breakages in the future. Finally, I am very confused about Fission builders. I could not figure out what they were, or why they are required.

Also, the name makes it difficult to search for.

Fn

Another contender for the silly name crown is Fn. It’s Open Source but the main contributors work for Oracle. The main workflow uses the Fn CLI, but under the hood the functions use Docker containers. There is some information in this blog post and the documentation is here. And it is possible to deploy the framework components with Helm. There is also a new addition called Fn Flow that orchestrates multiple functions similar to OpenWhisk’s workflows.

But the most important difference is the way in which you work. Fn is focused on being easy to use, but this makes it very opinionated. It provides hot functions (all the other frameworks can provide this too) and streaming functions (this is more unique - not entirely clear how this would work with other frameworks).

It was started in 2016, which makes it as old as OpenWhisk and has a decent number of contributors. Despite this I was put off by the amount of opinion enforced on developers and this opinion doesn’t fit well with Kubernetes (you can’t deploy with Kubernetes manifests AFAICT). This was a requirement for me, so I didn’t try it. However, it is compatible with the Serverless framework which could mitigate this a little.

IronFunctions

Iron Functions is supported by a company of the same name. This added some initial complication since the github readme redirects you to the “documentation” which is actually the main page for Iron the company. From there if you click on “docs” then there is no mention of Iron Functions. The real docs are in the docs directory of the repository.

It’s Docker based, like many of the other frameworks and one interesting feature is that it has like-for-like support for AWS Lambda functions. You can take your code from Lambda and run it straight away on Iron; that’s nice for migration.

Unfortunately it doesn’t appear to have native support to deploy manifests to Kubernetes, like Fn, and it wasn’t supported by the Serverless framework. This put me of enough not to try it. It also wasn’t popular enough to appear on Google trends.

Severless Framework

I’ve mentioned this a lot through the article, so it deserves a section in its own right.

It is not a platform, it does run any functions. It is a software development kit for serverless. In fact, it is essentially just a packaging mechanism. But the beauty lies in the fact that once packaged in the Serverless way, you can deploy the same code to Lambda, Google Functions, Azure Functions, OpenWhisk, OpenFaas, Kubeless or Fn.

This portability is very attractive. It allows developers to build their code in standard way, but still allow standards, pricing, feature set or usability to decide where to deploy it.

Furthermore, it allows us to defer the choice of framework somewhat. I mentioned earlier that I like the direction that Kubeless is taking with respect to Kubernetes, but it is not mature enough for use. If we use Serverless then we can build code on OpenFaas or Lambda now, and easily port to Kubeless later.

The only downside is the King Kong of stupid names. Every. Single. Time. I have to postfix “Serverless the framework, not the technology”. That and the limited number of supported languages. But other than that, I think this is the safest option to take.

Funktion

Funktion was a defunct solutions from RedHat.