A Comparison of Serverless Frameworks for Kubernetes

Published

Sep 1, 2018

Author

Dr. Phil Winder

CEO

The term Serverless has become synonymous with AWS Lambda. Decoupling from AWS has two benefits; it avoids lock in and improves flexibility.

The misnomer Serverless, is a set of techniques and technologies that abstract away the underlying hardware completely. Obviously these functions still run on “servers” somewhere, but the point is we don’t care. Developers only need to provide code as a function. Functions are then used or consumed via an API, usually REST, but also through message bus technologies (Kafka, Kinesis, Nats, SQS, etc.).

This provides a comparison and recommendation for a Serverless framework for the Kubernetes platform.

How to Test Terraform Infrastructure Code

Published

Aug 22, 2018

Author

Dr. Phil Winder

CEO

Infrastructure as code has become a paradigm, but infrastructure scripts are often written and run only once. This works for simplistic infrastructure requirements (e.g. k8s deployments). But when there is a requirement for more varied infrastructure or greater resiliency then testing infrastructure code becomes a requirement. This blog post introduces a current project that has found tools and patterns to deal with this problem.

Cloud Native Data Science: Best Practices

Published

May 4, 2018

Author

Dr. Phil Winder

CEO

Following the Cloud Native best practices of immutability, automation and provenance will serve you well in a CNDS project. But working with data brings its own subtle challenges around these themes.

Cloud Native Data Science: Technology

Published

May 3, 2018

Author

Dr. Phil Winder

CEO

Technology choices in data-driven products are, as you would expect, largely directed by the type and amount of data. The first and most crucial decision to make is whether the data will be processed in a batch or streaming fashion.

Cloud Native Data Science: Strategy

Published

May 2, 2018

Author

Dr. Phil Winder

CEO

Data Science has become an important part of any business because it provides a competitive advantage. Very early on, Amazon’s data on book purchases allowed them to deliver personalised recommendations whilst customers were browsing their site. Their main competitor in the US at the time was Borders, who mainly operated in physical stores. This physicality prevented them from seamlessly providing customers with personalised recommendations [1]. This example highlights how strategic business decisions and data science are inextricably linked.

Life and Death Decisions: Testing Data Science

Published

Apr 25, 2018

Author

Abstract

We live in a world where decisions are being made by software. From mortgage applications to driverless vehicles, the results can be life-changing. But the benefits of automation are clear. If businesses use data science to automate decisions they will become more productive and more profitable.

So the question becomes: how can we be sure that these algorithms make the best decisions? How can we prove that an autonomous vehicle will make the right decision when life depends on it? How can we prove that data science works?

How to List all AMIs for each region in AWS

Published

Apr 3, 2018

Author

Dr. Phil Winder

CEO

A current project required a list of Amazon Machine Images (AMIs) for all regions for use in terraform. I couldn’t find a script to do this for me, so here you will find one that uses the aws cli, jq and a bit of Bash.

Principal Component Analysis

Published

Jan 28, 2018

Author

Dimensionality Reduction - Principal Component Analysis

Welcome! This workshop is from Winder.ai. Sign up to receive more free workshops, training and videos.

Sometimes data has redundant dimensions. For example, when predicting weight from height data you would expect that information about their eye colour provides no predictive power. In this simple case we can simply remove that feature from the data.

With more complex data it is usual to have combinations of features that provide predictive power. Dimensionality reduction methods attempt to apply a mathematical transformation to the data to shift the data into orthogonal dimensions. In other words they attempt to map data from complex combinations of dimensions (e.g. 3-D) into lower numbers of dimensions (e.g. 1-D).

Distance Measures with Large Datasets

Published

Jan 1, 2018

Author

Distance Measures for Similarity Matching with Large Datasets

Today I had an interesting question from a client that was using a distance metric for similarity matching.

A distance measure is the measure of how similar one observation is compared to a set of other observations. This is most commonly used in nearest-neighbour models, but is used as a measure of similarity across a range of clustering and classification algorithms.

603: Nearest Neighbour Tips and Tricks

Published

Jan 1, 2018

Author

Dimensionality and domain knowledge

• Is it right to use the same distance measure for all features?

E.g. height and sex? CPU and Disk space?

• Some features will have more of an effect than others due to their scales.

???

In this version of the algorithm all features are used in the distance calculation.

This treats all features the same. So a measure of height has the same effect as the measure of sex.