DevOps and Data Science: DataDevOps?

Published

Mar 28, 2019

Author

Dr. Phil Winder

CEO

I’ve seen a few posts recently about the emergence of a new field that is kind of like DevOps, but not quite, because it involves too much data. Verbally, about two years ago and in blog form about a year ago, I used the word DataDevOps, because that’s what I did. I develop and operate Data Science platforms, products and services.

But more recently I have read of the emergence of DataOps. Apparently enterprises have realised that it takes more than a PhD in Data Science to create products and value (not that I begrudge the value of a PhD, I have one, after all!). It also takes engineering. Specifically, software engineering, to perform a series of tasks that support the wafer-thin slice of the product cake that represents the Data Science model.

Local Jenkins Development Environment on Minikube on OSX

Published

Mar 11, 2019

Author

Dr. Phil Winder

CEO

Developing Jenkinsfile pipelines is hard. I think my world record for the number of attempts to get a working Jenkinsfile is around 20. When you have to continually push and run your pipeline on a managed Jenkins instance, the feedback cycle is long. And the primary bottleneck to developer productivity is the length of the feedback cycle.

Scikit Learn to Pandas: Data types shouldn't be this hard

Published

Feb 3, 2019

Author

Dr. Phil Winder

CEO

Nearly everyone using Python for Data Science has used or is using the Pandas Data Analysis/Preprocessing library. It is as much of a mainstay as Scikit-Learn. Despite this, one continuing bugbear is the different core data types used by each: pandas.DataFrame and np.array. Wouldn’t it be great if we didn’t have to worry about converting DataFrames to numpy types and back again? Yes, it would. Step forward Scikit Pandas.

7 Reasons Why You Shouldn't Use Helm in Production

Published

Jan 14, 2019

Author

Dr. Phil Winder

CEO

Helm is billed as “the package manager for Kubernetes”. The goal was to provide a high-level package management-like experience for Kubernetes. This was a goal for all the major containerisation platforms. For example, Apache Mesos has Mesos Frameworks. And given the standardisation on package management at an OS level (yum, apt-get, brew, choco, etc.) and an application level (npm, pip, gem, etc.), this makes total sense, right?

Using Data Science to block hackers

Published

Oct 28, 2018

Author

Dr. Phil Winder

CEO

Executive Summary

Winder.AI was engaged by Bitsensor to research and implement Data Science algorithms that could automate the detection and classification of web attackers. After gathering data, researching a Machine Learning solution and implementing Cloud-Native software, we delivered three new features:

• Tool classification - detect which automated tools were being used to perform the attack
• Attacker grouping - provide the capability of detecting distributed attacks by the same attacker
• Killchain classification - establish the phase of an attack (e.g. reconnaissance, exploitation, etc.)

Client

Bitsensor is a startup in the Netherlands that specializes in protecting public-facing websites and applications. They distribute their web-application firewall product to a range of customers throughout Europe. The goal is to provide an outstanding out-of-the-box experience that can protect exposed services from hackers, with little setup.

Bulding a Cloud-Native PaaS

Published

Oct 25, 2018

Author

Dr. Phil Winder

CEO

Executive Summary

Winder.AI worked with its partner, Container Solutions, to deliver core components of the Weave Cloud Platform-as-a-Service (PaaS).

• Kubernetes and Terraform implementations on Google Cloud Platform
• Delivered crucial billing components to track and bill for per-second usage
• Helped initiate, architect and deliver Weave Flux, a Git-Ops CI/CD enabler

Client

Weaveworks makes it fast and simple for developers and DevOps teams to build and operate powerful containerized applications. They minimize the complexity of operating workloads in Kubernetes by providing automated continuous delivery pipelines, observability and monitoring. Weaveworks also contributes to several open source projects, including Weave Scope, Weave Cortex and Weave Flux. It was one of the first members of the Cloud Native Computing Forum. Founded in 2014, the Company is backed by Google Ventures and Accel Partners. For more information, visit www.weave.works.

How Winder.AI Made Enterprise Cloud Migration Possible

Published

Oct 22, 2018

Author

Dr. Phil Winder

CEO

Executive Summary

• Truly global company, tens of thousands of staff across tens of regions.
• Problem: Colossal amounts of data, lack the computational flexibility to remain competitive.
• Solution: Cloud data platform leveraging Microservices, Serverless object storage and database technologies.
• Benefits: 4x faster, more memory and number of gpus compared to best on-premise hardware. 10x quicker time to market. 10 Petabytes of data.

A very large enterprise in the oil and gas industry asked Winder.AI to help them migrate mission critical workflows to the cloud and create competitive differentiators through the application of Data Science and providing AI consulting.

A Comparison of Serverless Frameworks for Kubernetes

Published

Sep 1, 2018

Author

Dr. Phil Winder

CEO

The term Serverless has become synonymous with AWS Lambda. Decoupling from AWS has two benefits; it avoids lock in and improves flexibility.

The misnomer Serverless, is a set of techniques and technologies that abstract away the underlying hardware completely. Obviously these functions still run on “servers” somewhere, but the point is we don’t care. Developers only need to provide code as a function. Functions are then used or consumed via an API, usually REST, but also through message bus technologies (Kafka, Kinesis, Nats, SQS, etc.).

This provides a comparison and recommendation for a Serverless framework for the Kubernetes platform.

How to Test Terraform Infrastructure Code

Published

Aug 22, 2018

Author

Dr. Phil Winder

CEO

Infrastructure as code has become a paradigm, but infrastructure scripts are often written and run only once. This works for simplistic infrastructure requirements (e.g. k8s deployments). But when there is a requirement for more varied infrastructure or greater resiliency then testing infrastructure code becomes a requirement. This blog post introduces a current project that has found tools and patterns to deal with this problem.

Cloud Native Data Science: Best Practices

Published

May 4, 2018

Author

Dr. Phil Winder

CEO

Following the Cloud Native best practices of immutability, automation and provenance will serve you well in a CNDS project. But working with data brings its own subtle challenges around these themes.