Pachyderm ❤️ Spark ❤️ MLFlow - Scalable Machine Learning Provenance and Tracking
Tue Aug 23, 2022, by Enrico Rotundo, in Case Study, Mlops
This article shows how you can employ three frameworks to orchestrate a machine learning pipeline composed of an Extract, Transform, and Load step (ETL), and an ML training stage with comprehensive tracking of parameters, results and artifacts such as trained models. Furthermore, it shows how Pachyderm’s lineage integrates with an MLflow’s tracking server to provide artifact provenance.