Reproducible Data Science with Pachyderm

Learn how to build version-controlled, end-to-end data pipelines using Pachyderm 2.0

Svetlana Karslioglu

BIRMINGHAM—MUMBAI

Reproducible Data Science with Pachyderm

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Publishing Product Manager: Reshma Raman

Senior Editor: David Sugarman

Content Development Editor: Nathanya Dias

Technical Editor: Rahul Limbachiya

Copy Editor: Safis Editing

Project Coordinator: Aparna Ravikumar Nair

Proofreader: Safis Editing

Indexer: Pratik Shirodkar

Production Designer: Sinhayna Bais

Marketing Coordinator: Shifa Ansari

First published: March 2022

Production reference: 1070222

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80107-448-3

www.packt.com

Contributors

About the author

Svetlana Karslioglu is a seasoned documentation professional with over 10 years of experience in top Silicon Valley companies. During her tenure at Pachyderm, she authored much of the open source documentation for Pachyderm and was also in charge of the documentation infrastructure. Throughout her career, she has spoken at local conferences and given talks advocating for open infrastructure and unbiased research in artificial intelligence. When Svetlana is not busy writing books, she spends time with her three children and her husband, Murat.

This book is dedicated to my husband, Murat, who supported me through each step of the process, tirelessly watched our three kids, and cooked breakfasts, lunches, and dinners while I was busy writing this book. My husband gave me the most valuable gift of all – time to fulfill my dreams. For this, I thank you, Muratjim.

About the reviewers

Soumo Chakraborty is a data architect at Kyndryl – Data & AI team. He helps clients continually succeed by using big data and AI services. He started 15 years ago as a trainee desktop engineer and life has been about continuous learning for him – leading to his current role. He is very passionate about learning new skills around the cloud, MLOps, AIOps, and data engineering, and putting them into business use cases. Soumo loves traveling and dreams of touring the world. During his leisure time, he and his 6-year-old watch YouTube videos of the Amazon rainforest and write tech blogs.

I dedicate this effort to my parents, who taught me to keep moving, my spouse, Richa, for teaching me to challenge new heights, Manoj, a mentor for life, and to all my well-wishers.

Manoj Palaniswamy is a senior technical staff member at Kyndryl and plays a technical leadership role within the Application, Data and AI practice. He has been working in the IT industry for more than 17 years and his domain expertise includes enterprise AI strategy, data management, AIOps, MLOps, AI platform engineering, and hybrid cloud IT infrastructure and analytics. He has led many cross-cultural technical teams across the globe on complex technical projects and closely works with clients to translate their business requirements into technical solutions. Manoj holds two patents in the area of machine learning and workload optimization on VMs.

Hendrik Vincent Koops is a senior data scientist at RTL Netherlands, working on AI multimedia projects. He received a B.A. degree in audio and sound design with honors in 2008 and an M.A. degree in music composition in 2008, both at the HKU University of the Arts Utrecht. He then received B.S. (2012) and M.S. (2014) degrees in artificial intelligence at Utrecht University. After completing research at the Department of Electrical and Computer Engineering at Carnegie Mellon University, he received a Ph.D. degree in computer science from Utrecht University in 2019. At RTL Netherlands, he is responsible for developing scalable audio-visual machine learning solutions to make video content more discoverable, searchable, and valuable.