What is deep work? Cal Newport, a computer science professor at Georgetown University, defines deep work as: “… a professional activity performed in a state of distraction-free concentration... These efforts create new value, improve your skill, and are hard to replicate.” If you want to do deep work, you need to focus for long…

Common engineering challenges and recipes for solutions — Introduction Over the last ten years or so, authoring and executing Spark jobs has become considerably simpler, mainly thanks to: High level APIs — which make it easier to express logic. Managed cloud-based platforms — highly scalable object storage and one click ephemeral clusters based on spot instances make it infinitely…

Implementing a data pipeline and a lightweight Deep Learning data lake using ClearML on AWS — Introduction Hour One is an AI-centric start-up, and its main product transforms text into videos of virtual human presenters. Generating realistic, smooth, and compelling videos of human presenters speaking and gesturing in multiple languages based on text alone is a challenging task, that requires training complex Deep Learning models — and…

Introduction In the past few years, machine learning data management practices have evolved dramatically, with the introduction of new design patterns and tools such as feature stores, data and model monitoring practices, and feature generation frameworks. Most advances in data management for machine learning are focused on classical (feature-based) data, and…

What led me here? A couple of years ago, I decided I needed a change. I was wrapping up 2.5 years as a VP of R&D at a medical imaging startup, building deep-learning-based solutions for the radiology domain. I was leading a group of 35 engineers and DL researchers, in a very complex and…

Deep learning is notorious for consuming large amounts of GPU resources during training. However, there are multiple parts within the Deep Learning workflow that require large amounts of CPU resources : Running large scale inference jobs Pre-processing input data — and materializing it on disk as preparation for training These…

This is part II of a two-part series, describing our solution for running distributed training on spot instances using TorchElastic and Kubernetes. Part I introduced our overall technology selection, design principles and benchmarks. In Part II, we will walk you through the process of creating a simplified version of the…

Deep Learning development is becoming more and more about minimizing the time from idea to trained model. To shorten this lead time, researchers need access to a training environment that supports running multiple experiments concurrently, each utilizing several GPUs. Until recently, training environments with tens or hundreds of GPUs were…

Great writeup, thanks for sharing. I am under the impression that the backwards pass is in fact the synchronization barrier. When your "main" worker is busy doing stuff like running the validation or writing checkpoints, for example at the end of the nth epoch, the other workers already start the next epoch. The other workers will complete exactly one mini-batch, and then get "stuck" since they will wait for the "main" worker to exchange gradients with them.