Sign in

Luuk van der Velden
Python Data Engineer @ (we’re hiring in NL)

Continuous AI on Azure

Luuk van der Velden & Rik Jongerius


MLOps seeks to deliver fresh and reliable AI products through continuous integration, continuous training and continuous delivery of machine learning systems. When new data becomes available, we update the AI model and deploy it (if improved) with DevOps practices. Azure DevOps pipelines support such practices and is our platform of choice. AI or Machine Learning is however focused around AzureML, which has its own pipeline and artifact system. Our goal is to combine DevOps with AzureML pipelines in an end-to-end solution. We want to continuously train models and conditionally deploy them on our…

Making Sense of Big Data

by Luuk van der Velden & Rik Jongerius

Databricks is an analytics Eco-system now available on most major cloud providers Google, AWS, and Azure. Databricks cluster computations use the distributed Spark engine. Recently introduced single-node Spark clusters do not support distributed computations, why?

Single-node Databricks clusters

On multi-node clusters a Python interpreter with PySpark runs on the driver node to collect results, while the worker nodes execute JVM jar files or Python UDFs. The option of running single-node clusters was introduced in October 2020 and is motivated as follows.

Standard Databricks Spark clusters consist of a driver node and one or more worker…

Kickstart AI was announced on the 10th of October 2019 to boost the development of Dutch AI- talent and technology through the collaboration of companies and universities. The goal is to boost the AI community and make the Netherlands a worldwide relevant AI-knowledge-hub. Kickstart AI acts in the context of national initiatives to grow our AI capabilities. But, how can the Netherlands become a worldwide AI leader as technical superiority seems out of the question?

Dutch tulips
Dutch tulips
Photo by jamal . from Pexels

A leader in the ethics of high tech

The Netherlands has been a worldwide leader in the ethical application of high technology in society, since the 1960s (article by TU Delft). The book…

Microsoft and Python Machine Learning: a modern love story, Part 2 of 2

Microsoft Azure is conquering our hearts as AI practitioners and wooing us with support for open-source frameworks such as PyTorch, Tensorflow and Scikit-learn on AzureML. Here we build a workflow around the tools that MS gives us and it is up to us to decide whether we are tempted. In part 1, we launched a Python script on our AzureML remote compute target with minimal fuss from our VSCode devcontainer. This also meant that most of the configuration of the remote environment was out of our hands. Here we will dive into AzureML Environment to configure a PyTorch GPU workload…

Microsoft and Python Machine Learning: a modern love story, Part 1 of 2

Microsoft is making great strides in conquering our hearts and mingling with the open source Ecosystem. The same ecosystem they openly worked against in the 00s. I confess I am in the process of being assimilated into the Microsoft world and am feeling good about it. In this blog I describe a development workflow for launching ML models in the cloud (AzureML) and developing code in VSCode remotely.

See also: Part 2: Unifying remote and local AzureML environments

1 VSCode ML devcontainer

Being fairly late to VSCode it took me by surprise after my usual period of denial. The remote development functionality introduced in…

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store