Continuous AI on Azure
Luuk van der Velden & Rik Jongerius
Goal
MLOps seeks to deliver fresh and reliable AI products through continuous integration, continuous training and continuous delivery of machine learning systems. When new data becomes available, we update the AI model and deploy it (if improved) with DevOps practices. Azure DevOps pipelines support such practices and is our platform of choice. AI or Machine Learning is however focused around AzureML, which has its own pipeline and artifact system. Our goal is to combine DevOps with AzureML pipelines in an end-to-end solution. We want to continuously train models and conditionally deploy them on our…
Making Sense of Big Data
by Luuk van der Velden & Rik Jongerius
Databricks is an analytics Eco-system now available on most major cloud providers Google, AWS, and Azure. Databricks cluster computations use the distributed Spark engine. Recently introduced single-node Spark clusters do not support distributed computations, why?
Single-node Databricks clusters
On multi-node clusters a Python interpreter with PySpark runs on the driver node to collect results, while the worker nodes execute JVM jar files or Python UDFs. The option of running single-node clusters was introduced in October 2020 and is motivated as follows.
Standard Databricks Spark clusters consist of a driver node and one or more worker…
Kickstart AI was announced on the 10th of October 2019 to boost the development of Dutch AI- talent and technology through the collaboration of companies and universities. The goal is to boost the AI community and make the Netherlands a worldwide relevant AI-knowledge-hub. Kickstart AI acts in the context of national initiatives to grow our AI capabilities. But, how can the Netherlands become a worldwide AI leader as technical superiority seems out of the question?
A leader in the ethics of high tech
The Netherlands has been a worldwide leader in the ethical application of high technology in society, since the 1960s (article by TU Delft). The book…
Microsoft and Python Machine Learning: a modern love story, Part 2 of 2
Microsoft Azure is conquering our hearts as AI practitioners and wooing us with support for open-source frameworks such as PyTorch, Tensorflow and Scikit-learn on AzureML. Here we build a workflow around the tools that MS gives us and it is up to us to decide whether we are tempted. In part 1, we launched a Python script on our AzureML remote compute target with minimal fuss from our VSCode devcontainer. This also meant that most of the configuration of the remote environment was out of our hands. Here we will dive into AzureML Environment to configure a PyTorch GPU workload…
Microsoft and Python Machine Learning: a modern love story, Part 1 of 2
Microsoft is making great strides in conquering our hearts and mingling with the open source Ecosystem. The same ecosystem they openly worked against in the 00s. I confess I am in the process of being assimilated into the Microsoft world and am feeling good about it. In this blog I describe a development workflow for launching ML models in the cloud (AzureML) and developing code in VSCode remotely.
See also: Part 2: Unifying remote and local AzureML environments
1 VSCode ML devcontainer
Being fairly late to VSCode it took me by surprise after my usual period of denial. The remote development functionality introduced in…
