Why you want Databricks Auto Loader
Auto Loader is one of the standout features in Databricks and this post will introduce you to why you'd want to use it to address common data ingestion challenges.
Feb 2022
Auto Loader is one of the standout features in Databricks and this post will introduce you to why you'd want to use it to address common data ingestion challenges.
This post is for anyone who is unaware that interactive Databricks clusters can be deleted 30 days after termination, unless the cluster is "pinned".
In part 4, the final part of this beginner’s mini-series of how to handle bad data, we will look at how we can retain flexibility to capture bad data and proceed uninterrupted.
In the 3rd instalment of this 4-part mini-series, we will look at how we can handle bad data using PERMISSIVE mode. It is the default mode when reading data using the DataFrameReader but there’s a bit more to it than simply replacing bad data with NULLs.
In the second part, we’ll continue to focus on the DataFrameReader class and look at the option, **DROPMALFORMED** to **remove** bad data.
Receiving bad data is often a case of “when” rather than “if”, so the ability to handle bad data is critical in maintaining the robustness of data pipelines.
One of the most common reasons to perform a restore is to do so for a table. In this post, we’ll be looking into how one of delta lake’s neat features allows us to accomplish fast and simple table restores to previous versions.
Secret redaction within Databricks is a great feature that helps to prevent exposure of your secrets unintentionally. This post will look at a short demo of why we need to remain cautious of secret exposure, even with secret redaction in place.
This short post looks at some considerations when using key vault secrets in Data Factory to securely pass information in pipeline activities. This is not an exhaustive list however but do take note.
This post shows how to quickly set up a managed identity for Databricks activities in Data Factory (ADF), to eliminate the need to manage credentials.
In this short post, we'll look at concurrency default values in ADF and implications of changing them or not.
In this post, we'll explore the Azure Data Factory (ADF) activity policy, it's configuration and default behaviour implications.
This article shows how to utilise the json editor and key vault secret references in Azure Data factory (ADF) to provide an alternative experience for linked service connectors that do not have built-in parameterisation support.