Monitor Synapse Notebooks from a Microsoft Teams Channel

Working with Spark Notebooks in Azure Synapse Spark can be challenging, especially when it comes to detecting errors in various types of for loops. While you may aim for a successful notebook execution, it’s often necessary to identify and address the functions that failed. One way to efficiently monitor notebook execution without failing the entire notebook or Synapse Pipeline is by sending these errors to a Teams channel. In this blog, we’ll explore how you can set up error monitoring in Azure Synapse Spark using Teams, allowing you to detect errors more effectively.

In my scenario, this approach proves to be useful as my notebook involves processing parquet files into delta tables, with most of the tasks running concurrently. While it’s acceptable for one or two entities to fail during this process, I still want my pipeline to proceed without halting, even if not all entities were processed successfully.

Receiving notifications in a Teams Channel proves to be very beneficial in this case!

How to create a webhook in MS Teams

To begin with, we need to create an incoming webhook on our desired Teams Channel.

  1. Open Microsoft Teams and navigate to the channel where you want to create the incoming webhook.
  2. Click on the three dots next to the channel name and select “Connectors” from the dropdown menu.
  3. In the Connectors screen, search for “Incoming Webhook” and select it.
  4. Give the incoming webhook a name and (optionally) an icon, then click “Create”.
  5. You’ll be taken to a screen with a unique URL for the incoming webhook. Copy this URL, as you’ll need it later.
  6. Click “Save” to save the settings and activate the incoming webhook.

You can now use the webhook URL to send notifications or messages to the channel.

Microsoft documentation: Create an Incoming Webhook – Teams | Microsoft Learn

Add a Python decorator function

With the code in a decorator function, we ensure that we implement the DRY (Don’t Repeat Yourself) programming principle. We will split the code into two functions.

import requests
def send_error_to_teams(error_message, webhook_url):
""" This function will use the requests library to make a POST request to the webhook URL with the error message as the payload """
payload = {
"text": error_message
response =, json=payload)
def send_error_to_teams_decorator(webhook_url):
""" This function takes a function as its argument and returns a new function that wraps the original function.
Inside the wrapper function, you can catch any errors that occur and send them to the Teams channel using the send_error_to_teams function """
def decorator(original_function):
def wrapper(*args, **kwargs):
result = original_function(*args, **kwargs)
return result
except Exception as e:
error_message = f"Error in {original_function.__name__}: {str(e)}"
send_error_to_teams(error_message, webhook_url)
print(f"Error send to teams channel: {error_message}")
return wrapper
return decorator

The first function posts our message on the Teams Channel webhook

The second function is a decorator function. In Python, a decorator is a special function that can be used to modify or extend the behavior of other functions. Decorators are essentially functions that take another function as an argument and then return a modified version of that function.

Sending messages to a Teams channel

To use the decorator we need to apply it to the function, we want to send the errors to our channel.

WEBHOOK_URL = "your_teams_webhook_url"
def my_function():
raise Exception("This try block always fails!")
except Exception as e:
raise ValueError(f"ValueError: {str(e)}")

It is important to raise a ‘ValueError’ when using a ‘try – except’ block in your functions. If not returned by the function, the exception will not be sent to your Teams channel.

Add the decorator to all specific functions that you would like to get notified for! You could even use the ‘send_error_to_teams’ function to write different messages to your Teams channel.

See the result here!

If you want to take this to the next level, take a look at Adaptive Cards: Types of cards – Teams | Microsoft Learn

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this:
search previous next tag category expand menu location phone mail time cart zoom edit close