Processing Semantic Models Using Semantic Link In Microsoft Fabric

Introduction

I was excited when Microsoft introduced the ability to process semantic models in pipelines with a Semantic model refresh activity. It made life much easier not trying to time a semantic model refresh to match up with the completion of an ETL pipeline that may have run long for some reason.

There are some things about it that I do not care for though. Like many other activities in pipelines it requires you to create a connector that you have to maintain. I have also had random issues where the connectivity needs to be refreshed occasionally.

While I like playing with features that are available in Fabric, I need production pipelines and notebooks to be solid. After digging around a while ago, I found Microsoft's python library called Semantic Link which allows a user to interact in many ways with a semantic model using a python library called semantic-link-sempy.

Now when I build ETL pipelines, I no longer use Semantic Model Refresh activities. I instead opt into using a Notebook activity at the end of my pipelines to call a notebook that processes my models.

Introduction to Semantic Link

Semantic Link is a pretty cool library when needing to do different things with semantic models in Microsoft Fabric. One of the great things about it being built for Microsoft Fabric is it is by default included in the environment so there are no calls needed to install it when you run a notebook. Let us take a look at an example of processing a semantic model.

# Import Semantic Link
import sempy.fabric as fabric

# Set your workspace and model name
workspace = "Semantic Models - Dev"
ds = "Timekeeper"

# Refresh the model
fabric.refresh_dataset(dataset=ds, workspace=workspace, refresh_type="full")

One of the main things I will point out is the refresh_type parameter. There are different types of processing you can do to a semantic model. In this case we are using full. This will pull all data into a table if the table is set to import mode, process calculated columns, and process measures. Below is a list different processing types you can use.

Another type I use frequently is calculate. I mainly use it if I publish a model with some measure changes. If you do not process the model in any way, Power BI will likely break your visuals that use that measure and tell you you need to process the model. Instead of using full type, you can use calculate so that it fixes the error without importing all the data again.

You can check out more information on Microsoft Learn.

Processing Multiple Semantic Models

Now that you know how to process a semantic model using Semantic Link, we will take it a step further and process multiple at the same time. With python there are multiple ways to do this, but I tend to use the concurrent.futures library. It allows me to process multiple models at one time.

There is supposedly a way to capture the status of the models as they are processing by using this method as well, but I haven't had a need to go down that route yet.

import time
import sempy.fabric as fabric
from concurrent.futures import ThreadPoolExecutor

workspace = "Semantic Models - Dev"
datasets = ["SemanticModel1", "SemanticModel2", "CSS SemanticModel3"]
max_workers = 5

def start(ds):
    try:
        rid = fabric.refresh_dataset(dataset=ds, workspace=workspace, refresh_type="full")
        return ds, rid, None
    except Exception as e:
        return ds, None, str(e)

with ThreadPoolExecutor(max_workers=max_workers) as pool:
    results = list(pool.map(start, datasets))

Something you will need to keep in mind is if you are using lower capacities or have large models, you can run into capacity limits trying to process too many at the same time.

Conclusion

Semantic model refresh activity in pipelines gives you a way to process semantic models in Microsoft Fabric. The Semantic Link library in a notebook gives you a more reliable method of processing semantic models that can be extended through the use of Python.

I don't have a comments section yet, so feel free to send me feedback on this blog.