Launch Deal · 50% off 1st month ·

3.1 — Monitor Ingestion, Transformation and Semantic Model Refresh

DP-700 Domain 3 · Skill 3.1

What You Need to Know

Fabric offers several monitoring tools. Each answers a different question, and exam questions usually describe the question rather than name the tool.

Which tool answers which question

Question Tool
Did last night's pipeline, notebook, dataflow, Copy job or semantic model refresh succeed, and how long did it take? Monitor hub → Job runs: every job you can see, filterable by item type, status and workspace
Which activity in a pipeline run failed, and what did it read or write? The pipeline's run details (activity table, input and output JSON, Gantt view)
Why was a Spark job slow, and what happened on each stage and executor? The Spark application detail page (from the Monitor hub or the notebook), the Spark UI and the Spark history server, with driver and executor logs
Which warehouse queries were slow, frequent or failed? Query insights views: queryinsights.exec_requests_history, long_running_queries, frequently_run_queries
How much capacity (CU) is each item using, and are we being throttled? The Microsoft Fabric Capacity Metrics app (or Monitor hub → Capacity and cost)
Detailed logs for many items, queryable with KQL and kept for longer Workspace monitoring: a monitoring eventhouse that collects logs and metrics

Workspace monitoring

  • Turning it on creates a monitoring item with an eventhouse (KQL database). Supported items send logs and metrics to it.
  • Supported items include pipeline activity runs, Copy jobs, Dataflow Gen2, warehouse queries, semantic model operations, eventhouses, eventstreams and mirrored databases.
  • Collection starts when you turn it on. Earlier activity isn't backfilled.
  • Data is kept for 30 days by default. You can change the retention and caching policies.
  • Contributors and higher can query it with KQL, build dashboards and create alerts from queries. The monitoring database is read-only.

Monitor ingestion

  • Pipelines: the run history shows each activity's status, duration, rows read and written, and throughput. Fixes for failures are in lesson 3.3.
  • Copy jobs and mirroring: each has its own monitoring pane. Mirroring shows replication status and rows replicated per table.
  • Eventstreams: the Data insights tab shows incoming and outgoing events and errors per node. Runtime logs show warnings and errors.
  • Eventhouses: ingestion results and failures (.show ingestion failures, or the workspace monitoring tables).

Monitor transformation

  • Notebooks and Spark job definitions: the Spark application view shows jobs, stages, duration, idle time and the Spark advisor's recommendations.
  • Dataflow Gen2: the refresh history shows each query and table, its status and the error for each entity.
  • Warehouse T-SQL: Query insights, plus DMVs such as sys.dm_exec_requests for queries running now.

Monitor semantic model refresh

  • The refresh history in the model's settings, and the Monitor hub, list each refresh with its status, duration and error.
  • To start a refresh and follow its progress from code, use semantic link (sempy.fabric) in a notebook or the Power BI enhanced refresh REST API. Both return a refresh ID whose status you can poll.
  • In a pipeline, the Semantic model refresh activity runs a refresh as a step and waits until it finishes.
  • For Direct Lake models, Keep your Direct Lake data up to date reframes the model automatically when the Delta tables change. If it's off, you need an explicit refresh (reframe) after each load.

Key Concept: Monitor hub for did it run. Run and Spark detail pages for why it failed or was slow. Query insights for which SQL. Capacity Metrics for how much CU. Workspace monitoring for logs you can query.

Exam Traps

  • Looking for capacity consumption or throttling in the Monitor hub's job list instead of the Capacity Metrics app
  • Expecting workspace monitoring to show activity from before it was turned on
  • Using the Monitor hub to refresh a semantic model from code (use semantic link or the REST API)
  • Forgetting that a Direct Lake model with automatic updates off doesn't show new data until it's refreshed

Check your understanding

Practice Scenario

🔒 This practice scenario is part of Skillmarked Pro

The lessons stay free. Pro unlocks the other 16 DP-700 practice scenarios and the 50-question timed mock exam, plus everything for every other exam.

See plans

Launch Deal: first month €9.95, then €19.90/month. Enter code LAUNCH50 at checkout. Ends 1 December.

Cancel anytime. Or €199 a year, which saves 17%.

Not sure yet? Try 10 free mock exam questions.

Already Pro? Log in.