1.7 — Orchestrate Processes with Pipelines and Notebooks
DP-700 Domain 1 · Skill 1.7
What You Need to Know
Orchestration decides what runs, in which order, when, and what happens on failure. Fabric gives you two orchestrators: pipelines (low-code control flow across many item types) and notebooks (code-first orchestration of other notebooks).
Pipeline or notebook?
| Choose a pipeline when | Choose a notebook (notebookutils) when |
|---|---|
| You coordinate different item types: Copy activity, Dataflow Gen2, notebooks, stored procedures, semantic model refresh | Everything you run is a notebook and the logic is code |
| You want built-in retry, timeout, dependency conditions and alerts (Outlook or Teams activities) with no code | You want the notebooks to share one Spark session to save startup time and capacity |
| You need a low-code copy from 100+ connectors, including on-premises sources through a gateway | The order is data-driven (a DAG generated in Python) |
Pipeline control flow
- Dependency conditions between activities: On success (green), On fail (red), On completion (runs either way) and On skip. An activity with several incoming conditions runs only when all of them are met.
- Activity settings: Retry count and retry interval, timeout, and Secure input/output to hide values in the run history.
- Control activities: ForEach (sequential or parallel, with a batch count), If Condition, Switch, Until, Wait, Lookup, Get Metadata, Set variable, Invoke pipeline, Fail.
- Notifications: put an Office 365 Outlook or Teams activity on the On fail path to alert engineers when a step fails.
Schedules and event-based triggers
- Schedules run a pipeline (or a notebook, or another item) every n minutes or hours, daily, weekly or monthly, at specific times, in a time zone you pick. One item can have several schedules.
- Event-based triggers start a pipeline when something happens. Examples are an Azure Blob Storage event (blob created or deleted) or a OneLake event (file created in a lakehouse folder). Fabric builds them with Activator and eventstreams, using Real-Time hub events.
- To refresh something only after a notebook succeeds, chain the steps in one pipeline with On success. Don't schedule them separately and hope the timing works out.
Parameters and dynamic expressions
- Pipeline parameters are set when a run starts and don't change during it. Variables can change during a run (Set variable).
- Expressions start with
@. Use@pipeline().parameters.myParamto get the typed value (an int stays an int).@{...}inserts the value into a string and always returns a string. Use@@to write a literal@. - System variables include
@pipeline().RunId,@pipeline().TriggerTime(UTC) and@pipeline().Pipeline. - Read activity output with
@activity('Lookup1').output.value, for example as the Items of a ForEach that copies a list of tables. - To pass a value into a notebook, mark one notebook cell as the parameters cell, then set the base parameters on the Notebook activity. The notebook returns a value with
notebookutils.notebook.exit(value).
Notebook orchestration
DAG = {"activities": [
{"name": "stores", "path": "nb_dim_store"},
{"name": "visits", "path": "nb_fact_visits", "dependencies": ["stores"],
"retry": 1, "args": {"run_date": "2026-11-09"}}],
"concurrency": 2}
notebookutils.notebook.runMultiple(DAG)
notebookutils.notebook.run(name, timeout, args)runs one notebook and returns its exit value as a string.runMultipleruns a list in parallel, or a DAG wheredependenciessets the order, with per-activityretry,argsandtimeoutPerCellInSeconds.- Only
dependenciescontrols the order. The position of an activity in the list doesn't.validateDAG()catches cycles and missing names.
Key Concept: A pipeline orchestrates anything and gives you retries and alerts without code. runMultiple orchestrates notebooks cheaply in one shared session.
Exam Traps
- Using On completion when the next step must run only if the previous one succeeded
- Putting the retry on the downstream activity instead of the activity that fails
- Writing
@{pipeline().parameters.n}when the target needs an int (it returns a string) - Expecting the order of activities in a DAG list to set the run order (only
dependenciesdoes) - Scheduling a semantic model refresh at a fixed time instead of chaining it after the load succeeds
Check your understanding
Practice Scenario
A pipeline runs a Notebook activity that loads a lakehouse. If the notebook fails, the data engineers must get an email. If it succeeds, a Semantic model refresh activity must run. How should you connect the activities?
Log in to save your progress.
Sources
- Parameters for Data Factory in Microsoft Fabric, Microsoft Learn
- Run, schedule, or use events to trigger a pipeline, Microsoft Learn
- NotebookUtils notebook run and orchestration, Microsoft Learn
- Build event-driven pipelines with OneLake and Azure Blob Storage events, Microsoft Learn
- Decision guide: pipeline, dataflow or Spark, Microsoft Learn
You've finished the free part of DP-700
The lessons stay free. Pro unlocks the other 16 DP-700 practice scenarios and the 50-question timed mock exam, plus everything for every other exam.
Launch Deal: first month €9.95, then €19.90/month. Enter code LAUNCH50 at checkout. Ends 1 December.
Cancel anytime. Or €199 a year, which saves 17%.
Not sure yet? Try 10 free mock exam questions.