Launch Deal · 50% off 1st month ·

1.7 — Orchestrate Processes with Pipelines and Notebooks

DP-700 Domain 1 · Skill 1.7

What You Need to Know

Orchestration decides what runs, in which order, when, and what happens on failure. Fabric gives you two orchestrators: pipelines (low-code control flow across many item types) and notebooks (code-first orchestration of other notebooks).

Pipeline or notebook?

Choose a pipeline when Choose a notebook (notebookutils) when
You coordinate different item types: Copy activity, Dataflow Gen2, notebooks, stored procedures, semantic model refresh Everything you run is a notebook and the logic is code
You want built-in retry, timeout, dependency conditions and alerts (Outlook or Teams activities) with no code You want the notebooks to share one Spark session to save startup time and capacity
You need a low-code copy from 100+ connectors, including on-premises sources through a gateway The order is data-driven (a DAG generated in Python)

Pipeline control flow

  • Dependency conditions between activities: On success (green), On fail (red), On completion (runs either way) and On skip. An activity with several incoming conditions runs only when all of them are met.
  • Activity settings: Retry count and retry interval, timeout, and Secure input/output to hide values in the run history.
  • Control activities: ForEach (sequential or parallel, with a batch count), If Condition, Switch, Until, Wait, Lookup, Get Metadata, Set variable, Invoke pipeline, Fail.
  • Notifications: put an Office 365 Outlook or Teams activity on the On fail path to alert engineers when a step fails.

Schedules and event-based triggers

  • Schedules run a pipeline (or a notebook, or another item) every n minutes or hours, daily, weekly or monthly, at specific times, in a time zone you pick. One item can have several schedules.
  • Event-based triggers start a pipeline when something happens. Examples are an Azure Blob Storage event (blob created or deleted) or a OneLake event (file created in a lakehouse folder). Fabric builds them with Activator and eventstreams, using Real-Time hub events.
  • To refresh something only after a notebook succeeds, chain the steps in one pipeline with On success. Don't schedule them separately and hope the timing works out.

Parameters and dynamic expressions

  • Pipeline parameters are set when a run starts and don't change during it. Variables can change during a run (Set variable).
  • Expressions start with @. Use @pipeline().parameters.myParam to get the typed value (an int stays an int). @{...} inserts the value into a string and always returns a string. Use @@ to write a literal @.
  • System variables include @pipeline().RunId, @pipeline().TriggerTime (UTC) and @pipeline().Pipeline.
  • Read activity output with @activity('Lookup1').output.value, for example as the Items of a ForEach that copies a list of tables.
  • To pass a value into a notebook, mark one notebook cell as the parameters cell, then set the base parameters on the Notebook activity. The notebook returns a value with notebookutils.notebook.exit(value).

Notebook orchestration

DAG = {"activities": [
    {"name": "stores", "path": "nb_dim_store"},
    {"name": "visits", "path": "nb_fact_visits", "dependencies": ["stores"],
     "retry": 1, "args": {"run_date": "2026-11-09"}}],
  "concurrency": 2}
notebookutils.notebook.runMultiple(DAG)
  • notebookutils.notebook.run(name, timeout, args) runs one notebook and returns its exit value as a string.
  • runMultiple runs a list in parallel, or a DAG where dependencies sets the order, with per-activity retry, args and timeoutPerCellInSeconds.
  • Only dependencies controls the order. The position of an activity in the list doesn't. validateDAG() catches cycles and missing names.

Key Concept: A pipeline orchestrates anything and gives you retries and alerts without code. runMultiple orchestrates notebooks cheaply in one shared session.

Exam Traps

  • Using On completion when the next step must run only if the previous one succeeded
  • Putting the retry on the downstream activity instead of the activity that fails
  • Writing @{pipeline().parameters.n} when the target needs an int (it returns a string)
  • Expecting the order of activities in a DAG list to set the run order (only dependencies does)
  • Scheduling a semantic model refresh at a fixed time instead of chaining it after the load succeeds

Check your understanding

Practice Scenario

A pipeline runs a Notebook activity that loads a lakehouse. If the notebook fails, the data engineers must get an email. If it succeeds, a Semantic model refresh activity must run. How should you connect the activities?

Report issue

Log in to save your progress.

You've finished the free part of DP-700

The lessons stay free. Pro unlocks the other 16 DP-700 practice scenarios and the 50-question timed mock exam, plus everything for every other exam.

See plans

Launch Deal: first month €9.95, then €19.90/month. Enter code LAUNCH50 at checkout. Ends 1 December.

Cancel anytime. Or €199 a year, which saves 17%.

Not sure yet? Try 10 free mock exam questions.