Launch Deal · 50% off 1st month ·

3.3 — Identify and Resolve Pipeline, Dataflow and Notebook Errors

DP-700 Domain 3 · Skill 3.3

What You Need to Know

Troubleshooting questions describe a symptom and ask for the most likely cause or the best fix. Work through it in order: where did it fail (which activity, query or cell), what does the error say, is it transient, and is it about permission, connection, data or code?

Pipelines

Symptom Likely cause Fix
Copy activity fails with a login or connection error Expired credentials, a gateway that's offline, or a firewall or private network Update the connection's credentials; check the gateway's status; use a VNet or on-premises gateway
It fails now and then and succeeds on rerun A transient network or source throttling problem Set Retry and a retry interval on that activity
"Column not found" or a type conversion error The source schema changed, or the mapping is stale Update the mapping or schema; turn on fault tolerance to log and skip bad rows
An activity times out The default timeout is too short, or a downstream step is stuck Raise the activity's timeout and look into the slow step
A later activity "didn't run" Its dependency condition wasn't met (On success after a failure) or it was skipped Check the dependency conditions; use On completion or On fail paths on purpose
You need to rerun only the broken part Use Rerun from failed activity in the run history

The run details show each activity's input and output JSON and the full error message. Start there.

Dataflow Gen2

Symptom Likely cause Fix
Refresh fails on one table Data errors (type conversion) or a source query error in that query Open the refresh history, find the failing table, and fix the step or replace the errors
"Couldn't write to destination" The destination schema doesn't match, or permissions on the destination item are missing Fix the column mapping and types, or use dynamic schema; check access to the destination
On-premises source fails The gateway is offline or not mapped to the connection Start and update the gateway, and map the connection
Refresh is slow or times out Large volumes, query folding is lost, or staging is misused Make sure the steps fold to the source; turn staging on for large intermediate results; or move heavy logic to a notebook

Notebooks and Spark job definitions

Symptom Likely cause Fix
OutOfMemoryError on the driver collect() or toPandas() on a large DataFrame, or a broadcast that's too large Keep processing distributed; write the result instead of collecting it; increase the driver size
Executor out of memory, or very slow stages Data skew (one huge partition), or too few or too many partitions Repartition or salt the skewed keys; let AQE handle skew; size the executors
ModuleNotFoundError The library isn't in the session's environment Add it to the attached Environment and publish (don't just %pip install in production)
Can't find a table or path A different (or no) default lakehouse is attached Attach the right lakehouse, or use the full ABFS path
A scheduled run fails but an interactive run works It runs as a different identity or without the interactive session's parameters Check the owner's permissions and the parameters cell; read the snapshot of the failed run
A job is stopped after hours The workspace's maximum job lifetime, or a session timeout Raise the limit, or split the job up

To diagnose a notebook, open the run snapshot (the notebook as it ran, with cell outputs), the Spark application page (jobs, stages, logs) and the Spark advisor's messages.

Key Concept: Read the actual error at the exact activity, query or cell. Transient errors need retries. Errors about permissions or connections need configuration. Errors about data or schema need a mapping or code fix.

Exam Traps

  • Adding retries to fix a permission or schema error (retries only help with transient failures)
  • Rerunning a whole long pipeline when Rerun from failed activity would do
  • Calling collect() on large data and then giving the driver more memory, instead of keeping the work distributed
  • Using %pip install in a scheduled notebook instead of an Environment

Check your understanding

Practice Scenario

🔒 This practice scenario is part of Skillmarked Pro

The lessons stay free. Pro unlocks the other 16 DP-700 practice scenarios and the 50-question timed mock exam, plus everything for every other exam.

See plans

Launch Deal: first month €9.95, then €19.90/month. Enter code LAUNCH50 at checkout. Ends 1 December.

Cancel anytime. Or €199 a year, which saves 17%.

Not sure yet? Try 10 free mock exam questions.

Already Pro? Log in.