3.3 — Identify and Resolve Pipeline, Dataflow and Notebook Errors
DP-700 Domain 3 · Skill 3.3
What You Need to Know
Troubleshooting questions describe a symptom and ask for the most likely cause or the best fix. Work through it in order: where did it fail (which activity, query or cell), what does the error say, is it transient, and is it about permission, connection, data or code?
Pipelines
| Symptom | Likely cause | Fix |
|---|---|---|
| Copy activity fails with a login or connection error | Expired credentials, a gateway that's offline, or a firewall or private network | Update the connection's credentials; check the gateway's status; use a VNet or on-premises gateway |
| It fails now and then and succeeds on rerun | A transient network or source throttling problem | Set Retry and a retry interval on that activity |
| "Column not found" or a type conversion error | The source schema changed, or the mapping is stale | Update the mapping or schema; turn on fault tolerance to log and skip bad rows |
| An activity times out | The default timeout is too short, or a downstream step is stuck | Raise the activity's timeout and look into the slow step |
| A later activity "didn't run" | Its dependency condition wasn't met (On success after a failure) or it was skipped | Check the dependency conditions; use On completion or On fail paths on purpose |
| You need to rerun only the broken part | Use Rerun from failed activity in the run history |
The run details show each activity's input and output JSON and the full error message. Start there.
Dataflow Gen2
| Symptom | Likely cause | Fix |
|---|---|---|
| Refresh fails on one table | Data errors (type conversion) or a source query error in that query | Open the refresh history, find the failing table, and fix the step or replace the errors |
| "Couldn't write to destination" | The destination schema doesn't match, or permissions on the destination item are missing | Fix the column mapping and types, or use dynamic schema; check access to the destination |
| On-premises source fails | The gateway is offline or not mapped to the connection | Start and update the gateway, and map the connection |
| Refresh is slow or times out | Large volumes, query folding is lost, or staging is misused | Make sure the steps fold to the source; turn staging on for large intermediate results; or move heavy logic to a notebook |
Notebooks and Spark job definitions
| Symptom | Likely cause | Fix |
|---|---|---|
OutOfMemoryError on the driver |
collect() or toPandas() on a large DataFrame, or a broadcast that's too large |
Keep processing distributed; write the result instead of collecting it; increase the driver size |
| Executor out of memory, or very slow stages | Data skew (one huge partition), or too few or too many partitions | Repartition or salt the skewed keys; let AQE handle skew; size the executors |
ModuleNotFoundError |
The library isn't in the session's environment | Add it to the attached Environment and publish (don't just %pip install in production) |
| Can't find a table or path | A different (or no) default lakehouse is attached | Attach the right lakehouse, or use the full ABFS path |
| A scheduled run fails but an interactive run works | It runs as a different identity or without the interactive session's parameters | Check the owner's permissions and the parameters cell; read the snapshot of the failed run |
| A job is stopped after hours | The workspace's maximum job lifetime, or a session timeout | Raise the limit, or split the job up |
To diagnose a notebook, open the run snapshot (the notebook as it ran, with cell outputs), the Spark application page (jobs, stages, logs) and the Spark advisor's messages.
Key Concept: Read the actual error at the exact activity, query or cell. Transient errors need retries. Errors about permissions or connections need configuration. Errors about data or schema need a mapping or code fix.
Exam Traps
- Adding retries to fix a permission or schema error (retries only help with transient failures)
- Rerunning a whole long pipeline when Rerun from failed activity would do
- Calling
collect()on large data and then giving the driver more memory, instead of keeping the work distributed - Using
%pip installin a scheduled notebook instead of an Environment
Check your understanding
Practice Scenario
🔒 This practice scenario is part of Skillmarked Pro
The lessons stay free. Pro unlocks the other 16 DP-700 practice scenarios and the 50-question timed mock exam, plus everything for every other exam.
Launch Deal: first month €9.95, then €19.90/month. Enter code LAUNCH50 at checkout. Ends 1 December.
Cancel anytime. Or €199 a year, which saves 17%.
Not sure yet? Try 10 free mock exam questions.
Already Pro? Log in.
Sources
- Monitor pipeline runs, Microsoft Learn
- View refresh history and monitor your dataflows, Microsoft Learn
- Apache Spark application detail monitoring, Microsoft Learn
- Spark advisor, Microsoft Learn
- Workspace administration settings: jobs, Microsoft Learn