2.8 — Spark Structured Streaming, KQL and Windowing Functions
DP-700 Domain 2 · Skill 2.8
What You Need to Know
This skill is code-level streaming. You process a stream with Spark Structured Streaming into Delta tables, process it with KQL in an eventhouse, and group events into time windows.
Spark Structured Streaming into a lakehouse
raw = (spark.readStream.format("kafka") # or "eventhubs", or files with a schema
.option("subscribe", "telemetry").load())
parsed = raw.select(F.from_json(F.col("value").cast("string"), schema).alias("e")).select("e.*")
agg = (parsed.withWatermark("eventTime", "10 minutes")
.groupBy(F.window("eventTime", "5 minutes"), "deviceId")
.agg(F.avg("temp").alias("avgTemp")))
(agg.writeStream.format("delta")
.outputMode("append")
.option("checkpointLocation", "Files/checkpoints/telemetry_5m")
.trigger(processingTime="1 minute")
.toTable("silver_telemetry_5m"))
| Concept | What to remember |
|---|---|
| checkpointLocation | Required. It stores offsets and state so that a restarted query resumes exactly where it stopped. Every query needs its own checkpoint |
| outputMode | append writes only new, final rows (needs a watermark for aggregations). update writes rows that changed. complete rewrites the whole result |
| trigger | processingTime="1 minute" runs micro-batches on a schedule. availableNow=True processes everything available and then stops, which suits scheduled incremental batches |
| withWatermark | How long to wait for late events before a window is finalized. Later events are dropped |
| foreachBatch | Runs batch code on each micro-batch, for example a Delta MERGE to upsert streaming changes |
Run long-lived streaming notebooks as a Spark job definition (or a scheduled notebook with a retry policy), so that they restart if they fail.
KQL processing in an eventhouse
- Update policy: when data lands in a source table, a KQL function transforms it into a target table. Use it to parse JSON, cast types and split one raw table into clean tables as the data arrives.
- Materialized view: an aggregation that's always up to date. For example,
summarize arg_max(Timestamp, *) by DeviceIdkeeps the latest row per device, which removes duplicates.summarize count() by bin(Timestamp, 1h)keeps hourly counts. - Query-time processing:
where,extend,project,summarize,join,parseandmv-expand. Putwherefilters first so that less data is scanned.
Telemetry
| where Timestamp > ago(1h) and Temp > 80
| summarize Readings = count(), MaxTemp = max(Temp) by DeviceId, bin(Timestamp, 5m)
| order by MaxTemp desc
Windowing functions
| Window | Behavior | Spark | KQL / eventstream SQL |
|---|---|---|---|
| Tumbling | Fixed size, no overlap, each event in exactly one window ("every 5 minutes") | window(ts, "5 minutes") |
bin(Timestamp, 5m) / TumblingWindow(minute, 5) |
| Hopping | Fixed size, moves forward by a smaller hop, so windows overlap ("the last 10 minutes, every 2 minutes") | window(ts, "10 minutes", "2 minutes") |
HoppingWindow(minute, 10, 2) |
| Sliding | Produces output only when an event enters or leaves the window | Not built in | SlidingWindow(minute, 10) |
| Session | Groups events that arrive close together and closes after a gap with no events | session_window(ts, "5 minutes") |
row_window_session() / SessionWindow(...) |
| Snapshot | Groups events that have exactly the same timestamp | Not built in | SnapshotWindow() |
In an eventstream, the no-code Group by transformation applies these windows without SQL.
Key Concept: In Spark, the checkpoint gives you exactly-once restarts and the watermark handles lateness. In KQL, update policies transform data and materialized views aggregate it. For windows, decide whether they may overlap (hopping) or not (tumbling).
Exam Traps
- Reusing one checkpoint folder for two streaming queries
- Using
completeoutput mode to append to a growing Delta table - Picking a tumbling window when the requirement says "the last 10 minutes, every minute" (that's hopping)
- Writing
whereaftersummarizein KQL when it could filter the raw rows first
Check your understanding
Practice Scenario
🔒 This practice scenario is part of Skillmarked Pro
The lessons stay free. Pro unlocks the other 16 DP-700 practice scenarios and the 50-question timed mock exam, plus everything for every other exam.
Launch Deal: first month €9.95, then €19.90/month. Enter code LAUNCH50 at checkout. Ends 1 December.
Cancel anytime. Or €199 a year, which saves 17%.
Not sure yet? Try 10 free mock exam questions.
Already Pro? Log in.
Sources
- Get streaming data into lakehouse with Spark structured streaming, Microsoft Learn
- Update policy overview (KQL), Microsoft Learn
- Materialized views (KQL), Microsoft Learn
- Process events using a SQL operator, Microsoft Learn
- Windowing functions (Stream Analytics query language), Microsoft Learn