Launch Deal · 50% off 1st month ·

2.8 — Spark Structured Streaming, KQL and Windowing Functions

DP-700 Domain 2 · Skill 2.8

What You Need to Know

This skill is code-level streaming. You process a stream with Spark Structured Streaming into Delta tables, process it with KQL in an eventhouse, and group events into time windows.

Spark Structured Streaming into a lakehouse

raw = (spark.readStream.format("kafka")          # or "eventhubs", or files with a schema
       .option("subscribe", "telemetry").load())
parsed = raw.select(F.from_json(F.col("value").cast("string"), schema).alias("e")).select("e.*")

agg = (parsed.withWatermark("eventTime", "10 minutes")
       .groupBy(F.window("eventTime", "5 minutes"), "deviceId")
       .agg(F.avg("temp").alias("avgTemp")))

(agg.writeStream.format("delta")
    .outputMode("append")
    .option("checkpointLocation", "Files/checkpoints/telemetry_5m")
    .trigger(processingTime="1 minute")
    .toTable("silver_telemetry_5m"))
Concept What to remember
checkpointLocation Required. It stores offsets and state so that a restarted query resumes exactly where it stopped. Every query needs its own checkpoint
outputMode append writes only new, final rows (needs a watermark for aggregations). update writes rows that changed. complete rewrites the whole result
trigger processingTime="1 minute" runs micro-batches on a schedule. availableNow=True processes everything available and then stops, which suits scheduled incremental batches
withWatermark How long to wait for late events before a window is finalized. Later events are dropped
foreachBatch Runs batch code on each micro-batch, for example a Delta MERGE to upsert streaming changes

Run long-lived streaming notebooks as a Spark job definition (or a scheduled notebook with a retry policy), so that they restart if they fail.

KQL processing in an eventhouse

  • Update policy: when data lands in a source table, a KQL function transforms it into a target table. Use it to parse JSON, cast types and split one raw table into clean tables as the data arrives.
  • Materialized view: an aggregation that's always up to date. For example, summarize arg_max(Timestamp, *) by DeviceId keeps the latest row per device, which removes duplicates. summarize count() by bin(Timestamp, 1h) keeps hourly counts.
  • Query-time processing: where, extend, project, summarize, join, parse and mv-expand. Put where filters first so that less data is scanned.
Telemetry
| where Timestamp > ago(1h) and Temp > 80
| summarize Readings = count(), MaxTemp = max(Temp) by DeviceId, bin(Timestamp, 5m)
| order by MaxTemp desc

Windowing functions

Window Behavior Spark KQL / eventstream SQL
Tumbling Fixed size, no overlap, each event in exactly one window ("every 5 minutes") window(ts, "5 minutes") bin(Timestamp, 5m) / TumblingWindow(minute, 5)
Hopping Fixed size, moves forward by a smaller hop, so windows overlap ("the last 10 minutes, every 2 minutes") window(ts, "10 minutes", "2 minutes") HoppingWindow(minute, 10, 2)
Sliding Produces output only when an event enters or leaves the window Not built in SlidingWindow(minute, 10)
Session Groups events that arrive close together and closes after a gap with no events session_window(ts, "5 minutes") row_window_session() / SessionWindow(...)
Snapshot Groups events that have exactly the same timestamp Not built in SnapshotWindow()

In an eventstream, the no-code Group by transformation applies these windows without SQL.

Key Concept: In Spark, the checkpoint gives you exactly-once restarts and the watermark handles lateness. In KQL, update policies transform data and materialized views aggregate it. For windows, decide whether they may overlap (hopping) or not (tumbling).

Exam Traps

  • Reusing one checkpoint folder for two streaming queries
  • Using complete output mode to append to a growing Delta table
  • Picking a tumbling window when the requirement says "the last 10 minutes, every minute" (that's hopping)
  • Writing where after summarize in KQL when it could filter the raw rows first

Check your understanding

Practice Scenario

🔒 This practice scenario is part of Skillmarked Pro

The lessons stay free. Pro unlocks the other 16 DP-700 practice scenarios and the 50-question timed mock exam, plus everything for every other exam.

See plans

Launch Deal: first month €9.95, then €19.90/month. Enter code LAUNCH50 at checkout. Ends 1 December.

Cancel anytime. Or €199 a year, which saves 17%.

Not sure yet? Try 10 free mock exam questions.

Already Pro? Log in.