Launch Deal · 50% off 1st month ·

2.7 — Streaming Loads, Engines and Eventstreams

DP-700 Domain 2 · Skill 2.7

What You Need to Know

Streaming data arrives continuously and has to be processed within seconds or minutes, not in nightly batches. Fabric has three streaming engines. They often work together: an eventstream ingests and routes the events, then an eventhouse (KQL) or Spark Structured Streaming processes them.

Choose a streaming engine

Engine Code Strengths Pick it when
Eventstream No code (plus an optional SQL operator) Connectors for Event Hubs, IoT Hub, Kafka, CDC sources, Fabric and Azure events; light transformations; sending one stream to several destinations You need to capture and route events with simple filters and windows, without code
Eventhouse / KQL KQL Ingests very high volumes, queries time series in under a second, update policies, materialized views, Real-Time Dashboards Interactive near-real-time analytics on telemetry and logs
Spark Structured Streaming PySpark or Scala Complex logic, joins with lakehouse tables, any library, writes Delta tables A code-first team needs streaming transformations into a lakehouse

Eventstream building blocks

  • Sources: Azure Event Hubs, IoT Hub, Event Grid, Service Bus, Kafka (Apache, Confluent, Amazon MSK), Amazon Kinesis, Google Pub/Sub, MQTT, CDC from Azure SQL, PostgreSQL, MySQL, Cosmos DB and SQL Server on a VM, a custom endpoint (your own app or a Kafka client), sample data, and Fabric and Azure events (OneLake, job, workspace item, Blob Storage).
  • Transformations: Filter, Manage fields (add, remove, rename, cast), Aggregate, Group by (with time windows), Union, Expand (one row per array element), Join (two streams), and a SQL operator for code.
  • Destinations: eventhouse (direct ingestion, or event processing before ingestion when transformations are needed), lakehouse (Delta tables), Activator, derived stream (a transformed stream you can route on or share in the Real-Time hub), custom endpoint, Spark notebook.
  • One eventstream can feed several destinations at once without them interfering with each other.

A streaming loading pattern

  1. Capture: the source sends events to an eventstream (or straight to an eventhouse).
  2. Shape in flight: filter out noise, cast types, and aggregate into windows if consumers don't need raw events.
  3. Land raw and refined: send raw events to an eventhouse table (or a lakehouse bronze table), and send the transformed stream to its own table.
  4. Transform at ingestion: in an eventhouse, update policies transform each newly ingested batch into a target table, and materialized views keep aggregates (such as the latest row per device) up to date.
  5. Act: send the conditions you care about to Activator for alerts, or show them on a Real-Time Dashboard.

Ingestion details worth knowing

  • An eventhouse destination with direct ingestion writes events as they are, with no processing.
  • A lakehouse destination needs the stream's fields to fit a table schema. Eventstream converts the events to Delta.
  • Kafka-compatible endpoints let existing Kafka producers send to an eventstream without code changes.

Key Concept: An eventstream moves and lightly shapes events. An eventhouse stores and queries them in near real time. Spark Structured Streaming transforms them with code into Delta.

Exam Traps

  • Building a polling pipeline every minute instead of an eventstream for continuous events
  • Choosing Spark for a team that wants no-code stream routing
  • Forgetting that an eventstream destination with transformations needs processing before ingestion (or a derived stream)
  • Using a pipeline schedule to feed a Real-Time Dashboard

Check your understanding

Practice Scenario

🔒 This practice scenario is part of Skillmarked Pro

The lessons stay free. Pro unlocks the other 16 DP-700 practice scenarios and the 50-question timed mock exam, plus everything for every other exam.

See plans

Launch Deal: first month €9.95, then €19.90/month. Enter code LAUNCH50 at checkout. Ends 1 December.

Cancel anytime. Or €199 a year, which saves 17%.

Not sure yet? Try 10 free mock exam questions.

Already Pro? Log in.