Databricks Data Engineer Professional

The exam

60 scored questions · 120 min · Python and SQL · October 2026 guide, live from 9 October 2026. Assumes the Associate basics.

Weights

New names

Asset Bundles → Declarative Automation Bundles · DLT → Lakeflow pipelines · APPLY CHANGES → AUTO CDC · Delta Sharing → OpenSharing · Repos → Git folders · Delta cache → disk cache.

Libraries

Never install PySpark on serverless; bump the version of a changed wheel.

UDFs

Built-in first · governed, from SQL anywhere → UC function (LANGUAGE SQL/PYTHON, EXECUTE) · fast Python → pandas UDF (Arrow) · per-group pandas → applyInPandas · spark.udf.register = this session only.

ST or MV

Append-only, once per row, low latency → streaming table. Must reflect later dimension changes, BI aggregates → materialized view. Updates/deletes in source → AUTO CDC.

AUTO CDC

KEYS · SEQUENCE BY (out-of-order events; no nulls) · APPLY AS DELETE WHEN · STORED AS SCD TYPE 1|2 · SCD 2 adds __START_AT, __END_AT (null = current) · TRACK HISTORY ON * EXCEPT (…). Python: dp.create_auto_cdc_flow.

Jobs

For each: {{input}}, concurrency default 1, no nesting · If/else on {{tasks.t.values.k}} · run-specific values from outside: Jobs API run-now with job_parameters.

Serverless jobs

Standard mode: cheaper, ~4–6 min startup · Performance optimized: time-sensitive · auto-optimization retries: turn off for at-most-once tasks · high memory (Preview): notebooks, 32 GB.

Tests

assertDataFrameEqual, assertSchemaEqual (pyspark.testing) · chain steps with df.transform(f) · unit = one function, synthetic data; integration = stages end to end.

Streaming state

foreachBatch is at least once: txnAppId + txnVersion = batchId for idempotent Delta writes; MERGE from a stream needs it. One checkpoint per query.

Sources

Raw files → binaryFile · Kafka value is binary: cast to string before from_json · Iceberg → CREATE TABLE … USING ICEBERG (managed) · many files → file notification with file events.

CDC

CDC feed → AUTO CDC · SQL Server, MySQL (Preview), PostgreSQL (Preview) → Lakeflow Connect: gateway (classic, continuous, to a staging volume) + pipeline (serverless); history tracking = SCD 2.

Sharing and federation

Windows

Top N → ROW_NUMBER() + QUALIFY · LAG/LEAD · 7-day → ROWS BETWEEN 6 PRECEDING AND CURRENT ROW · RANGE groups equal ORDER BY values · explode_outer keeps null arrays · filter(a, x -> x > 80).

VARIANT

parse_json (fails) vs try_parse_json (NULL) · v:a.b::double · variant_get(v, '$.a', 'string') · DBR 15.4+ · not a clustering, partition or Z-order key; can't be grouped or sorted.

AI functions

Any prompt → ai_query(endpoint, prompt) · fixed tasks → ai_analyze_sentiment, ai_classify, ai_extract, ai_summarize · not on classic warehouses.

Expectations

Warn (default, kept) · DROP ROW / expect_or_drop · FAIL UPDATE / expect_or_fail · expect_all_or_drop: any failed rule drops · quarantine = inverted rules to a second table.

System tables

Signals

Pipeline health and expectations → event log, flow_progress · run failed or slow → job notifications · stream behind → backlog threshold · metric threshold → SQL alert · gate a job on it → SQL alert task + If/else · from a script → SDK jobs.list_runs.

Layout

Clustering ✗ partitioning or Z-order · OPTIMIZE incremental, OPTIMIZE FULL reclusters all.

Upkeep

Predictive optimization: OPTIMIZE, VACUUM, ANALYZE on managed tables only; default for accounts from 11 Nov 2024; ALTER CATALOG … ENABLE PREDICTIVE OPTIMIZATION.

Cache and CDF

Disk cache: local copies, sees changed files · df.cache(): may go stale. CDF: delta.enableChangeDataFeed, table_changes(t, 5, 10), _change_type incl. update_preimage/postimage; with the (legacy) property, only changes after enabling.

Query profile

No skipping → cluster on filter (stats on first 32 columns) · sort-merge with small side → broadcast · spill → more partitions · one slow task → skew (AQE, salting).

Controls

Jobs: run and cancel only → CAN MANAGE RUN · ABAC: account-level governed tags + policy on catalog/schema covers future tables; needs MANAGE (or owner) + EXECUTE · Data Classification auto-tags PII → ABAC mask.

Techniques

Hash (irreversible, joinable) · tokenize (reversible via vault) · suppress · generalize · mask.

Erasure

DELETE → REORG TABLE … APPLY (PURGE) → VACUUM after retention (7 days default).

Inheritance and metadata

Catalog or schema grant covers current and future objects; still needs USE CATALOG + USE SCHEMA · create table = those + CREATE TABLE · COMMENT ON, SET TAGS; column tags don't inherit · reports from information_schema.

Diagnose

Skew, spill → Spark UI stages · won't start → compute event log · collect() crash → driver OOM · ConcurrentModificationException → conflicting writers · checkpoint-incompatible change → new checkpoint.

Repair and deploy

Repair run: unsuccessful tasks + dependants, current settings, parameter overrides. bundle deploy -t prod --var="k=v" · BUNDLE_VAR_k · CI as a service principal · mode: development prefixes names, pauses schedules.

Models

Cluster facts on filter/join keys · optimized writes + auto compaction · star or snowflake, avoid 3NF · MVs precompute · metric views define measures once: SELECT d, MEASURE(m) … GROUP BY d.