Databricks Data Engineer Associate

The exam

45 scored questions · 90 min · pass mark not published · PySpark and SQL · May 2026 guide.

Weights

New names

Asset Bundles → Declarative Automation Bundles · Repos → Git Folders · Workflows → Lakeflow Jobs · DLT → Lakeflow pipelines.

Compute

Access modes: standard (was shared, multi-user) · dedicated (was single user; ML runtime, GPU, R, RDDs).

Cost: billed in DBUs. Serverless charges only while running; classic charges for uptime, so use autoscaling and auto-termination.

Delta Lake and Unity Catalog

Delta: ACID, time travel (VERSION AS OF, TIMESTAMP AS OF), DESCRIBE HISTORY, schema enforcement. Unity Catalog: catalog.schema.table, one governance layer for data and AI.

Pick the method

COPY INTO

Idempotent: already-loaded files are skipped. COPY_OPTIONS ('mergeSchema' = 'true') evolves the schema; 'force' = 'true' reloads.

Auto Loader

format("cloudFiles") + cloudFiles.schemaLocation + checkpointLocation; trigger(availableNow=True) runs as a batch. Directory listing (default) or file notification with file events (recommended, scales better). Bad values go to _rescued_data.

Semi-structured

payload:customer.id reads a path in a JSON string · from_json(col, schema) parses it · explode() gives one row per array element.

DataFrame moves

dropna, fillna, cast, withColumnRenamed, dropDuplicates([keys]), explode (row per array element), split, filter.

Joins and unions

F.broadcast(small) avoids shuffling the big side. union appends by position and keeps duplicates; unionByName matches names; SQL UNION dedupes, UNION ALL doesn't.

Aggregates

approx_count_distinct is fast and approximate. describe() gives count, mean, stddev, min, max; summary() adds percentiles.

Tuning defaults

Adaptive query execution is on by default.

Gold objects

Dedupe and upsert

Keep the latest row per key: QUALIFY ROW_NUMBER() OVER (PARTITION BY id ORDER BY ts DESC) = 1. Upsert: MERGE INTO t USING s ON … WHEN MATCHED THEN UPDATE … WHEN NOT MATCHED THEN INSERT ….

Expectations

CONSTRAINT c EXPECT (cond) warns (default) · add ON VIOLATION DROP ROW to drop or ON VIOLATION FAIL UPDATE to stop. Python: @dp.expect, expect_or_drop, expect_or_fail.

Run if

Control flow

If/else task (branch) · For each task (loop, concurrency default 1) · Run Job task · retries per task · repair run re-runs the failed tasks plus their dependents (you can pick which).

Triggers

Scheduled (cron) · file arrival (UC volume or external location) · table update · continuous · manual. Time-based for fixed timetables; data-driven for irregular arrival.

Git Folders

Branch, pull, commit and push in the workspace UI; pull requests happen in the Git provider.

Bundles

databricks.yml defines resources and targets (dev, test, prod). mode: development prefixes names and pauses schedules. Variables: ${var.name}, set per target, with --var, or BUNDLE_VAR_name.

bundle validate → bundle deploy -t prod → bundle run -t prod <job_key> · bundle destroy. In CI, run the CLI as a service principal, not a user.

Spark UI

Layout and upkeep

Liquid clustering (CLUSTER BY, CLUSTER BY AUTO) replaces partitioning and ZORDER; keys change without a rewrite. Predictive optimization runs OPTIMIZE, VACUUM, ANALYZE on UC managed tables.

Failures

Won't start: quota, init script, library install · driver OOM: collect() / toPandas() · executor OOM: skew or big partitions.

Monitoring jobs

Runs view: statuses, run times and failure rates against past runs · task graph shows the upstream task that blocked or failed.

Managed vs external

DROP on managed deletes data after 7 days (UNDROP until then); on external, files stay. External → managed: SET MANAGED (14-day UNSET MANAGED).

Privileges

Read a table: USE CATALOG + USE SCHEMA + SELECT. Grants inherit downwards. Grant to groups; run jobs as a service principal.

Row filters, masks, ABAC

ALTER TABLE t SET ROW FILTER f ON (col) · ALTER TABLE t ALTER COLUMN c SET MASK f, with is_account_group_member(). ABAC: governed tags + one policy on a catalog or schema covers every matching table.