Study guidePro

Databricks Spark Developer Associate

Every domain of the Databricks Certified Associate Developer for Apache Spark exam explained for Spark 3.5, with PySpark and SQL side by side, diagrams, the defaults the questions test and the traps they set, following the October 2025 exam guide.

Aligned to the October 2025 exam guide · checked against the Databricks docs on 27 October 2026 · certlake.com/exam/spark-developer-associate/study-guide

Exam at a glance


Questions	45 scored, multiple choice
Time	90 minutes
Language	Python (PySpark)
Test aids	None, not even the API docs
Exam guide	October 2025 (live from 30 October 2025)
Spark version	Apache Spark 3.5 (see below)

Domain	Weight	≈ Questions
Apache Spark Architecture and Components	20%	9
Using Spark SQL	20%	9
Developing DataFrame/DataSet API Applications	30%	13–14
Troubleshooting and Tuning	10%	4–5
Structured Streaming	10%	4–5
Using Spark Connect to deploy applications	5%	2–3
Using Pandas API on Apache Spark	5%	2–3

1. Apache Spark Architecture and Components20% of the exam

Why Spark: one engine for SQL, streaming, machine learning and pandas-style code; it keeps data in memory between steps, optimises whole queries before running them, and recovers lost work from lineage. The challenges are the costs of moving data: shuffles, skewed keys, results pulled to the driver, Python UDF overhead and memory tuning.

The cluster

One driver per application. Executors are JVM processes on worker nodes that live as long as the application. Each core runs one task at a time, so two executors with 4 cores each run 8 tasks in parallel.

Component	Role
Driver	Runs your program and the `SparkSession`; plans the work, schedules tasks, collects results
Cluster manager	Allocates resources: standalone, YARN or Kubernetes (Mesos: deprecated since 3.2, removed in 4.0)
Executor	A JVM process per application on a worker node: runs tasks, keeps cached data, lives as long as the application
Core (slot)	Runs one task at a time, so 4 executors × 4 cores run 16 tasks at once

A 1-partition DataFrame runs as 1 task, however big the cluster.

Execution hierarchy

An action starts a job. Spark cuts the job into stages at each shuffle (a wide transformation such as groupBy, join or orderBy); narrow steps are pipelined inside one stage. Each stage runs one task per partition. AQE may coalesce the 200 shuffle partitions at run time.

Application → jobs (each action triggers at least one) → stages (split wherever data must be shuffled) → tasks (one per partition per stage).
Narrow transformations (select, filter, withColumn, union) stay within a partition and are pipelined into one stage. Wide ones (groupBy, join, distinct, orderBy, repartition) usually need a shuffle (a broadcast join doesn't): rows are redistributed by key, written to disk and read over the network.
Lazy evaluation: transformations only add to the plan; actions run it (count, collect, show, take, first, write, foreach). That lets the Catalyst optimizer combine steps and push filters down first.

Partitions

200 partitions after a shuffle (spark.sql.shuffle.partitions); files are read in splits of up to 128 MB (spark.sql.files.maxPartitionBytes). AQE coalesces small shuffle partitions at run time (domain 4).
repartition(n) or repartition("col"): full shuffle, raises or lowers the count; by column, rows are hashed by key, so a hot key stays skewed.
coalesce(n): merges partitions without a full shuffle, only lowers the count. Use it to write fewer files.

Caching and storage levels

from pyspark import StorageLevel

# lazy: the next action fills the cache
df.cache()
df.count()
df.unpersist()
other.persist(StorageLevel.DISK_ONLY)

DataFrame.cache() = persist() = MEMORY_AND_DISK_DESER in PySpark 3.5: what doesn't fit in memory goes to disk. RDD.cache() uses MEMORY_ONLY: what doesn't fit isn't cached and is recomputed. _2 levels replicate each partition on two nodes.
Cached partitions are evicted least-recently-used first; the Storage tab shows each one's storage level, partitions and size.
Memory: execution (shuffles, joins, sorts) and storage (cache) share spark.memory.fraction = 0.6 of (heap − 300 MB); execution can evict cached blocks. Garbage collection cost grows with the number of objects: high GC time (Executors tab) calls for serialized caching or smaller partitions.

Sessions, Datasets, modules, faults

SparkSession.builder … .getOrCreate() returns the existing session if there is one. spark.newSession() has its own SQL config, temp views and UDFs but shares the SparkContext. spark.stop() ends the application.
The typed Dataset API is Scala and Java only (a DataFrame is Dataset[Row]); Python has DataFrames.
Modules: Spark Core (scheduling, memory, RDDs), Spark SQL and DataFrames (Catalyst), Structured Streaming (on the Spark SQL engine), MLlib (pyspark.ml), pandas API on Spark.
Fault tolerance: a failed task is retried, and the job fails once the same task fails 4 times (spark.task.maxFailures); lost partitions are recomputed from lineage.

The rest of this guide is part of Pro

6 more domains, explained with snippets, decision tables and diagrams:

2. Using Spark SQL · 20%
3. Developing DataFrame/DataSet API Applications · 30%
4. Troubleshooting and Tuning · 10%
5. Structured Streaming · 10%
6. Using Spark Connect to deploy applications · 5%
7. Using Pandas API on Apache Spark · 5%

See Pro plans Read a full guide free (Data Engineer Associate)

Know it, or just recognise it?

The free diagnostic shows which domains you'd lose marks on, in about 10 minutes and without signing up.

Take the free diagnostic

Cheat sheet

The two-page cheat sheet: rules, defaults and traps for the last week before the exam.

Open the cheat sheet

Spotted something out of date? Email support@certlake.com. CertLake is not affiliated with Databricks, Inc.