Databricks Spark Developer Associate

The exam

45 scored questions · 90 min · Python (PySpark) · no docs allowed · October 2025 guide. Written for Spark 3.5; 4.x marks a change in Spark 4.0.

Weights

DataFrame API 30 · Architecture 20 · Spark SQL 20 · Troubleshooting 10 · Streaming 10 · Spark Connect 5 · pandas API 5 (%).

Spark 4.0 changes

ANSI mode on by default (bad cast errors, not NULL; overflow errors, not wraps; use try_cast) · Mesos removed · pandas-on-Spark ops_on_diff_frames on, errors under ANSI · to_koalas removed.

Components

Application → job per action → stages split at shuffles → task per partition per stage. A 1-partition DataFrame = 1 task.

Narrow vs wide

Narrow (same stage): select, filter, withColumn, union. Wide (usually a shuffle, new stage): groupBy, join, distinct, orderBy, repartition. Transformations are lazy; actions run: count, collect, show, take, first, write, foreach.

Partitions

Shuffle partitions 200 · file read split 128 MB · task fails 4× → job fails. repartition(n): full shuffle, up or down · coalesce(n): no full shuffle, down only.

Caching

DataFrame cache() = persist() = MEMORY_AND_DISK_DESER (3.5); RDD.cache() = MEMORY_ONLY (doesn't fit → recomputed). Lazy until an action. LRU eviction. GC cost grows with object count (Executors tab). Datasets: Scala/Java only.

Save modes

Not atomic. partitionBy + overwrite is static by default: all old partitions deleted. Option partitionOverwriteMode = dynamic replaces only written ones.

Sources

• Default format Parquet. spark.read.format("csv") then .option("header", True), .load(p).
• SQL on files: SELECT * FROM parquet.`/path` (also orc, json, csv, text, delta). CSV this way: header read as data, columns _c0, _c1 as strings.
• JDBC: partitionColumn + lowerBound + upperBound + numPartitions read in parallel. Bounds set the stride only: no rows filtered.

Tables and views

Managed (no path): DROP deletes data. External (path/LOCATION): DROP keeps data. Temp view: one session. Global temp view: global_temp.v, whole application.

SQL to read

WHERE before grouping, HAVING after. Nulls first in ASC, last in DESC. COUNT(col) skips nulls. INSERT OVERWRITE replaces.

Columns and rows

withColumn, withColumnRenamed, drop, select(col.alias()), toDF(*names). F.split(c, " ").getItem(0). filter = where; combine with & | ~ and parentheses. F.when(...).otherwise(...). explode drops null/empty arrays; explode_outer keeps them.

Inspect and collect

Row.asDict(), printSchema(), orderBy(F.col("a").desc()).

Clean and aggregate

• na.drop() any null · how="all" all null · subset=[...] · thresh=n (overrides how) · na.fill({...}). dropDuplicates(["id"]) keeps one row per id.
• groupBy().agg(F.sum, F.avg, F.countDistinct). approx_count_distinct = HyperLogLog, rsd 0.05. describe(); summary() adds 25/50/75%.

Dates

from_unixtime(secs) → yyyy-MM-dd HH:mm:ss string (session time zone) · unix_timestamp reverse · to_date, date_format(d, "MMM d, yyyy") · year, month, dayofweek (1 = Sunday). MM month, mm minute.

Joins and unions

how: inner (default), left, right, full, left_semi, left_anti, cross. Keys as a list → one key column; as an expression → both. F.broadcast(small) copies it to every executor. union = unionAll: by position, keeps duplicates. unionByName(allowMissingColumns=True).

UDFs and shared variables

• F.udf(f) with no type returns StringType. Python UDFs are opaque to the optimizer: prefer built-ins, then pandas UDFs. spark.udf.register for SQL.
• Stateless UDF: per row. Running state across a stream: applyInPandasWithState (experimental; state store).
• Broadcast variable: read-only, cached per executor. Accumulator: tasks add, driver reads; exact only in actions, may double-count in transformations. Both need SparkContext.

Windows

Window.partitionBy("d").orderBy("t") · row_number 1,2,3 · rank gaps · dense_rank none · lag/lead. With orderBy, default frame = RANGE unbounded preceding → current row: ties get the same running total.

AQE (on by default since 3.2)

Re-plans at shuffle boundaries: coalesces small partitions, switches sort-merge to broadcast join, splits skewed partitions (> 5× median and > 256 MB). Advisory partition size 64 MB.

Joins and skew

• Auto broadcast under 10 MB (autoBroadcastJoinThreshold under spark.sql.); -1 disables. Hint requests it (not guaranteed); priority BROADCAST > MERGE > SHUFFLE_HASH.
• Skew: one task ≫ median in the Stages tab. Fix: AQE skew join, broadcast, salt the key.

Plans and UI

explain() = physical only; explain(True) = all plans; modes simple, extended, codegen, cost, formatted. Exchange = shuffle. UI on port 4040; history server needs spark.eventLog.enabled. Executors tab: GC time, stderr/stdout logs (also per task on a stage's page).

Model

Unbounded table, micro-batches by default, incremental results. Exactly-once = replayable source + checkpoint (WAL) + idempotent sink. File source needs a schema. Socket and rate: testing only.

Output modes and sinks

File sink: append, exactly-once. Kafka, foreach: at-least-once. foreachBatch(df, batch_id): any target, your idempotency.

Triggers

Default: next batch when the last ends · processingTime="1 minute" · availableNow=True: all data, then stop · once: deprecated · continuous: experimental, at-least-once.

Windows and dedup

window("ts", "10 minutes") tumbling; add "5 minutes" slide → sliding. Watermark = max event time − delay; state behind it dropped. dropDuplicates without watermark: state grows forever; with watermark include the event-time column; dropDuplicatesWithinWatermark (3.5+) when retries change the time. No count()/show() on a stream.

Spark Connect (3.4+)

Thin client sends unresolved logical plans (protobuf) over gRPC; results return as Arrow. builder.remote("sc://host:15002"), SPARK_REMOTE. No SparkContext or RDDs (so no accumulators, sc.broadcast). Benefits: client isolation, independent upgrades, local IDE debugging.

Deploy modes

Pandas API on Spark

import pyspark.pandas as ps · sdf.pandas_api() · psdf.to_spark() · to_pandas() collects to the driver. pandas syntax, distributed, for data bigger than one machine. Default index distributed-sequence; row order not guaranteed.

Pandas UDFs

@F.pandas_udf("double") + type hints pd.Series -> pd.Series. Arrow batches (10,000 rows), vectorized: much faster than row-at-a-time UDFs. Series → scalar = aggregate.