Databricks Machine Learning Professional

The exam

59 scored questions · 120 min · pass mark not published · Python · September 2025 guide.

Weights

New names

Lakehouse Monitoring → data profiling · DLT → Lakeflow pipelines · Workflows → Lakeflow Jobs · Asset Bundles → Declarative Automation Bundles · AI Gateway → Unity Gateway · Feature Store → Feature Store in UC (FeatureEngineeringClient) · legacy Online Tables (no longer supported) → Online Feature Store (Lakebase) · Hyperopt (not in Runtime ML after 16.4 LTS) → Optuna or Ray Tune · AutoML not built in from Runtime 18.0 ML.

Spark ML or single node

Spark ML pieces

Estimator fit() → transformer transform(); Pipeline is an estimator, PipelineModel a transformer. StringIndexer → OneHotEncoder (linear models) → VectorAssembler. Tune with ParamGridBuilder + CrossValidator (k fits per point, refits the best) or TrainValidationSplit (one split, cheaper); parallelism= fits concurrently. Evaluators: BinaryClassification (areaUnderROC), Multiclass (f1), Regression (rmse, r2).

Scaling

Ray on Spark: setup_ray_cluster() (ray.util.spark), Runtime 12.2 LTS ML+. TorchDistributor(num_processes, local_mode, use_gpu).run(fn); local_mode=True = driver only.

pandas function API

groupBy(k).applyInPandas(fn, schema): one pandas DataFrame per group (must fit in memory) → per-group models. mapInPandas: batches, no grouping → inference. @pandas_udf(returnType): column → column; hints pick the variant.

Tuning and MLflow

Optuna distributed: MlflowStorage(experiment_id=…) + MlflowSparkStudy(study_name, storage).optimize(objective, n_trials, n_jobs) (MLflow 3). Defaults: TPE sampler, MedianPruner stops poor trials. Nested runs: parent = search, child per parameter set via start_run(nested=True). Log with log_params, log_metric(step=), log_artifact, log_dict, log_figure.

Feature Store

• Time series table: timeseries_columns="ts", ts in primary_keys; FeatureLookup with timestamp_lookup_key → AS OF join, latest value at or before the label time; lookback_window ignored online.
• write_table(mode="merge") only; a streaming DataFrame returns a StreamingQuery.
• Online: create_online_store(name=…, capacity="CU_1") → publish_table(…); publish_mode TRIGGERED (default) / CONTINUOUS (both need CDF) / SNAPSHOT.
• On-demand: UC Python UDF + FeatureFunction(udf_name=, input_bindings=, output_name=); computed at request time and in training.

Lifecycle map

Deploy code: same code dev → staging → prod, retrain on prod data, catalog per environment, service principal runs prod. Git + CI · bundles define resources · MLflow tracks · UC versions models · aliases mark champion/challenger · Lakeflow Jobs orchestrate · Model Serving · inference tables + data profiling monitor.

Testing

pyspark.testing.assertDataFrameEqual; functions in .py modules, tests in tests/ run by pytest. New hyperparameters → re-run training, evaluation, deployment tests.

Declarative Automation Bundles

resources: jobs, experiments, registered_models, model_serving_endpoints, quality_monitors; targets: dev (mode: development) / prod (mode: production). bundle validate → bundle deploy -t prod → bundle run.

Retraining

Triggers: schedule, table-update or file-arrival trigger, alert on drift or quality. Pick the winner on the same recent hold-out, using the decision's metric (log loss when calibrated probabilities drive thresholds); move @champion only on a margin; keep the old version to roll back.

Data profiling

Outputs <table>_profile_metrics, <table>_drift_metrics, a dashboard. slicing_exprs → slice_key/slice_value. drift_type: CONSECUTIVE (vs previous window) or BASELINE (vs baseline table). Alerts: Databricks SQL alert on a metrics-table query.

Endpoint health

Serving UI: latency, request rate, error rate, CPU, memory (14 days); export to Prometheus/Datadog. Inference tables (via AI Gateway, now Unity Gateway) log requests and responses to UC.

Rollouts

One endpoint, two served entities, traffic_config.routes percentages; update the config to shift.

Custom PyFunc

Subclass mlflow.pyfunc.PythonModel: load_context(context) reads context.artifacts[...]; predict(context, model_input, params=None) holds pre/post-processing. pyfunc.log_model(name=, python_model=, artifacts={…}, input_example=, registered_model_name="cat.sch.m"); UC needs a signature. Test: load_model(uri), then .predict(df).

Deploy and query

get_deploy_client("databricks"), then .create_endpoint(name, config) with served_entities (entity_name, entity_version, workload_size) and .predict(endpoint=, inputs=). REST: POST /serving-endpoints/<name>/invocations, Authorization: Bearer <token>, body dataframe_split / dataframe_records / instances / inputs. SQL: ai_query.