Databricks Machine Learning Professional

Spark ML or single node?

For inference: batch and streaming scoring of huge tables suit a Spark ML PipelineModel (for a stream, apply the logged model as an MLflow UDF, mlflow.pyfunc.spark_udf). Real-time, low-latency requests suit a single-node model behind Model Serving, since a Spark model needs a Spark session to score.

Group-wise models with the pandas function API

import pandas as pd
from sklearn.linear_model import LinearRegression

def fit_store(pdf: pd.DataFrame) -> pd.DataFrame:
    m = LinearRegression().fit(
        pdf[["price", "promo"]], pdf["units"])
    return pd.DataFrame({
        "store_id": [pdf["store_id"].iloc[0]],
        "r2": [m.score(pdf[["price", "promo"]],
                       pdf["units"])]})

out = (df.groupBy("store_id")
         .applyInPandas(fit_store,
             schema="store_id long, r2 double"))

applyInPandas gets one pandas DataFrame per group (a whole group must fit in an executor's memory); mapInPandas gets an iterator of batches with no grouping, which suits inference. A pandas UDF (@pandas_udf) is column in, column out, its type hints pick the UDF variant, and the return type goes in @pandas_udf(...).

Distributed tuning: Optuna and Ray Tune

import mlflow
from mlflow.optuna.storage import MlflowStorage
from mlflow.pyspark.optuna.study import (
    MlflowSparkStudy)

storage = MlflowStorage(experiment_id=exp_id)
study = MlflowSparkStudy(
    study_name="gbt-tuning", storage=storage)
study.optimize(objective, n_trials=32, n_jobs=4)
print(study.best_params)

• MlflowStorage keeps the study in the MLflow experiment; MlflowSparkStudy runs the trials on Spark executors (MLflow 3, preinstalled in Runtime 17.0 ML+).
• Optuna's pruners (default MedianPruner) stop poor trials early; the default sampler is TPE.
• Ray Tune: Tuner(...).fit() on a Ray-on-Spark cluster; log to MLflow with Ray's MLflow integration.

Advanced MLflow

with mlflow.start_run(run_name="search"):
    for p in grid:
        with mlflow.start_run(nested=True):
            mlflow.log_params(p)
            mlflow.log_metric("cv_auc", cv_auc(p))
    mlflow.log_dict(summary, "summary.json")
    mlflow.log_figure(fig, "confusion.png")

• One parent run for the search, a child run per parameter set (logging the cross-validated metric), then the best set retrained and logged with its model.
• log_param(s), log_metric(s) (with step=), log_artifact(s), log_dict, log_figure, log_table, set_tag(s).

Custom models with real-time feature engineering: subclass mlflow.pyfunc.PythonModel and compute features from the request inside predict (see domain 3), so training and serving share one code path.

Feature pipelines, online serving, on-demand features

• Automated pipelines: a scheduled job or Lakeflow pipeline computes features and calls fe.write_table(name=…, df=…, mode="merge") (merge is the only mode). Pass a streaming DataFrame and write_table returns a StreamingQuery, keeping features fresh as events arrive.
• Online Feature Store: fe.create_online_store(name=…, capacity="CU_1"), then fe.publish_table(online_store=…, source_table_name=…, online_table_name=…). publish_mode is TRIGGERED (default, incremental) or CONTINUOUS (streaming), both needing Change Data Feed on the source, or SNAPSHOT (one full copy).
• The legacy Databricks SDK online_tables API is gone with Online Tables; the SDK's feature_store service still deletes online tables (delete_online_table).
• A model logged with fe.log_model and served looks up published features by key; Feature Serving endpoints serve a feature spec to non-model clients.
• On-demand features are Python UDFs in Unity Catalog (CREATE FUNCTION … LANGUAGE PYTHON) wrapped in FeatureFunction. They're computed from request data at inference time and by create_training_set at training time: one definition, no skew. Runtime 13.3 LTS ML+.