How XGBoost Works

XGBoost keeps the same general idea as gradient boosting: add trees sequentially, where each new tree improves the current ensemble, but it makes the boosting process more optimized, more regularized, and more efficient.

At a high level, the improvements are:

• better optimization through gradients and Hessians
• regularized tree complexity and leaf weights
• split selection using gain instead of RSS
• efficient candidate split selection through weighted quantile sketch
• native missing-value handling
• system-level speedups through parallel split finding and DMatrix

Flow
    -> predict every observation
    -> compute gradient and Hessian for every row
    -> construct candidate split points
    -> evaluate split gain using summed gradients and Hessians
    -> choose the best split
    -> build the tree
    -> add the tree to the ensemble
    -> repeat

Objective Function

Traditional GBM mainly tries to reduce training error, while XGBoost adds regularization directly into the objective function.

Conceptually, the objective is:

objective = training loss + tree complexity penalty

For a new tree, the regularized objective includes terms like:

Omega(f) = gamma * T + (1/2) * lambda * sum_j(w_j^2) + alpha * sum_j |w_j|

where:

• T is the number of leaves
• w_j is the prediction score for leaf j
• gamma penalizes adding more leaves
• lambda is L2 regularization on leaf weights
• alpha is L1 regularization on leaf weights

The intuition is practical: L2 shrinks leaf predictions toward zero, making corrections smaller and smoother, while L1 can push very small leaf predictions exactly to zero. gamma makes the model justify every new split.

So compared with classical GBM, XGBoost does not only ask, “Does this split reduce error?” It asks, “Does this split reduce error enough to justify the added complexity?”

Gradients and Hessians

For each observation, XGBoost calculates two values from the current prediction:

• g_i: the gradient, or the slope of the loss
• h_i: the Hessian, or the curvature of the loss

The gradient tells the model which direction the prediction should move, while the Hessian tells it how quickly the slope is changing.

Each observation has a scalar gradient and scalar Hessian because the derivative is taken with respect to that observation’s single prediction, y_hat_i, not with respect to every feature column.

This is the key distinction from ordinary GBM:

The second-order approximation gives XGBoost more information about the loss surface, which leads to better split and leaf-weight decisions.

Leaf Weights

XGBoost can calculate the optimal leaf weight in closed form. For one leaf, let:

G = sum of gradients in the leaf
H = sum of Hessians in the leaf

Without L1 regularization, the leaf weight is:

w* = -G / (H + lambda)

This is the correction that the leaf adds to predictions for observations that land in that region, and lambda shrinks the correction to reduce overly aggressive updates.

With L1 regularization, small leaf weights can be pushed to zero through thresholding, which is one reason XGBoost can create simpler trees.

Split Gain

XGBoost chooses splits by calculating how much the regularized objective improves.

For a candidate split:

gain = 1/2 * [
    G_L^2 / (H_L + lambda)
  + G_R^2 / (H_R + lambda)
  - G_P^2 / (H_P + lambda)
] - gamma

where:

• L is the left child
• R is the right child
• P is the parent node
• G is the sum of gradients
• H is the sum of Hessians
• gamma is the minimum split gain penalty

The split is useful only if the gain is large enough after subtracting the complexity penalty. This is different from ordinary CART, where split quality is usually based on RSS reduction.

Candidate Split Points

Classical GBM may consider every unique feature value as a possible split, which can be expensive for high-cardinality features.

XGBoost reduces the search space by constructing representative candidate split points using weighted quantile sketch.

The weighted quantile sketch does not choose the final split; it only creates a smaller set of candidate thresholds, and XGBoost still evaluates those candidates using the gain formula.

This is especially helpful when a feature has many unique values, because instead of checking one million possible thresholds, XGBoost can evaluate a much smaller set of candidates while keeping nearly the same split quality.

Missing Values

XGBoost handles missing values natively by learning a default direction for each split. During training, it tries sending missing values left and then right, and whichever direction reduces the objective more becomes the learned default direction.

During prediction, if a value is missing, it follows that learned direction.

candidate split
    -> missing values left  -> compute objective
    -> missing values right -> compute objective
    -> choose better direction

This avoids manual imputation in many cases and is usually better than using artificial placeholder values like -999, because the model learns the missing-value behavior from the data rather than treating missingness as a fake numeric value.

Parallel Training and DMatrix

Boosting itself is sequential across trees because tree m depends on the predictions from trees 1 through m - 1. Within a tree, though, XGBoost can parallelize split finding because different features can be evaluated at the same time across CPU cores.

Across trees: sequential
Within a tree: highly parallelizable

XGBoost also uses DMatrix, a data structure optimized for tree building that stores features in a compressed, column-oriented format, tracks missing values, and supports efficient column-wise access during split finding. This matters most for large or sparse datasets.

Important Parameters

Important note: min_child_weight is about the sum of Hessians in a child node. For MSE, the Hessian is 1, so min_child_weight = 50 behaves like requiring about 50 observations, but for other losses, the Hessian is not always 1.

Feature Importance

XGBoost commonly reports feature importance in three ways:

The same interpretation warning still applies: correlated features compete, so a feature can look unimportant because another correlated feature captured the same signal first.

High feature importance also does not imply causality or business importance. It only means the model used that feature in a particular way. For interpretation, SHAP values are often more useful, although correlated features still require caution.

Practical Tuning Order

A reasonable tuning flow is:

1. Tune tree complexity: max_depth, min_child_weight.
2. Tune randomness: subsample, colsample_bytree.
3. Tune regularization: lambda, alpha, gamma.
4. Tune learning behavior: learning_rate, n_estimators.

Use early stopping and cross validation to avoid overfitting, and use random search or Bayesian optimization when manually trying every combination becomes inefficient.

In practice, smaller learning rates with more trees often generalize better, as long as early stopping is used.