v0.6.0

MichiBoost v0.6.0

Diff since v0.5.0

New features

Regression losses

• Six parametrized losses, passed as instances to loss_function:
  • Huber(δ) — squared error within δ, linear past it; robust to outliers
  • Quantile(α) — fits the α-quantile for prediction intervals or asymmetric over/under penalties
  • Expectile(α) — same asymmetry as Quantile but with squared error for a smooth fit
  • MAPE() — penalizes relative error; targets must be non-zero
  • Tweedie(p) — non-negative, right-skewed targets with a spike at zero; predicts on a log scale
  • LogLinQuantile(α) — quantiles of positive targets that span orders of magnitude
• Tweedie and LogLinQuantile fit on a log scale; predict exponentiates back to the target scale

Continuing training

• fit!(model, X, y; init_model=base) resumes from a fitted model, inherits its trees, and adds iterations more
• Reuses the inherited model's feature borders and categorical encoder; border_count is ignored when init_model is set
• Task and feature counts must match, or fit! errors before training starts
• Works with early_stopping_rounds: the metric runs over the full ensemble, and best_iteration counts from the inherited model's first tree

Training snapshots

• snapshot_path writes the partial model to disk via JLD2 every snapshot_interval iterations and once after the loop exits
• Each write overwrites the previous file; the final snapshot matches the model fit! returns, including early-stopping truncation
• Resume a crashed run by loading the snapshot with load_model and passing it as init_model

Other changes

• Tag-valued kwargs (loss_function, boosting_type, eval_metric, auto_class_weights) now accept a Symbol (:RMSE) alongside tag types (Losses.RMSE) and strings ("RMSE")
• Documentation cleanup across the guide and API reference

Internal changes

• Reworked the train loop and histogram code
• Multiclass histograms defer per-class totals to a post-fill summation

No breaking changes.

Closed issues:

• Implement training continuation strategy (#14)
• Implement snapshot mechanism (#15)

v0.5.0

MichiBoost v0.5.0

Diff since v0.4.0

New features

Model evaluation and selection

• score(model, X, y) — coefficient of determination for regressors, accuracy for classifiers
• Stratified k-fold cross-validation via `cv(pool; stratified=true, ...)
• eval_metric kwarg drives early stopping with a configurable metric (AUC, F1, Accuracy, R^2, Logloss, MultiLogloss, MAE, RMSE)
• get_best_iteration(model) / get_best_score(model) expose the early-stopping winner
• `eval_metrics(model, pool; metrics =[...]) evaluates one or more metrics at every boosting iteration
• staged_predict(model, X) / staged_predict_proba(model, X) return per-iteration predictions for convergence diagnostic

Class weighting

• class_weights=Dict(label => weight, ...) on MichiBoostClassifier
• auto_class_weights="Balanced" or "SqrtBalanced" computes weights from label frequencies automatically

Ensemble post-processing

• shrink!(model, n) truncates the tree ensemble in place (useful for rolling back to get_best_iteration).

Model introspection and config

• isfitted(model) predicate
• get_params(model) returns a copy of the hyperparameter dict; set_params!(model; kwargs...) updates them

Custom loss functions

• Subtype MichiBoost.LossFunction and implement gradient_hessian!, initial_prediction, loss
• Optional traits task-type (:regression, :binary, :multiclass) and link_inverse (maps raw scores to probabilities for predict_proba)
• Wrapper validates the task matches the model kind; custom losses round-trip through save_model / load_model

Compile-time-checked enums

• New sobmodules Metrics.*, Losses.*, BoostingTypes.*, AutoClassWeights.*, and PredictionTypes.* provide singleton tag types (e.g. loss_function=Losses.RMSE, eval_metric=Metrics.AUC). CatBoost-style strings ("RMSE", "AUC") are still accepted

Breaking changes

• Model serialization switched to JLD2 — models saved with v0.4.0's Serialization backend (.jls files) will not load. Re-save with the new backend; .jld2 is now the conventional extension

Internal changes

• Codebase reformatted with JuliaFormatter / BlueStyle

Closed issues:

• Stratified cross-validation in cv() (#6)
• Add score(model, X, y) method (#7)
• Adding custom loss mechanism (#10)

v0.4.0

MichiBoost v0.4.0

Diff since v0.3.0

New features

• Documentation site: Documenter.jl built docs deployed
• Benchmark suite implemented: benchmark/README.md documents the four-axis methodology (correctness, speed sweep, scaling on UCI Covertype, per-feature costs) and reports head-to-head numbers against CatBoost.jl

Performance improvements

• Multiclass histogram axis flip: HistCacheMC gradient/hessian arrays reshaped to (n_classes, max_leaves, n_bins), so the for c in 1:n_classes inner loop sweeps stride-1 memory. Per-class accumulators in SplitBuffersMC flipped to match
• B-invariant precomputation in _sweep_gain_mc: total_g^2/(total_h + λ) depends only on (class, leaf) and is now computed once into buf.total_score per call instead of recomputed for every bin index. Savings scale with n_bins * n_leaves * n_classes
• Multiclass prefetch path removed: _fill_num_leaf_mc! and _fill_cat_leaf_mc! now do single-pass direct reads from gradients/hessians into the histograms
• :dynamic scheduling on multiclass feature loops: _find_best_split_across_leaves_mc now uses :dynamic, smoothing thread imbalance from variable per-feature bin counts
• SplitBuffersMC footprint reduced: dropped fields related to prefetch

Breaking changes

• HistCacheMC array shapes changed: num_hist_g, num_hist_h, cat_hist_g, cat_hist_h are now Array{Float64,3} of shape (n_classes, max_leaves, n_bins)
• SplitBuffersMC field set changed:
  • local_gradients_mc, local_hessians_mc, local_bins, local_cat_values removed
  • total_score::Matrix{Float64} of shape (n_classes, max_leaves) added
  • per-class fields (total_g, total_h, left_g, left_h, parent_hist_g_scratch, parent_hist_h_scratch) now have n_classes as their first axis

Closed issues:

• Setup Documentation (#3)

v0.3.0

MichiBoost v0.3.0

Diff since v0.2.0

New features

• Benchmark suite split by topic: the single benchmark_vs_catboost.jl is replaced by four per-area scripts: correctness.jl (held-out metric gates), speed_sweep.jl (n*p size grid + categorical cardinality + inference batch sweep), scaling.jl (Covertype real dataset + --sweep threading curve), and feature_costs.jl (CV, early stopping, SHAP, RSM, sample weights, save/load). A new diagnostic profile.jl is provided, and a dedicated benchmark/README.md with methodology and results.
  • Correctness benchmark uses task metrics instead of agreement: regression is gated on out-of-sample RMSE/MAE/R^2, classification on log-loss/AUC/accuracy, each asserted within a fixed tolerance of CatBoost, replacing the previous correlation/class-agreement gates which could pass while model quality was poor.

Performance improvements

• Histogram subtraction trick: when building children, only the smaller child's histogram is built from scratch; the larger is derived as parent − smaller. Roughly halves histogram-building work.
• Parallelised tree construction: _apply_split! (sample partitioning across child leaves) and the final leaf-value reduction loop now run one leaf per thread, writing to disjoint slices of the output.
• Incremental early-stopping evaluation: _evaluate_loss no longer walks the full tree ensemble on every ES check — a running eval_preds buffer is extended with just the newly-built tree each iteration. Reduces ES cost from O(T^2) to O(T) over a training run.
• Row-chunk parallel inference: _predict_raw rewritten to parallelise over row ranges instead of trees, eliminating the per-thread n * n_classes partials buffer and the serial final reduction. Each thread walks all trees against its own row slice, writing directly into a disjoint view of preds.
• Parallel apply_borders: quantisation now parallelises over features with a function-barrier helper, with a small-input fallback so tiny inference batches don't pay fork-join overhead.
• Function-barrier helpers eliminate Core.Box in threaded loops: g_sum / h_sum accumulators inside Threads.@threads bodies were being boxed because the compiler couldn't prove they didn't escape the closure. Moving the per-leaf work into small helpers dropped ~1.1M boxed Float64s per iteration to zero.
• In-place gradient + hessian kernels: new gradient_hessian!(g, h, lf, y, pred, scratch) fused kernels write both outputs in one pass and reuse caller-provided scratch for softmax/sigmoid, eliminating the multiple per-iteration n * n_classes matrix allocations the old allocating negative_gradient / hessian path produced.
• Capped categorical bin count via quantile cuts: ordered target statistics can produce up to n_samples distinct encoded values at low raw cardinality, making per-iteration histogram work O(n_samples). cat_sorted_vals is now quantile-truncated to at most border_count + 1 entries, so per-feature bin work is bounded regardless of raw cardinality (e.g., k=5 cardinality training dropped from CB 6.26x to CB 1.33x).
• Zero-copy Pool construction from matrix: columns are now views into the input matrix instead of fresh per-column copies, saving ~1.6 MB per call on a 10k * 20 matrix and meaningfully improving inference latency.
• Index-based loops replace for idx in group: rewritten as for k in 1:n; idx = group[k] everywhere a SubArray was being iterated, eliminating ~480k iterator-state tuple allocations per iteration.
• Concrete LeafGroupView element type: _apply_split! now returns Vector{LeafGroupView} instead of Vector{Any} / Vector{SubArray}, so downstream group[k] / length(group) / isempty(group) stay statically typed and don't dispatch dynamically.
• SplitCandidate is now isbits: feature_type::Symbol replaced with is_categorical::Bool, so Vector{SplitCandidate} stores elements inline rather than as boxed references.

Bug fixes

• MAE loss leaf values: the Newton step sum(g)/(sum(h)+λ) is only valid for smooth losses; for MAE it bounded each leaf's contribution to ±1, causing severe underfitting. Fixed with weighted-median-of-residuals leaf refinement (same approach as XGBoost / LightGBM / CatBoost). MAE held-out score now matches or beats CatBoost on benchmark data.

Breaking changes

• SplitCandidate.feature_type::Symbol removed, replaced with is_categorical::Bool. Any caller constructing SplitCandidate(j, :numerical, …) should pass SplitCandidate(j, false, …) (or true for categorical). Tree-level split_feature_types::Vector{Symbol} stored inside SymmetricTree / SymmetricTreeMultiClass is unchanged, so model serialisation is preserved.
• build_symmetric_tree signature additions: new keyword arguments leaf_refine_values, leaf_refine_weights (used by the MAE path; default nothing) and explicit per-thread buffers / hist_cache objects. Defaults keep existing callers working.
• Categorical split candidates are coarser: at very low raw cardinality, split thresholds are now drawn from a quantile-subsampled set (<= border_count + 1 candidates) instead of every unique encoded value. Can change model output on datasets where the extra candidates carried real signal; correctness gates still pass within tolerance.

v0.2.0

MichiBoost v0.2.0

Diff since v0.1.0

New features

• Precomputed gradient/hessian scaling by sample weights: gradients and hessians are multiplied by weights in the main loop (ones by default), replacing the previous unweighted path.
• Public shap_values(wrapper, data) API: shap_values now works directly on MichiBoostRegressor/ MichiBoostClassifier wrappers with optional cat_features.
• Benchmark suite reorganized: benchmarks moved out of test/ into a top-level benchmark/ directory with its own Project.toml; added a "real dataset" quick test (test_real_quick.jl), plus CV and early-stopping benchmark scenarios, better CLI handling, and categorical feature names now passed to CatBoost for a fair comparison.

Performance improvements

• Threaded tree-level parallelism in predict: inference now splits trees across threads with per-thread partial-prediction buffers and pre-allocated leaf-index buffers, for both binary/regression and multiclass.
• Threaded _predict_raw: internal raw-score path parallelized to match predict, so SHAP, CV loss eval, and early-stopping checks all benefit from the same tree-level parallelism.
• In-place predict_tree!: predict_tree! now accepts a reusable leaf_indices::Vector{Int} buffer; _evaluate_loss (used during early stopping) now reuses pre-allocated tree_preds and leaf_indices instead of allocating per tree.
• SplitBuffers/SplitBuffersMC struct: single pre-allocated, thread-local buffer bundle (histograms, totals, left/right accumulators, index scratch, cache-friendly local gradient/hessian/bin arrays) reused across iterations and trees.
• Precomputed sorted categorical values: cat_sorted_vals is computed once per training run and reused across all iterations, avoiding per-level Set/sort work.
• Cache-friendly histogram path for large leaves: _fill_num_hist!/_fill_cat_hist! gather gradients, hessians, and bins into local scratch arrays when n >= 512, then do the scatter add in a second pass.
• Adaptive bin assignment in apply_borders: linear scan with early exit for <=32 borders, searchsortedfirst beyond that.

Bug fixes

• SHAP formula corrected: path-conditioned means and the correct /2 factor; each row now sums exactly to raw_prediction − E[raw_prediction] (uniform over leaves) instead of "approximately." SHAP inference is also now parallelized per-sample.
• SHAP summation property test added: new test asserts the exact summation identity.

Breaking changes

• src/trees/ split into base/ and multiclass/: build.jl, histograms.jl, predict.jl, and shap.jl now each have a base and multiclass sibling; shap.jl kept at the top level as the public dispatcher.
• Multiple dispatch replaces _mc/_multiclass suffixes: build_symmetric_tree_multiclass -> build_symmetric_tree, predict_tree_mc! -> predict_tree!, predict_tree_multiclass -> predict_tree, initial_prediction_multiclass -> initial_prediction. The multiclass variants are selected by argument types (Matrix gradients, SymmetricTreeMultiClass trees) rather than by name.
• train signature additions: build_symmetric_tree now takes buffers and cat_sorted_vals keyword arguments (defaults provided, so existing callers keep working).

v0.1.0

MichiBoost v0.1.0

Pure Julia gradient boosting with symmetric trees. Supports regression (RMSE, MAE), binary and multiclass classification, ordered target encoding for categorical features, cross-validation, early stopping, RSM feature subsampling, and model serialization.

Note

Training speed on medium/large datasets is currently 2–3× slower than CatBoost's C++ backend. Performance improvements are planned for v0.2.0, targeting histogram buffer pre-allocation and reduced GC pressure