Near-Earth Objects (NEOs) regularly approach Earth, and a small subset may be labeled as potentially hazardous due to their size, velocity, and close-approach characteristics. Manually reviewing every observed object can be costly, so a data-driven screening model can help experts prioritize objects that deserve earlier inspection. This study uses the NASA Nearest Earth Objects dataset to build an explainable binary classification pipeline for predicting whether a NEO is labeled as hazardous. The task remains binary classification, but the model also outputs the estimated probability of hazardous=True, which is treated as a risk score for prioritization. We compare Majority Baseline, Logistic Regression, class-weighted Logistic Regression, Random Forest, HistGradientBoosting, XGBoost, and LightGBM models, and we apply lightweight hyperparameter tuning to the main tree-based models. The best validation model is random_forest_balanced. After sigmoid calibration and validation-based threshold tuning, the final calibrated model achieves F1 = 0.5332, recall = 0.7474, precision = 0.4145, PR-AUC = 0.5595, and ROC-AUC = 0.9269 on the test set using threshold = 0.19. We further conduct a five-variation ablation study which shows that threshold tuning, class balancing, and log features each contribute to recall, while calibration trades recall for probability quality necessary for risk scoring. Explainability analysis using permutation importance and SHAP shows that miss_distance, absolute_magnitude, and diameter-related features are the most influential factors. The final system provides both global and local explanations, supporting expert review rather than replacing domain judgment.
Keywords: Near-Earth Objects, binary classification, class imbalance, risk scoring, probability calibration, SHAP, explainable AI
Near-Earth Objects are important in planetary defense and astronomical monitoring. Most NEOs do not pose immediate danger, but some objects require closer review when their size, velocity, or close-approach conditions suggest higher risk. In practice, experts often need to screen many observed objects before performing deeper orbital analysis. A data mining model that ranks objects by risk and explains its predictions can reduce review burden and support early prioritization.
This study focuses on predicting whether a NEO is labeled as hazardous from observed attributes. The problem is naturally a supervised binary classification task. However, a class label alone is not sufficient for risk triage, because experts also need to know which objects should be checked first. Therefore, this project uses a two-layer prediction design: first, the model outputs the estimated probability of hazardous=True; second, a decision threshold converts this probability into the final class label.
The hazardous probability in this study should not be interpreted as the true physical probability of Earth impact. It is the model-estimated probability that an object is labeled as hazardous under this dataset's label definition and available feature set. For that reason, this study includes probability calibration and model explainability to make the output more transparent and less likely to be overinterpreted.
The contributions of this work are:
- A reproducible binary classification pipeline for NEO hazardous-label prediction.
- A comparison of multiple models and class-weight strategies under class imbalance.
- Validation-based threshold tuning for a risk-screening decision setting.
- Calibration and explainability analysis using calibration curves, Brier score, permutation importance, and SHAP.
- Global feature importance and local case explanations for report and presentation use.
- A five-variation ablation study quantifying the contribution of each major design decision (calibration, threshold tuning, class balancing, log features) to recall and probability quality.
This study uses the NASA Nearest Earth Objects dataset from Kaggle. The dataset author notes that the data comes from NASA Open API and JPL CNEOS close-approach data. The license is CC0 Public Domain. The local experiment uses data/neo.csv.
The dataset contains 90,836 rows and 10 columns:
id, name, est_diameter_min, est_diameter_max, relative_velocity, miss_distance, orbiting_body, sentry_object, absolute_magnitude, hazardous
The target column is hazardous. The remaining columns are used either as model features or as identifiers for case studies.
No missing values were observed in the local dataset snapshot. The column orbiting_body is always Earth, and sentry_object is always False; therefore, both are constant columns and are not useful as model features. The columns id and name are identifiers, so they are excluded from model input but kept for local case interpretation.
The target distribution is:
| Class | Count | Ratio |
|---|---|---|
hazardous = False |
81,996 | 90.27% |
hazardous = True |
8,840 | 9.73% |
This distribution shows clear class imbalance. A model that always predicts False can still achieve about 90.27% accuracy, but it has zero recall for hazardous objects. Therefore, this study does not use accuracy as the main model selection metric. Instead, it focuses on recall, precision, F1, PR-AUC, and ROC-AUC.
The main numeric feature summary is:
| Feature | Min | Q1 | Median | Q3 | Max | Mean |
|---|---|---|---|---|---|---|
est_diameter_min |
0.000609 | 0.019256 | 0.048368 | 0.143402 | 37.892650 | 0.127432 |
est_diameter_max |
0.001362 | 0.043057 | 0.108153 | 0.320656 | 84.730541 | 0.284947 |
relative_velocity |
203.35 | 28619.02 | 44190.12 | 62923.60 | 236990.13 | 48066.92 |
miss_distance |
6745.53 | 17210820.24 | 37846579.26 | 56548996.45 | 74798651.45 | 37066546.03 |
absolute_magnitude |
9.23 | 21.34 | 23.70 | 25.70 | 33.20 | 23.53 |
For the size and distance features, the mean is far larger than the median (for example, est_diameter_min has mean 0.1274 versus median 0.0484, with a maximum of 37.89), which indicates a heavy right-skew with a long tail of large objects. The distribution plots below make this skew visible and directly motivate the log1p transforms applied to size, velocity, and distance in Section 3.2.
Pearson correlation shows that absolute_magnitude has a correlation of -0.3653 with hazardous, indicating that lower absolute magnitude is associated with a higher probability of being labeled hazardous. relative_velocity has a positive correlation of 0.1912 with the target. miss_distance has only weak linear correlation with the target, at 0.0423. However, later model importance results suggest that miss_distance can still be important in a nonlinear model. The heatmap also shows that est_diameter_min and est_diameter_max are perfectly correlated (r = 1.0), because both are derived from absolute_magnitude under a fixed albedo assumption. This redundancy is the direct motivation for collapsing them into the engineered est_diameter_mean and est_diameter_range features described in Section 3.2.
The prediction task is binary classification:
hazardous ∈ {True, False}
The model output is:
P(hazardous = True | observed features)
The probability is then converted to a class label using a decision threshold:
if P(hazardous=True) >= threshold:
predict hazardous
else:
predict non-hazardous
This design supports both classification and risk-based ranking.
The data is split into train, validation, and test sets using stratified splitting, with an approximate ratio of 70%, 15%, and 15%. Stratification is important because the positive class accounts for only 9.73% of the dataset. Without stratification, the validation or test distribution may become unstable.
The final split sizes are:
| Split | Rows |
|---|---|
| Train | 63,585 |
| Validation | 13,625 |
| Test | 13,626 |
The validation set is used for model selection and threshold tuning. The test set is used only for final evaluation.
The columns id, name, orbiting_body, and sentry_object are removed from model input. The first two are identifiers, while the latter two are constant in the current data snapshot.
The base numeric features are:
est_diameter_minest_diameter_maxrelative_velocitymiss_distanceabsolute_magnitude
The following engineered features are added:
| Feature | Definition | Purpose |
|---|---|---|
est_diameter_mean |
(est_diameter_min + est_diameter_max) / 2 |
Reduce diameter redundancy with a single size estimate |
est_diameter_range |
est_diameter_max - est_diameter_min |
Represent diameter estimate range |
log_est_diameter_mean |
log1p(est_diameter_mean) |
Reduce skewness in size |
log_relative_velocity |
log1p(relative_velocity) |
Reduce skewness in velocity |
log_miss_distance |
log1p(miss_distance) |
Reduce skewness in distance |
The final model input contains 10 features.
This study compares:
- Majority Baseline: always predicts
hazardous=False. - Logistic Regression: a linear model with StandardScaler.
- Balanced Logistic Regression: Logistic Regression with
class_weight="balanced". - Balanced Random Forest: Random Forest with
class_weight="balanced_subsample". - HistGradientBoostingClassifier: a gradient boosting model for nonlinear tabular patterns.
- XGBoost: a histogram-based boosted tree model using
scale_pos_weightfor imbalance. - LightGBM: an efficient leaf-wise boosting model using
scale_pos_weight.
The study also applies lightweight hyperparameter tuning to Random Forest, HistGradientBoosting, XGBoost, and LightGBM. Tuning uses RandomizedSearchCV, stratified 2-fold cross-validation, four candidate settings per model, and average_precision (PR-AUC) as the scoring metric. This includes tuning evidence while keeping the experiment reproducible and computationally manageable.
Each model first outputs the probability of hazardous=True. A threshold is then selected on the validation set. The search range is 0.05 to 0.95 with a step size of 0.01. The primary selection criterion is F1, with recall and precision used as secondary considerations.
The best model is also calibrated using sigmoid calibration. Calibration is not intended to maximize all classification metrics. Its purpose is to make the probability output more suitable as a risk score. The probability quality is evaluated using calibration curve and Brier score.
This study uses two explanation methods:
- Permutation Importance: measures how much PR-AUC decreases when a feature is randomly shuffled.
- SHAP: provides both global feature contribution patterns and local explanations for individual predictions.
Permutation importance summarizes global model behavior. SHAP provides per-sample attribution and helps explain why a specific object receives a high or low hazardous probability.
The validation results at threshold = 0.5 are:
| Model | Accuracy | Precision | Recall | F1 | PR-AUC | ROC-AUC |
|---|---|---|---|---|---|---|
| Majority Baseline | 0.9027 | 0.0000 | 0.0000 | 0.0000 | 0.0973 | 0.5000 |
| Logistic Regression | 0.9078 | 0.6606 | 0.1086 | 0.1865 | 0.4170 | 0.8883 |
| Balanced Logistic Regression | 0.7869 | 0.3073 | 0.9487 | 0.4642 | 0.3951 | 0.8908 |
| HistGradientBoosting | 0.9163 | 0.7218 | 0.2270 | 0.3454 | 0.5591 | 0.9255 |
| HistGradientBoosting Tuned | 0.9155 | 0.7384 | 0.2044 | 0.3201 | 0.5591 | 0.9253 |
| XGBoost | 0.7918 | 0.3169 | 0.9864 | 0.4797 | 0.5411 | 0.9212 |
| XGBoost Tuned | 0.8080 | 0.3312 | 0.9540 | 0.4916 | 0.5536 | 0.9248 |
| LightGBM | 0.8037 | 0.3288 | 0.9766 | 0.4920 | 0.5647 | 0.9281 |
| LightGBM Tuned | 0.8027 | 0.3275 | 0.9751 | 0.4903 | 0.5687 | 0.9292 |
| Random Forest Tuned | 0.8094 | 0.3351 | 0.9744 | 0.4987 | 0.5548 | 0.9259 |
| Balanced Random Forest | 0.8979 | 0.4808 | 0.6154 | 0.5399 | 0.5880 | 0.9347 |
The Majority Baseline has high accuracy because the dataset is imbalanced, but its recall and F1 are both zero. This confirms that accuracy is misleading for this task. XGBoost and LightGBM obtain very high recall, but their precision is low, producing many false positives. Tuning improves LightGBM's PR-AUC to 0.5687, but it still does not exceed the Balanced Random Forest PR-AUC of 0.5880. Balanced Random Forest performs best in F1, PR-AUC, and ROC-AUC, so it is selected for calibration and final testing.
The lightweight tuning cross-validation PR-AUC results are:
| Tuned Model | Best CV PR-AUC |
|---|---|
| LightGBM Tuned | 0.5284 |
| XGBoost Tuned | 0.5276 |
| Random Forest Tuned | 0.5233 |
| HistGradientBoosting Tuned | 0.5208 |
These results show that XGBoost and LightGBM are competitive during tuning, but their validation-set performance does not surpass the untuned balanced Random Forest. Based on validation PR-AUC, F1, and ROC-AUC together, this study keeps random_forest_balanced as the final model.
Because the same feature pipeline and train/test split are shared across all models, we can also compare their computational cost fairly. The table below reports, for each base model, the median fit time over the 63,585-row training set, the batch scoring time over the full 13,626-row test set, and the single-object scoring latency (CPU only; n_jobs=-1 where supported). Times are measured with neo-benchmark.
| Model | Fit time (s) | Batch predict, 13,626 rows (ms) | Throughput (rows/s) | Single-row latency (ms) |
|---|---|---|---|---|
| Majority Baseline | 0.001 | 0.01 | — | 0.007 |
| LightGBM | 0.39 | 5.7 | 2,400,000 | 0.77 |
| HistGradientBoosting | 0.39 | 11.9 | 1,140,000 | 1.96 |
| XGBoost | 0.57 | 3.5 | 3,850,000 | 1.18 |
| Balanced Logistic Regression | 0.72 | 0.85 | 16,100,000 | 0.44 |
| Logistic Regression | 0.77 | 0.80 | 17,100,000 | 0.46 |
| Balanced Random Forest | 1.29 | 59.3 | 230,000 | 38.4 |
The chosen model, Balanced Random Forest, is in fact the most expensive of all candidates: it takes the longest to train (1.29 s) and has the slowest scoring (59 ms for the entire test set, 38 ms for a single object — the latter reflecting thread-dispatch overhead of the 120-tree ensemble on a one-row input). Logistic Regression is roughly 70× faster per object, and the boosted-tree models fall in between. Unlike a real-time serving scenario — for example sub-millisecond ad click prediction, where Logistic Regression's latency advantage would be decisive — NEO hazard screening is an offline, batch-oriented triage step. Scoring the full test set in 59 ms (about 230,000 objects per second) is far beyond what expert review throughput requires, so prediction latency is not a binding constraint. This is why the project trades the small extra cost of Random Forest for its higher recall, PR-AUC, and probability quality: efficiency is adequate for all candidates at this data scale, and the decision is therefore driven by screening quality rather than speed.
The final test results compare three settings:
| Setting | Threshold | Accuracy | Precision | Recall | F1 | PR-AUC | Brier | ROC-AUC |
|---|---|---|---|---|---|---|---|---|
| Raw Random Forest | 0.34 | 0.8544 | 0.3838 | 0.8198 | 0.5228 | 0.5634 | 0.0673 | 0.9267 |
| Calibrated Random Forest | 0.19 | 0.8727 | 0.4145 | 0.7474 | 0.5332 | 0.5595 | 0.0600 | 0.9269 |
| Calibrated Random Forest, default threshold | 0.50 | 0.9119 | 0.5946 | 0.2986 | 0.3976 | 0.5595 | 0.0600 | 0.9269 |
The calibrated model with threshold = 0.19 achieves the best F1 score, 0.5332, with recall = 0.7474. The default threshold of 0.5 has higher accuracy and precision, but its recall drops to 0.2986. This supports the main design choice of the project: threshold tuning is necessary when the goal is risk screening under class imbalance.
For the calibrated Random Forest at threshold = 0.19, the test confusion matrix is:
| Predicted False | Predicted True | |
|---|---|---|
| True False | 10,900 | 1,400 |
| True True | 335 | 991 |
This setting recovers 991 hazardous objects and misses 335 hazardous objects. Compared with threshold = 0.5, the number of true positives increases from 396 to 991, but false positives also increase from 270 to 1,400. This is a typical trade-off in screening systems: reducing missed hazardous objects requires accepting more candidates for expert review.
To verify that each design choice in the final model contributes to the screening objective, this study performs an ablation analysis that systematically removes one component at a time and re-evaluates on the test set. Because the task prioritizes minimizing missed hazardous objects, the analysis treats recall, F1, and PR-AUC as the primary axes; accuracy is reported for completeness but does not drive the comparison.
Five configurations are compared:
| Version | Setting | Change from full model |
|---|---|---|
| V0 | Full Model | Balanced Random Forest + sigmoid calibration + validation-tuned threshold (0.19) |
| V1 | w/o Calibration | Raw Balanced Random Forest with threshold tuned on raw scores (0.34) |
| V2 | w/o Threshold Tuning | Calibrated model evaluated at the default threshold of 0.5 |
| V3 | w/o Class Balancing | Random Forest without class_weight="balanced_subsample", otherwise identical to V0 (threshold 0.24) |
| V4 | w/o Log Features | Model trained on the seven non-log features only, otherwise identical to V0 (threshold 0.29) |
The corresponding test-set metrics are:
| Version | Threshold | Accuracy | Precision | Recall | F1 | PR-AUC | ROC-AUC |
|---|---|---|---|---|---|---|---|
| V0 Full Model | 0.19 | 0.8727 | 0.4145 | 0.7474 | 0.5332 | 0.5595 | 0.9269 |
| V1 w/o Calibration | 0.34 | 0.8544 | 0.3838 | 0.8198 | 0.5228 | 0.5634 | 0.9267 |
| V2 w/o Threshold Tuning | 0.50 | 0.9119 | 0.5946 | 0.2986 | 0.3976 | 0.5595 | 0.9269 |
| V3 w/o Class Balancing | 0.24 | 0.8929 | 0.4609 | 0.5958 | 0.5197 | 0.5715 | 0.9277 |
| V4 w/o Log Features | 0.29 | 0.8954 | 0.4718 | 0.6244 | 0.5375 | 0.5880 | 0.9324 |
Removing calibration (V1) gives the highest recall, 0.8198, but also produces the largest false-positive count among all configurations. Calibration trades a small amount of recall for improved probability quality, which lowers the Brier score from 0.0673 to 0.0600 and makes the output more suitable as a continuous risk score. Removing threshold tuning (V2) has the most dramatic effect: recall collapses to 0.2986 and 930 hazardous objects are missed, while accuracy rises to 0.9119. This confirms that the default threshold of 0.5 is inappropriate for an imbalanced screening task and that accuracy alone would mask a substantial degradation in screening utility.
Removing class balancing (V3) lowers recall from 0.7474 to 0.5958, with 201 additional missed hazardous objects. Precision and accuracy both rise, but this is exactly the type of trade-off the screening objective does not favor. Removing log features (V4) presents a more nuanced result: PR-AUC, ROC-AUC, and F1 are all slightly higher than V0, but recall drops from 0.7474 to 0.6244, missing 163 additional hazardous objects. The log transformations reduce skewness in size and distance distributions, which appears to stabilize the decision boundary at low probability values where borderline hazardous objects accumulate. For ranking-style metrics the log features do not contribute, but for the recall objective that this study targets they remain beneficial.
Among the four ablated components, three (V2, V3, V4) produce lower test-set recall than the full model. V1 is the only ablation that achieves higher recall than V0 (0.8198 versus 0.7474), but it does so at the cost of probability quality: the Brier score deteriorates from 0.0600 to 0.0673, undermining the system's role as a risk score. Ranking by absolute impact on recall, the components are ordered as threshold tuning (Δ = 0.449), class balancing (Δ = 0.152), log features (Δ = 0.123), and calibration (Δ = 0.072). Threshold tuning therefore exerts the dominant influence, while calibration's contribution is best understood through probability quality rather than recall. This combined evidence supports the final design: a calibrated Balanced Random Forest with a validation-tuned threshold provides the best recall-versus-probability-quality balance for the risk-screening objective.
The top five permutation importance results are:
| Rank | Feature | Importance Mean |
|---|---|---|
| 1 | miss_distance |
0.0915 |
| 2 | log_miss_distance |
0.0893 |
| 3 | absolute_magnitude |
0.0451 |
| 4 | log_est_diameter_mean |
0.0443 |
| 5 | est_diameter_min |
0.0424 |
The model relies strongly on close-approach distance, magnitude, and diameter-related features. Although miss_distance has weak linear correlation with the target, it becomes highly important in the nonlinear Random Forest model. This suggests that simple linear correlation is not sufficient to describe all useful predictive signals in this dataset.
The SHAP global plots show a similar pattern: distance, magnitude, and size-related features dominate the model's hazardous probability output.
This study selects three test cases: one true positive, one false negative, and one false positive.
id: 3774091name: (2017 HP3)- True label: hazardous
- Predicted probability: 0.8957
- Threshold: 0.19
- Predicted label: hazardous
SHAP shows that log_miss_distance, miss_distance, est_diameter_min, absolute_magnitude, and log_est_diameter_mean all push the hazardous probability upward. This case shows that the model can correctly identify a high-risk object based on distance, size, and magnitude signals.
id: 3713941name: (2015 EO61)- True label: hazardous
- Predicted probability: 0.1897
- Threshold: 0.19
- Predicted label: non-hazardous
This case is very close to the decision threshold — the predicted probability of 0.1897 is only 0.0003 below the threshold of 0.19. SHAP shows that four size-related features (est_diameter_min, log_est_diameter_mean, est_diameter_range, est_diameter_mean) push the probability upward, while log_miss_distance pushes it downward; the model arbitrates this conflict conservatively. This type of boundary case is important in practice because a small threshold change can alter the final decision, and the V2 ablation result demonstrates the same sensitivity at the system level.
id: 3566975name: (2011 KO17)- True label: non-hazardous
- Predicted probability: 0.8954
- Threshold: 0.19
- Predicted label: hazardous
Although the true label is non-hazardous, the model assigns a high risk score. SHAP indicates that distance, magnitude, and size-related features strongly push the prediction upward. From a screening perspective, this false positive increases expert review workload, but it is less severe than missing a truly hazardous object.
The results show that threshold selection has a major impact on model behavior. The default threshold of 0.5 improves precision and accuracy but sharply reduces recall. In a planetary defense screening scenario, false negatives are usually more costly than false positives, so a lower threshold is reasonable. However, the threshold cannot be lowered arbitrarily because too many false positives would overload expert review. This study uses the validation-F1 threshold of 0.19 as a compromise between recall and precision.
The calibrated Random Forest obtains a Brier score of 0.0600, better than the raw model's 0.0673. This suggests that calibration improves probability quality. However, the probability still represents the model's estimate of the dataset label, not the physical probability of impact. A physically stronger hazard model would require more complete orbital features and domain-specific modeling.
The importance of absolute_magnitude and diameter-related features is consistent with domain intuition, because magnitude and object size are related. Larger or brighter objects are more likely to be labeled potentially hazardous. relative_velocity is positively correlated with the target, but permutation importance ranks it below distance and size features. miss_distance has high model importance despite weak linear correlation, suggesting nonlinear effects.
This study has several limitations:
- The
hazardouslabel is a dataset label, not the true probability of Earth impact. - The dataset has limited features and does not include full orbital parameters such as MOID, orbital inclination, semi-major axis, or eccentricity.
- XGBoost, LightGBM, and hyperparameter tuning are included, but the tuning uses a lightweight search; broader search spaces and more CV folds may improve performance further.
- SHAP explains model behavior but does not prove causality.
- The selected threshold is based on validation F1. If the real review cost function changes, the optimal threshold may also change.
This study builds a reproducible and explainable binary classification pipeline for NEO hazardous-label prediction. The results show that accuracy is not suitable as the main metric under strong class imbalance. PR-AUC, F1, recall, and confusion matrix analysis provide a better view of whether the model is useful for risk screening. Balanced Random Forest performs best on the validation set. After sigmoid calibration and threshold tuning, the final model achieves F1 = 0.5332, recall = 0.7474, and precision = 0.4145 on the test set. The five-variation ablation study confirms that threshold tuning is the single largest design lever (ΔRecall = 0.449 when removed), followed by class balancing and log features, while calibration trades a small amount of recall for substantially improved probability quality. Explainability analysis shows that close-approach distance, magnitude, and diameter-related features are the main drivers of the model's predictions.
Overall, the best positioning of this project is an explainable probability-based binary classification system. It estimates the probability that a NEO is labeled as hazardous and uses that risk score to support expert prioritization. Future work should add richer orbital data and a formal review-cost function to make the model closer to real planetary defense decision support.
- Sameep Vani. NASA Nearest Earth Objects. Kaggle Dataset. https://www.kaggle.com/datasets/sameepvani/nasa-nearest-earth-objects
- NASA Open APIs. https://api.nasa.gov/
- NASA Jet Propulsion Laboratory, Center for Near Earth Object Studies. Close-Approach Data. https://cneos.jpl.nasa.gov/ca/
- Lundberg, S. M., & Lee, S. I. (2017). A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems, 30, 4765-4774.
- Pedregosa, F., et al. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12, 2825-2830.
- Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5-32.
- Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785-794.
- Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T. Y. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. Advances in Neural Information Processing Systems, 30, 3146-3154.
- Platt, J. (1999). Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. Advances in Large Margin Classifiers, 10(3), 61-74.