Model Architecture & Performance
· Model: CatBoost Tabular + Text
· Outcome: Dramatic and/or Unusual UAP Sighting Classification
· Data: NUFORC Nov. 2021 – Apr. 2026 (20,024 reports)
· 60/20/20 split (Train/Validation/Test)
· Test size: 4,005 reports; Validation size: 4,005 reports
How it works
The label. NUFORC marks a minority of reports in two
tiers, indicated by the punctuation on the databank's Open link.
Both tiers are collapsed into a single positive class. The model therefore
predicts an editorial decision, not a property of the sighting.
The model. A single gradient-boosted decision tree
ensemble (CatBoost), trained on the witness narrative and the structured
fields together. Text is handled by CatBoost's native text pipeline rather
than by a neural encoder: narratives are tokenized on whitespace and
turned into bag-of-words and BM25 features over a dictionary capped at
50,000 terms. Those sit alongside shape, country, state, hour of sighting,
and a set of derived features, and the trees split across all of them.
There is no embedding model and no transformer anywhere in the pipeline.
Calibration and threshold. Raw boosted-tree scores are not
probabilities, so the fitted model is wrapped in a sigmoid (Platt)
calibration fitted on the validation split. The decision threshold of 0.17
was tuned on validation rather than left at 0.5, because the outcome is
imbalanced and 0.5 would predict almost nothing positive.
Where it runs. Deployment is a single always-on container
shared with the performance dashboard. The trained model ships as a native
CatBoost file plus a two-number calibration JSON, loaded once at startup.
Scoring one report is a pass over 663 shallow trees and completes in
milliseconds on a CPU. No GPU is used at any stage,
training or inference, and training runs on an ordinary laptop CPU.
The explanation. The word-level highlighting is LIME,
computed per request. It perturbs your text a few hundred times, re-scores
each variant, and fits a local linear model to attribute the score to
individual words. That is the slow part of a request, and the permutation
count is adjustable under LIME Settings.
Performance Metrics
| Metric | Mean | 95% CI |
|---|
| ROC-AUC | 0.836 | (0.816, 0.855) |
| Precision | 0.408 | (0.367, 0.450) |
| Recall | 0.495 | (0.450, 0.540) |
| Specificity | 0.909 | (0.899, 0.918) |
| Average Precision (AP) | 0.432 | (0.385, 0.479) |
| F1 (Weighted) | 0.447 | (0.408, 0.485) |
| Brier Score | 0.082 | (0.075, 0.089) |
Note. Held-out test set (n = 4,005). Confidence intervals from
5,000 bootstrap resamples, percentile method. Precision, recall,
specificity, and F1 are computed at the model's tuned decision threshold
of 0.17; ROC-AUC, average precision, and Brier score are
threshold-independent.
Hyperparameters
| Parameter | Value |
|---|
| n_estimators | 663 |
| depth | 8 |
| learning_rate | 0.05 |
| l2_leaf_reg | 10 |
| min_data_in_leaf | 10 |
| subsample | 0.7 |
| leaf_estimation_method | Newton |
| early_stopping_rounds | 150 |
| text tokenizer | Space (Word tokens) |
| dictionary max size | 50,000 |
| text feature calcers | BoW, BM25 |