OES-Resilience
NASA SMAP and MSL labeled telemetry anomaly dataset · criterion not met
The stated v0.5 success criterion was not met. OES32 was significantly better than EWMA on SMAP only and was not significantly different from the other baselines; it did not significantly lose to any baseline. Because it did not significantly beat all four baselines on at least one dataset, the criterion fails. maxabs had a numerically higher pooled event F1 on both datasets and CUSUM on MSL, but those differences were not significant.
Run record
Reported metrics
Values are copied from the declared run summary. This report does not calculate scores, rank methods, or combine metrics across scopes.
SMAP (all channels; target FP 0.01)
| Method | Metric | Value | Unit | 95% interval | Note |
|---|---|---|---|---|---|
| oes32 | event precision | 0.262857 | fraction | — | — |
| oes32 | event recall | 0.676471 | fraction | — | — |
| oes32 | event F1 | 0.378601 | fraction | [0.273556, 0.522167] | — |
| oes32 | false alarms per 1,000 test samples | 0.29599 | alarm runs / 1,000 samples | — | — |
| maxabs | event precision | 0.537037 | fraction | — | — |
| maxabs | event recall | 0.426471 | fraction | — | — |
| maxabs | event F1 | 0.47541 | fraction | [0.328767, 0.645841] | — |
| maxabs | false alarms per 1,000 test samples | 0.057362 | alarm runs / 1,000 samples | — | — |
| zscore | event precision | 0.031878 | fraction | — | — |
| zscore | event recall | 0.676471 | fraction | — | — |
| zscore | event F1 | 0.060887 | fraction | [0.032237, 0.155983] | — |
| zscore | false alarms per 1,000 test samples | 3.205408 | alarm runs / 1,000 samples | — | — |
| ewma | event precision | 0.046047 | fraction | — | — |
| ewma | event recall | 0.779412 | fraction | — | — |
| ewma | event F1 | 0.086957 | fraction | [0.060642, 0.131869] | — |
| ewma | false alarms per 1,000 test samples | 2.519354 | alarm runs / 1,000 samples | — | — |
| cusum | event precision | 0.093604 | fraction | — | — |
| cusum | event recall | 0.882353 | fraction | — | — |
| cusum | event F1 | 0.169252 | fraction | [0.122105, 0.256413] | — |
| cusum | false alarms per 1,000 test samples | 1.333101 | alarm runs / 1,000 samples | — | — |
MSL (all channels; target FP 0.01)
| Method | Metric | Value | Unit | 95% interval | Note |
|---|---|---|---|---|---|
| oes32 | event precision | 0.316456 | fraction | — | — |
| oes32 | event recall | 0.694444 | fraction | — | — |
| oes32 | event F1 | 0.434783 | fraction | [0.333291, 0.575758] | — |
| oes32 | false alarms per 1,000 test samples | 0.732412 | alarm runs / 1,000 samples | — | — |
| maxabs | event precision | 0.5 | fraction | — | — |
| maxabs | event recall | 0.5 | fraction | — | — |
| maxabs | event F1 | 0.5 | fraction | [0.350868, 0.657143] | — |
| maxabs | false alarms per 1,000 test samples | 0.244137 | alarm runs / 1,000 samples | — | — |
| zscore | event precision | 0.151515 | fraction | — | — |
| zscore | event recall | 0.694444 | fraction | — | — |
| zscore | event F1 | 0.248756 | fraction | [0.177911, 0.389381] | — |
| zscore | false alarms per 1,000 test samples | 1.898846 | alarm runs / 1,000 samples | — | — |
| ewma | event precision | 0.13198 | fraction | — | — |
| ewma | event recall | 0.722222 | fraction | — | — |
| ewma | event F1 | 0.223176 | fraction | [0.126982, 0.423358] | — |
| ewma | false alarms per 1,000 test samples | 2.319304 | alarm runs / 1,000 samples | — | — |
| cusum | event precision | 0.391304 | fraction | — | — |
| cusum | event recall | 0.75 | fraction | — | — |
| cusum | event F1 | 0.514286 | fraction | [0.387097, 0.681319] | — |
| cusum | false alarms per 1,000 test samples | 0.569654 | alarm runs / 1,000 samples | — | — |
Limitations
- This is one evaluation on public labeled SMAP/MSL spacecraft telemetry, not evidence about operational performance or any live system.
- SMAP and MSL are distinct dataset scopes and are not pooled in this report.
- The released data were pre-scaled using test-set min/max values; the protocol notes this inherited leakage affects every method.
- Thresholds were calibrated on training samples only; the test labels were not used for tuning, according to the published protocol.
- The raw telemetry is not redistributed here. Metric values are copied from the published aggregate output files; the individual baseline comparison table is bundled separately.
Artifact integrity
All referenced files match their manifest SHA-256 values. This confirms bundle consistency only; it does not establish authorship, timing, provenance authenticity, or measurement validity.
artifacts/oes-resilience-v0.5/smap_msl_protocol.json— protocol; SHA-256927cf78fe7acc1cdc2a044a800b1e7aef2d954cd7e3b60a5b6b5b31b552b8ea3artifacts/oes-resilience-v0.5/smap_msl_results/smap_msl_results.json— results; SHA-2565263abb0359e1f19ba9da8a6f32e3c5c6e9c788809f929c1c066930facf30f72artifacts/oes-resilience-v0.5/smap_msl_results/smap_msl_summary.csv— summary; SHA-256ff5de8c39a1bc94791d56e512311fe6e4f49dc372e9e7195236069c531fca2baartifacts/oes-resilience-v0.5/smap_msl_results/smap_msl_comparisons.csv— comparisons; SHA-25601c3d5422fe486f4bb771fa9c7eeba0cf0f578b4dad8119c3d02458741a881ccartifacts/oes-resilience-v0.5/smap_msl_results/smap_msl_run_metadata.json— run_metadata; SHA-256ef7941a5e583fc1d9b35f1a0756eca61c3091b2be60de640b57d954b74a81f78artifacts/oes-resilience-v0.5/smap_msl_results/SHA256SUMS— upstream_checksum_list; SHA-256ec66c3bf2e774132f36c90fe0783577da8bc5315b2af6ea420645ea3030d37d7
SHA-256 matching proves only byte-for-byte consistency between a referenced file and the digest recorded in this bundle. It does not prove authorship, when a file or protocol existed, provenance authenticity, or measurement or method validity.