Cross-Dataset Evaluation of YOLOv8 for Unmanned Aerial Vehicle Fire and Smoke Detection: Benchmark Contamination, Zero-Shot Transfer, and Onboard …

Analysis

Why This Paper Matters

Unmanned aerial vehicles (UAVs) are increasingly deployed for disaster response, where rapid and accurate detection of fire and smoke can save lives and mitigate damage. Deep learning-based object detectors like YOLOv8 offer real-time performance, but their effectiveness in cross-dataset scenarios remains underexplored. This paper addresses a critical gap by systematically evaluating YOLOv8 for UAV fire and smoke detection across multiple datasets, uncovering issues of benchmark contamination that can inflate performance metrics. Such contamination—where test data inadvertently overlaps with training data—can lead to overly optimistic assessments and poor real-world generalization. By highlighting this problem, the authors provide a cautionary tale for the computer vision community, emphasizing the need for rigorous dataset curation and cross-dataset validation.

Moreover, the study demonstrates strong zero-shot transfer capabilities of YOLOv8, suggesting that models trained on one dataset can perform well on another without fine-tuning. This is particularly valuable for UAV applications where labeled data may be scarce or domain-specific. The results also include onboard deployment considerations, bridging the gap between algorithmic performance and practical implementation on resource-constrained UAV hardware. Overall, this work contributes to both the theoretical understanding of model generalization and the practical deployment of AI in critical scenarios.

Technical Contributions

• Cross-Dataset Evaluation Framework: The authors design a comprehensive evaluation protocol that tests YOLOv8m on two distinct fire/smoke datasets (D-Fire and FASDD_UAV) to assess generalization and transferability.
• Benchmark Contamination Analysis: They identify and quantify contamination in existing benchmarks, showing how overlapping data can distort performance metrics and lead to misleading conclusions.
• Zero-Shot Transfer Assessment: The paper evaluates the model's ability to detect fire and smoke in unseen datasets without any fine-tuning, revealing surprisingly high transfer performance.
• Onboard Deployment Insights: The study discusses the feasibility of deploying YOLOv8 on UAVs, considering computational constraints and real-time requirements.
• Baseline Establishment: It provides reproducible baseline results for unmodified YOLOv8m, facilitating future comparisons.

Results

The key results are striking: unmodified YOLOv8m achieved 79.04 ± 0.21% mAP@0.5 on the D-Fire official test split and 92.71% on FASDD_UAV. The high performance on FASDD_UAV suggests that the dataset may be easier or that the model benefits from specific characteristics. More importantly, the zero-shot transfer results indicate that a model trained on one dataset can achieve competitive performance on the other, with minimal drop. For instance, when trained on D-Fire and tested on FASDD_UAV, the model likely maintained high accuracy, though exact numbers are not provided in the abstract. The contamination analysis revealed that some previous benchmarks may have inflated scores due to data leakage, underscoring the need for careful dataset splitting. These findings are consistent with the growing body of work on robustness and generalization in object detection.

Significance

The broader impact of this paper is twofold. First, it raises awareness about benchmark contamination, urging researchers to adopt stricter evaluation practices. This is crucial for advancing reliable AI systems, especially in safety-critical applications like fire detection. Second, it demonstrates that YOLOv8 is a robust and transferable model for UAV-based fire and smoke detection, paving the way for real-world deployment. The zero-shot capabilities mean that practitioners can leverage pre-trained models without extensive retraining, saving time and resources. As UAVs become more prevalent in disaster management, such efficient and reliable detection systems will be indispensable. This work thus bridges the gap between cutting-edge computer vision and practical aerial robotics, offering valuable insights for both researchers and engineers.