Constructing Dynamic Master Logic Models as Knowledge Graphs for Complex System Diagnostics Using Retrieval-Augmented Large Language Models

Analysis

Why This Paper Matters

Dynamic Master Logic (DML) models are crucial for understanding and diagnosing complex engineered systems, but their construction has traditionally been a manual, expert-intensive process. This paper addresses a critical bottleneck by automating the creation of DML models from technical documentation using large language models (LLMs) and retrieval-augmented generation (RAG). The ability to automatically generate executable functional models from text could transform how reliability and safety analyses are conducted, especially for large-scale systems where manual modeling is impractical.

The paper's focus on scaling to substantially larger and more complex systems than prior work is particularly significant. It moves beyond proof-of-concept and tackles the challenges of real-world application, such as handling extensive documentation and maintaining logical consistency across a large hierarchy. By representing the DML as a knowledge graph, the framework also enables downstream reasoning tasks like failure propagation and dependency tracing, which are essential for diagnostics and safety assessment.

Technical Contributions

The paper introduces several key innovations:

• Automated KG-DML Construction: A framework that uses RAG to extract and structure information from system descriptions into a knowledge graph representation of the DML hierarchy.
• Hierarchical Construction with Targeted Retrieval: The model is built layer-by-layer, using targeted retrieval to preserve functional dependencies and explicit logical relationships.
• Multi-Level Validation Methodology: A novel validation approach that assesses layer-specific precision and recall, logical gate consistency, and overall structural integrity, providing a comprehensive evaluation of the constructed model.
• Scalability to Complex Systems: The framework is designed to handle substantially larger and more complex systems than previous efforts, addressing a key limitation in automated DML generation.

Results

The paper applies the framework to the Low-Pressure Coolant Injection (LPCI) system of a decommissioned Boiling Water Reactor. The key result is that the automated construction process yields consistent reconstruction across repeated runs, indicating reliability and reproducibility. However, the abstract does not provide quantitative metrics such as precision, recall, or F1 scores, nor does it compare the automated model against a gold-standard expert-built DML. The validation methodology is described, but specific numerical outcomes are not reported in the abstract. This lack of quantitative detail makes it difficult to fully assess the accuracy and robustness of the approach.

Significance

This work has significant implications for the fields of reliability engineering, system safety, and AI-assisted engineering. By automating the construction of DML models, it could drastically reduce the time and expertise required to create these models, making them more accessible for a wider range of systems. The use of knowledge graphs as the output format also facilitates integration with other AI tools and reasoning engines, potentially enabling more sophisticated diagnostic and predictive capabilities.

From an AI perspective, this paper demonstrates a compelling application of LLMs and RAG for structured knowledge extraction from unstructured technical text. It highlights the potential of LLMs to bridge the gap between natural language documentation and formal engineering models. The multi-level validation approach could serve as a template for evaluating similar AI-generated structured models in other domains. Overall, this research paves the way for more intelligent and automated approaches to system modeling and analysis, with potential benefits for safety-critical industries.