Abstract

Large language models (LLMs) increasingly automate multiphysics simulation workflows by translating natural-language descriptions  into  executable  solver  configurations.  However, execution  success  alone  provides a misleading signal of validity: simulations  may complete without error while yielding  numerically invalid results, particularly in coupled multiphysics settings. This work investigates the validity boundaries of LLM-driven simulation automation by explicitly separating execution validity from numerical validity and localizing failures at distinct pipeline stages. 

We introduce a structured automation framework for COMSOL Multiphysics comprising: (1) a stage-explicit pipeline architecture (Geometry → Mesh → Material → Physics → Study → Results) enabling localized failure detection; (2) schema-grounded  retrieval-augmented  generation  (RAG)  providing  explicit  structural  guidance  during  physics configuration; and (3) a two-level validation protocol distinguishing execution validity (Lv1) from numerical validity (Lv2), with stricter validation requirements in coupled multiphysics settings, where all physics-specific numerical criteria must be satisfied simultaneously within a single simulation run. 

Through controlled experiments across six benchmarks spanning single-physics and coupled thermo-mechanical scenarios, we demonstrate that execution-focused metrics substantially overestimate workflow validity, with 30–60% of successfully executed simulations failing numerical validation. Multiphysics coupling emerges as the dominant source of silent numerical failures. Schema-grounded RAG substantially improves numerical validity primarily by shifting failures toward  earlier,  diagnosable  pipeline  stages  rather  than  increasing  raw  executability.  These  findings  demonstrate empirically that LLM-driven simulation automation requires explicit validation protocols, structural constraints during generation, and stage-level instrumentation to surface failure modes that execution signals alone cannot detect.