The travel gearbox in excavators is crucial for mobility, and faults in this component can lead to costly downtime. Reliable fault detection is essential, especially under conditions with fluctuating speeds and noise. Traditional methods relying on cyclo-stationary assumptions are inadequate due to non-cyclo-stationary signals. This study introduces a novel relative entropy-based approach for fault detection, utilizing the smeared envelope spectrum for the first time. The approach transforms both the envelope spectrum and speed profile into probability distributions, which are then made comparable through synchronization, considering gear dynamics. Relative entropy quantifies the similarity between these distributions, providing a robust metric for fault detection. The multi-order probabilistic approach (MOPA) estimates the speed profile, and inherent errors due to speed variations are minimized through an optimization process also using relative entropy. While precise speed estimation is not required for fault detection, excessively large errors can reduce the effectiveness of detection. The proposed optimization process ensures that the speed estimation errors remain within an acceptable range, enabling relative entropy to provide high fault detectability under various driving condition. Experimental validation on industrial field data demonstrates that the proposed method outperforms conventional methods, achieving 98.5% accuracy in distinguishing between normal and faulty gearbox states.