In order to improve the robustness of all-source navigation system, a multi-sensor fusion algorithm based on weighted factor graph model is proposed to solve the problem of sensor performance changes and measurement anomalies of UAV in complex environment. According to the gradient characteristics of fault information, an adaptive weight function is established to dynamically adjust the weights of different factors in real time, and a factor graph optimization model and a weighted nonlinear cost function are constructed to suppress the interference of outliers on state estimation. Finally, an all-source navigation system integrating multiple sensors is built, and the soft fault and mixed fault simulation tests are carried out for the proposed method. The results show that, compared with Huber robust kernel and noise modification methods, the proposed method reduces the position error by more than 34%, has better robustness and reliability, and does not increase the calculation cost.