In the paper, we research the small signal enhancement and receiving detection algorithm based on the communication data link system. In order to improve the reception performance of the signal transmitted in the communication data link system, a small signal detection algorithm based on the nonlinear system was proposed under the condition of low SNR. In nonlinear system, appropriate intensity of noise can enhance the amplitude of the output signal on a certain extent, which can realize the functions that enhancing and detecting small signal in the environment of strong background noise. In the paper, we analyze the MSK signal in the nonlinear bistable system based on data link system, using Matlab simulation to make out the waveform diagram of the output signal in time domain and frequency domain, then analyze the performance of the small signal enhancement and detection algorithm based on data link system. Matlab simulation results show that the algorithm proposed in the paper can effectively improve the reception performance of communication signals under the condition of low SNR, and has a great application prospect in the signal processing and receiving field.