Abstract: The Qingshankou Formation, Quantou Formation, and Denglouku Formation of Songnan Gas Field in Songliao Basin have unstable mudstone borehole walls. Considering this problem, the digital rock samples were established by high-precision CT scanning-based digital imaging technology, and the physical properties of the samples were extracted by numerical analysis of the digital rock microstructure image. A comprehensive comparative analysis was conducted in terms of the mudstone mineral composition, and physical and chemical properties of formations, as well as the microstructure damage variations in the rock samples caused by the physical and chemical actions when they were soaked in distilled water, solutions of different drilling fluid additives, and on-site drilling fluids. In this way, the instability forms and mechanisms of mudstone borehole walls were revealed. The results demonstrated that for the mudstone samples from the three formations, the content of clay minerals reached 34.56% on average, and they were mainly composed of mixed illite/smectite and illite, which all had strong hydration of spontaneous capillary imbibition. Due to different fabric characteristics, there was a great difference in the micro-fission expansion, fission process, and fracture mode and degree of hydration damage. The fission expansion process determined the trend and strength of secondary fractures, which further decided the macroscopic instability forms of mudstone borehole walls were mainly sheet stripping and falling block collapse. The mechanisms of various test fluids inhibiting the hydration of mudstone differed, and their effects also demonstrated significant differences. Hence, effectively controlling or inhibiting the intrusion of water molecules was the key to preventing the hydration of mudstone. The analysis suggests that the digital imaging analysis technology has great advantages in evaluating borehole wall instability regarding the intuitive characterization, quantitative description, and comparative analysis of the microstructure damage to the rock samples. The revealed instability forms and mechanisms of mudstone borehole walls provide a scientific basis for the optimal selection of drilling fluid additives and the formula of the drilling fluid system for Songnan Gas Field.