The Smelting activities are the main pathway for the anthropogenic release of heavy metals (HMs) into the soil-groundwater environment. It is vital to identify the factors affecting HMs pollution to better prevent and manage soil pollution. The present study conducted a comprehensive investigation of HMs in soil from a large abandoned Zn smelting site. An integrated approach was proposed to classify and quantify the factors affecting HMs pollution in the site. Besides, the quantitative relationship between hydrogeological characteristics, pollution transmission pathways, smelting activities and HMs pollution was established. Results showed that the soils were highly contaminated by HMs with a pollution index trend of As > Zn > Cd > Pb > Hg. In identifying the pollution hotspots, we conclude that the pollution hotspots of Pb, As, Cd, and Hg present a concentrated distribution pattern. Geo-detector method results showed that the dominant driving factors for HMs distribution and accumulation were the potential pollution source and soil permeability. Additionally, the main drivers are variable for different HMs, and the interaction among factors also enhanced soil HMs contamination. Our analysis illustrates how the confounding influences from complex environmental factors can be distilled to identify key factors in pollution formation to guide future remediation strategies.