Beneath the Adapter: Interpreting Internal Representation Change Induced by Parameter-Efficient Fine-Tuning in Vision Foundation Models

Abstract

Parameter-efficient fine-tuning (PEFT) has become the dominant strategy for adapting large pretrained vision transformers to downstream tasks, yet the field has largely evaluated these methods through the narrow lens of downstream accuracy rather than through what they do to the internal geometry of a model. This paper offers a conceptual and integrative synthesis of the emerging but fragmented literature on how techniques such as adapters, low-rank adaptation, visual prompt tuning, and bias-only tuning reshape the hidden representations of vision foundation models such as the Vision Transformer and CLIP. Drawing on representation-similarity tools including centered kernel alignment and canonical correlation analysis, the review develops a three-axis analytical framework, comprising locus, depth, and geometry of change, through which disparate empirical findings can be compared on common terms. The synthesis indicates that full fine-tuning produces pervasive, non-linear drift concentrated in later layers, whereas parameter-efficient methods tend to confine adaptation to narrow, near-identity subspaces whose location depends on where trainable parameters are inserted. This pattern helps explain why parameter-efficient methods often generalize better under distribution shift despite touching a minuscule fraction of total parameters. The paper contributes a unified vocabulary for representation-level interpretability research in efficient transfer learning, identifies methodological inconsistencies that limit cross-study comparison, and proposes a research agenda centered on standardized, layer-resolved representational auditing of adaptation methods prior to deployment