Composing Independently Trained Adapters in Vision Foundation Models: A Synthesis of Weight Space Merging Strategies

Abstract

The rapid proliferation of parameter efficient fine tuning methods, and particularly Low Rank Adaptation, has produced a growing ecosystem of small, independently trained adapters that specialize large vision foundation models such as CLIP, DINOv2, and the Segment Anything Model for narrow downstream tasks. A practical question now confronts practitioners and researchers alike: can these separately trained adapters be combined after the fact, without joint retraining or access to the original training data, to yield a single model that retains the competencies of each contributor. This paper synthesizes the emerging body of scholarship on adapter merging and composition in vision foundation models, situating recent techniques within the broader lineage of weight space model merging research that began with simple parameter averaging and matured into interference aware operations such as task arithmetic and sign election. Employing a systematic, theory driven literature synthesis rather than de novo experimentation, the study organizes existing approaches into four analytical categories, namely linear and weighted averaging, task vector arithmetic, conflict resolution methods, and learned or gated composition. The review identifies persistent tensions between merging simplicity and multitask fidelity, uncovers a marked asymmetry between the maturity of merging research in language models relative to vision systems, and highlights the special vulnerability of dense prediction and segmentation adapters to destructive interference. The findings suggest that vision specific structural priors, including spatial locality and cross modal alignment, warrant merging strategies that differ meaningfully from those developed for text only transformers, and the paper closes by outlining a research agenda for principled, evaluable, and reproducible adapter composition in vision systems.