Computer Science > Software Engineering
[Submitted on 15 Jul 2023 (v1), last revised 2 Feb 2024 (this version, v2)]
Title:AdvFusion: Multilingual Adapter-based Knowledge Transfer for Code Summarization
View PDF HTML (experimental)Abstract:Parameter Efficient Fine-Tuning (PEFT) is an alternate choice to full fine-tuning a language model. Though PEFT methods are used in natural language domain widely, there are limited studies on using PEFT for language models that are pre-trained on code and comment datasets (i.e., code-LMs). Previous research has also shown that code summarization, a task that intends to generate natural description of the given code snippet automatically and is known to benefit the program comprehension, benefits from multilingual fine-tuning approach. In multilingual fine-tuning, the code-LM is fine-tuned on a dataset consisting of different programming languages.
AdapterFusion is a specific PEFT approach that aims to extract and compose the latent knowledge from multiple (language) adapters for a downstream task. However, our experiments reveal that the AdapterFusion still learns from the same language, not taking advantage of other programming languages. Therefore, we change the architecture and propose AdvFusion, a PEFT approach that enforces the model to first learn from other programming languages, and then pay attention to the language of the target task. Therefore, the AdvFusion emphasizes the knowledge transfer among different programming languages, as stated in the multilingual fine-tuning.
Our results on the CodeSearchNet dataset using two code-LMs, show that Adapters, AdapterFusion, and our proposed AdvFusion can achieve results on-par with or higher than the full fine-tuning models for code summarization and method name prediction. Notably, the number of trainable parameters are 123x less and the training time is reduced by ~30%. AdvFusion exhibits a notable enhancement compared to AdapterFusion, showcasing a 0.9 to 1.7-point increase in BLEU-4 scores specifically for Ruby, JavaScript, and Go.
Submission history
From: Fuxiang Chen [view email][v1] Sat, 15 Jul 2023 17:17:16 UTC (1,679 KB)
[v2] Fri, 2 Feb 2024 10:47:32 UTC (1,468 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.