Abstract
Transfer Learning (TL) encompasses a number of Machine Learning Techniques that take a pre-trained model aimed at solving a task in a Source Domain and try to reuse it to improve the performance of a related task in a Target Domain An important issue in TL is that the effectiveness of those techniques is strongly dataset-dependent. In this work, we investigate the possible structural causes of the varying performance of Heterogeneous Transfer Learning (HTL) across domains characterized by different, but overlapping feature sets (this naturally determine a partition of the features into Source Domain specific subset, Target Domain specific subset, and shared subset). To this purpose, we use the Partial Information Decomposition (PID) framework, which breaks down the multivariate information that input variables hold about an output variable into three kinds of components: Unique, Synergistic, and Redundant. We consider that each domain can hold the PID components in implicit form: this restricts the information directly accessible to each domain. Based on the relative PID structure of the above mentioned feature subsets, the framework is able to tell, in principle: 1) which kind of information components are lost in passing from one domain to the other, 2) which kind of information components are at least implicitly available to a domain, and 3) what kind information components could be recovered through the bridge of the shared features. We show an example of a bridging scenario based on synthetic data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
For example, if a full dataset defining the XOR function in a Cartesian plane is available to an information source, say \(\beta \), the attempt to learn the corresponding classifier using a straight line as class separation boundary is bound to fail.
- 2.
In that example we will bridge from \(\alpha \) to \(\beta \) information that is synergistic to \(\gamma \) for the prediction of the target/output variable, so that the Target Domain view can exploit the synergy between the available \(\gamma \) and the non-available \(\alpha \).
- 3.
It is apparent that in this scenario \(\alpha \) and \(\gamma \) can, together, predict the target variable z. However, neither the Source Domain nor the Target Domain encompass both feature sets. However, since \(\alpha \) can in principle be partially recovered from \(\beta \), there is room for improving the target domain prediction, with respect to those of a model learned solely on the basis of view \((\beta ,\gamma ,z)\).
References
Day, O., Khoshgoftaar, T.M.: A survey on heterogeneous transfer learning. J. Big Data 4, 1–42 (2017)
Duan, L., Xu, D., Tsang, I.: Learning with augmented features for heterogeneous domain adaptation. arXiv preprint arXiv:1206.4660 (2012)
Kolchinsky, A.: A novel approach to the partial information decomposition. Entropy 24(3) (2022). https://doi.org/10.3390/e24030403. https://www.mdpi.com/1099-4300/24/3/403
Mozafari, A.S., Jamzad, M.: A SVM-based model-transferring method for heterogeneous domain adaptation. Pattern Recogn. 56, 142–158 (2016). https://doi.org/10.1016/j.patcog.2016.03.009
Williams, P.L., Beer, R.D.: Nonnegative decomposition of multivariate information. arXiv Information Theory (2010)
Zhou, J.T., Tsang, I.W., Pan, S.J., Tan, M.: Multi-class heterogeneous domain adaptation. J. Mach. Learn. Res. 20(57), 1–31 (2019). http://jmlr.org/papers/v20/13-580.html
Zhuang, F., et al.: A comprehensive survey on transfer learning. Proc. IEEE 109(1), 43–76 (2020)
Acknowledgements
The work was partially supported by the project MUSA - Multilayered Urban Sustainability Action - project, funded by the European Union - NextGenerationEU, (CUP G43C22001370007, Code ECS00000037). The work was also partially supported by the project SERICS (PE00000014) under the NRRP MUR program funded by the EU - NextGenerationEU.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Gianini, G., Barsotti, A., Mio, C., Lin, J. (2024). Heterogeneous Transfer Learning from a Partial Information Decomposition Perspective. In: Chbeir, R., Benslimane, D., Zervakis, M., Manolopoulos, Y., Ngyuen, N.T., Tekli, J. (eds) Management of Digital EcoSystems. MEDES 2023. Communications in Computer and Information Science, vol 2022. Springer, Cham. https://doi.org/10.1007/978-3-031-51643-6_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-51643-6_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-51642-9
Online ISBN: 978-3-031-51643-6
eBook Packages: Computer ScienceComputer Science (R0)