Abstract
This paper firstly proposes a simple yet efficient generalized approach to apply differential privacy to text representation (i.e., word embedding). Based on it, we propose a user-level approach to learn personalized differentially private word embedding model on user generated contents (UGC). To our best knowledge, this is the first work of learning user-level differentially private word embedding model from text for sharing. The proposed approaches protect the privacy of the individual from re-identification, especially provide better trade-off of privacy and data utility on UGC data for sharing. The experimental results show that the trained embedding models are applicable for the classic text analysis tasks (e.g., regression). Moreover, the proposed approaches of learning differentially private embedding models are both framework- and data-independent, which facilitates the deployment and sharing. The source code will be available when the paper is published.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abadi, M., et al.: Deep learning with differential privacy. ArXiv e-prints (2016)
Bansal, M., Gimpel, K., Livescu, K.: Tailoring continuous word representations for dependency parsing. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 809–815. Association for Computational Linguistics (2014). https://doi.org/10.3115/v1/P14-2131, http://aclweb.org/anthology/P14-2131
Batmanghelich, K.N., Saeedi, A., Narasimhan, K., Gershman, S.: Nonparametric spherical topic modeling with word embeddings. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) abs/1604.00126, pp. 537–542 (2016). http://arxiv.org/abs/1604.00126
Bayardo, R.J., Agrawal, R.: Data privacy through optimal k-anonymization. In: ICDE, pp. 217–228 (2005)
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)
Cynthia, D.: Differential privacy, pp. 1–12. ICALP (2006)
Dwork, C., Smithy, A.: Differential privacy for statistics: what we know and what we want to learn (2009)
Ebadi, H., Sands, D., Schneider, G.: Differential privacy: now it’s getting personal. In: Proceedings of the 42Nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2015, pp. 69–81. ACM, New York (2015). https://doi.org/10.1145/2676726.2677005, http://doi.acm.org/10.1145/2676726.2677005
Flekova, L., Gurevych, I.: Can we hide in the web? Large scale simultaneous age and gender author profiling in social media notebook for PAN at CLEF 2013. In: Working Notes for CLEF 2013 Conference, Valencia, Spain, 23–26 September 2013 (2013)
Fredrikson, M., Jha, S., Ristenpart, T.: Model inversion attacks that exploit confidence information and basic countermeasures. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, CCS 2015, pp. 1322–1333 (2015)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016). http://www.deeplearningbook.org
Kosinski, M., Matz, S., Gosling, S., Popov, V., Stillwell, D.: Facebook as a social science research tool. Am. Psychol. (2015)
Lee, J., Clifton, C.: How much is enough? Choosing \(\epsilon \) for differential privacy. In: Lai, X., Zhou, J., Li, H. (eds.) ISC 2011. LNCS, vol. 7001, pp. 325–340. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24861-0_22
Lee, J., Clifton, C.: Differential identifiability. In: Proceedings of KDD (2012)
Levy, O., Goldberg, Y.: Linguistic regularities in sparse and explicit word representations. In: Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pp. 171–180. Association for Computational Linguistics (2014). https://doi.org/10.3115/v1/W14-1618, http://aclweb.org/anthology/W14-1618
McMahan, H.B., Ramage, D., Talwar, K., Zhang, L.: Learning differentially private language models without losing accuracy. CoRR abs/1710.06963 (2017). http://arxiv.org/abs/1710.06963
McSherry, F.D.: Privacy integrated queries: an extensible platform for privacy-preserving data analysis. In: SIGMOD (2009)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. CoRR abs/1301.3781 (2013). http://arxiv.org/abs/1301.3781
Mohsen, A.M., El-Makky, N.M., Ghanem, N.: Author identification using deep learning. In: 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 898–903 (2016). https://doi.org/10.1109/ICMLA.2016.0161
Nguyen, H.H., Imine, A., Rusinowitch, M.: Detecting communities under differential privacy. In: Proceedings of the 2016 ACM on Workshop on Privacy in the Electronic Society, WPES 2016, pp. 83–93. ACM, New York (2016). https://doi.org/10.1145/2994620.2994624, http://doi.acm.org/10.1145/2994620.2994624
Papernot, N., Song, S., Mironov, I., Raghunathan, A., Talwar, K., Erlingsson, Ú.: Scalable private learning with PATE. In: Sixth International Conference on Learning Representation (ICLR 2018) (2018)
Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014), http://www.aclweb.org/anthology/D14-1162
Phan, N., Wu, X., Hu, H., Dou, D.: Adaptive laplace mechanism: differential privacy preservation in deep learning. CoRR abs/1709.05750 (2017). http://arxiv.org/abs/1709.05750
Popov, V., Kudinov, M., Piontkovskaya, I., Vytovtov, P., Nevidomsky, A.: Distributed fine-tuning of language models on private data. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=HkgNdt26Z
Taddy, M.: Document classification by inversion of distributed language representations. In: Proceedings of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 45–49. Association for Computational Linguistics (2015). https://doi.org/10.3115/v1/P15-2008, http://aclweb.org/anthology/P15-2008
Vu, X.S., Jiang, L.: Self-adaptive privacy concern detection for user-generated content. In: Proceedings of the 19th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing), Volume 1: Long papers (2018)
Vu, X.S., Jiang, L., Brändström, A., Elmroth, E.: Personality-based knowledge extraction for privacy-preserving data analysis. In: Proceedings of the Knowledge Capture Conference, K-CAP 2017, pp. 45:1–45:4. ACM, New York (2017). https://doi.org/10.1145/3148011.3154479, http://doi.acm.org/10.1145/3148011.3154479
Wang, R., Wang, X., Li, Z., Tang, H., Reiter, M.K., Dong, Z.: Privacy-preserving genomic computation through program specialization, pp. 338–347. CCS (2009)
Wu, Z., Wang, Z., Wang, Z., Jin, H.: Towards privacy-preserving visual recognition via adversarial training: a pilot study. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11220, pp. 627–645. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01270-0_37
Zhang, Y., Ding, N., Soricut, R.: SHAPED: shared-private encoder-decoder for text style adaptation. In: The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) (2018)
Acknowledgement
This work is supported by the Federated Database project funded by Umeå University, Sweden. The computations were performed on resources provided by the Swedish National Infrastructure for Computing (SNIC) at HPC2N center. The authors also thank the myPersonality project for data contribution.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 Springer Nature Switzerland AG
About this paper
Cite this paper
Vu, XS., Tran, S.N., Jiang, L. (2023). dpUGC: Learn Differentially Private Representation for User Generated Contents (Best Paper Award, Third Place, Shared). In: Gelbukh, A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2019. Lecture Notes in Computer Science, vol 13451. Springer, Cham. https://doi.org/10.1007/978-3-031-24337-0_23
Download citation
DOI: https://doi.org/10.1007/978-3-031-24337-0_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-24336-3
Online ISBN: 978-3-031-24337-0
eBook Packages: Computer ScienceComputer Science (R0)