Abstract
“You are what you read.” As this sentence implies, reading is important for building our minds. We are investing a huge amount of time for reading to input information. However the activity of “reading” is done only by each individual in an analog way and nothing is digitally recorded and reused. In order to solve this problem, we record reading activities as digital data and analyze them for various goals. We call this research “reading-life log.” In this chapter, we describe our achievements of the reading-life log. A target of the reading-life log is to analyze reading activities quantitatively and qualitatively: when, how much, what you read, and how you read in terms of your interests and understanding. Body-worn sensors including intelligent eyewear are employed for this purpose. Another target is to analyze the contents of documents based on the users’ reading activities: for example, which are the parts most people feel difficult/interesting. Materials to be read are not limited to books and documents. Scene texts are also important materials which guide human activities.
Keywords
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Table 7.1 and Figs. 7.1, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8, 7.9, 7.10, 7.11, 7.12, 7.13, 7.17, 7.18, 7.19, 7.20 are originally published in [1] and copyrighted by IEICE. They are granted to use in this article with the permission number 16KB0074. The research described in this chapter has been approved by the research ethics committee in Osaka Prefecture University.
References
K. Kise, S. Omachi, S. Uchida, M. Iwamura, A trial for development of fundamental technologies for new usage of character and document media. J. Inst. Electron. Inf. Commun. Eng. 98(4), 311–327 (2015)
D. Karatzas, F. Shafait, S. Uchida, M. Iwamura, L. Gomez, S.R. Mestre, J. Mas, D.F. Mota, J.A. Almazan, L.P. de las Heras, ICDAR 2013 robust reading competition, in Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013) (2013), pp. 1484–1493
D. Karatzas, L. Gomez, A. Nicolaou, S. Ghosh, A. Bagdanov, M. Iwamura, J. Matas, L. Neumann, V.R. Chandrasekhar, S. Lu, F. Shafait, S. Uchida, E. Valveny, ICDAR 2015 robust reading competition, in Proceedings of the 13th International Conference on Document Analysis and Recognition (ICDAR 2015) (2015), pp. 1156–1160
S. Uchida, Text Localization and Recognition in Images and Video, Handbook of Document Image Processing and Recognition (Springer, 2014)
Q. Ye, D. Doermann, Text detection and recognition in imagery: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 37(7), 1480–1500 (2015)
Y. Matsuda, S. Omachi, H. Aso, String detection from scene images by binarization and edge detection. IEICE Trans. Inf. Syst. (Japanese edition), J93-D (3), 336–344 (2010) (in Japanese)
R. Huang, P. Shivakumara, Y. Feng, S. Uchida, Scene character detection and recognition with cooperative multiple-hypothesis framework. IEICE Trans. Inf. Syst. E96-D (10), 2235–2244 (2013)
H. Takebe, S. Uchida, Scene character extraction by an optimal two-dimensional segmentation. IEICE Trans. Inf. Syst. (Japanese edition) (D), J97-D (3), 667–675 (2014) (in Japanese)
Y. Kunishige, Y. Feng, S. Uchida, Scenery character detection with environmental context, in Proceedings of the 11th International Conference on Document Analysis and Recognition (ICDAR 2011) (2011), pp. 1049–1053
A. Zhu, R. Gao, S. Uchida, Could scene context be beneficial for scene text detection?Pattern Recognit. 58C, 204–215 (2016)
A. Shahab, F. Shafait, A. Dengel, S. Uchida, How salient is scene text? in Proceedings of the 10th IAPR International Workshop on Document Analysis Systems (DAS2012) (2012), pp. 317–321
R. Gao, S. Uchida, A. Shahab, F. Shafait, V. Frinken, Visual Saliency Models for Text Detection in Real World. PLoS one 9(12), e114539 (2014)
S. Wang, S. Uchida, M. Liwicki, Y. Feng, Part-based methods for handwritten digit recognition. Front. Comput. Sci. 7(4), 514–525 (2013)
S. Toba, H. Kudo, T. Miyazaki, Y. Sugaya, S. Omachi, Ultra-low resolution character recognition system with pruning mutual subspace method, in Proceedings of the 2015 International Conference on Consumer Electronics—Taiwan (ICCE-TW) (2015), pp. 284–285
M. Goto, R. Ishida, Y. Feng, S. Uchida, Analyzing the distribution of a large-scale character pattern set using relative neighborhood graph, in Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013) (2013), pp. 3–7
M. Goldstein, S. Uchida, A comparative study on outlier removal from a large-scale dataset using unsupervised anomaly detection, in Proceedings of the 5th International Conference on Pattern Recognition Applications and Methods (ICPRAM2016) (2016), pp. 263–269
T. Saito, H. Yamada, K. Yamamoto, On the data base ETL9 of handprinted characters in JIS Chinese characters and its analysis. IEICE Trans. Inf. Syst. (Japanese edition) (D), J68-D(4), 757–764 (1985) (in Japanese)
T. Nakai, K. Kise, M. Iwamura, Use of affine invariants in locally likely arrangement hashing for camera-based document image retrieval, in Document Analysis Systems VII, vol. 3872, Lecture Notes in Computer Science, (2006), pp. 541–552
S. Ahmed, K. Kise, M. Iwamura, M. Liwicki, A. Dengel, Automatic ground truth generation of camera captured documents using document image retrieval, in Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013) (2013), pp. 528–532
S.M. Lucas, A. Panaretos, L. Sosa, A. Tang, S. Wong, R. Young, K. Ashida, H. Nagai, M. Okamoto, H. Yamamoto, H. Miyao, J. Zhu, W. Ou, C. Wolf, J.-M. Jolion, L. Todoran, M. Worring, X. Lin, ICDAR 2003 robust reading competitions: entries, results and future directions. Int. J. Doc. Anal. Recognit. (IJDAR) 7(2–3), 105–122 (2005)
S. Lucas, ICDAR 2005 text locating competition results, in Proceedings of the 8th International Conference on Document Analysis and Recognition (ICDAR2005), 1, (2005), pp. 80–84
A. Shahab, F. Shafait, A. Dengel, ICDAR 2011 robust reading competition challenge 2: reading text in scene images, in Proceedings of the 11th International Conference on Document Analysis and Recognition (ICDAR 2011) (2011), pp. 1491–1496
K. Wang, S. Belongie, Word spotting in the wild, in Proceedings of the 11th European Conference on Computer Vision (ECCV2010), Part I (2010), pp. 591–604
K. Wang, B. Babenko, S. Belongie, End-to-end scene text recognition, in Proceedings of the 13th International Conference on Computer Vision (ICCV2011) (2011), pp. 1457–1464
R. Gao, F. Shafait, S. Uchida, Y. Feng, A hierarchical visual saliency model for character detection in natural scenes. Camera-Based Document Analysis and Recognition, LNCS 8357, 18–29 (2014)
M. Iwamura, T. Matsuda, N. Morimoto, H. Sato, Y. Ikeda, K. Kise, Downtown osaka scene text dataset, in Proceedings of the 2nd International Workshop on Robust Reading (IWRR2016) (2016) (in printing)
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A.Y. Ng, Reading digits in natural images with unsupervised feature learning, in Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning (2011), p. 9
M. Iwamura, M. Tsukada, K. Kise, Automatic labeling for scene text database, in Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013) (2013), pp. 1397–1401
H. Saito, Y. Sugaya, S. Omachi, S. Uchida, M. Iwamura, K. Kise, Generation of character patterns from sample character images, in IEICE Technical Report, PRMU2010-287 (2011) (in Japanese)
T.S. Cho, S. Avidan, W.T. Freeman, The patch transform. IEEE Trans. Pattern Anal. Mach. Intell. 32(8), 1489–1501 (2010)
S. Belongie, J. Malik, and J. Puzicha, Shape context: a new descriptor for shape matching and object recognition, in Advances in Neural Information Processing Systems (2000), pp. 831–837
M. Iwamura, T. Sato, K. Kise, What is the most efficient way to select nearest neighbor candidates for fast approximate nearest neighbor search? in Proceedings of the 14th International Conference on Computer Vision (ICCV 2013) (2013), pp. 3535–3542
M. Iwamura, T. Tsuji, K. Kise, Memory-based recognition of camera-captured characters, in Proceedings of the 9th IAPR International Workshop on Document Analysis Systems (DAS2010) (2010), pp. 89–96
Y. Lamdan, H.J. Wolfson, Geometric hashing: a general and efficient model-based recognition scheme, in Proceedings of the 2nd International Conference on Computer Vision (ICCV1988) (1988), pp. 238–249
N. Asada, M. Iwamura, K. Kise, Improvement of word recognition accuracy with spellchecker based on tendency of recognition error of characters, IEICE Technical Report, 110(467), PRMU2010-268, pp. 183–188 (2011) (in Japanese)
M. Iwamura, T. Kobayashi, K. Kise, Recognition of multiple characters in a scene image using arrangement of local features, in Proceedings of the 11th International Conference on Document Analysis and Recognition (ICDAR 2011) (2011), pp. 1409–1413
D.G. Lowe, Object recognition from local scale-invariant features, in Proceedings of the International Conference on Computer Vision (1999), pp. 1150–1157
T. Kobayashi, M. Iwamura, T. Matsuda, K. Kise, An anytime algorithm for camera-based character recognition, in Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013) (2013), pp. 1172–1176
T. Matsuda, M. Iwamura, K. Kise, Performance improvement in local feature based camera-captured character recognition, in Proceedings of the 11th IAPR International Workshop on Document Analysis Systems (DAS2014) (2014), pp. 196–201
X. Liu, J. Samarabandu, An edge-based text region extraction algorithm for indoor mobile robot navigation, in Proceedings of the 2005 IEEE International Conference on Mechatronics and Automation, 2, (2005), pp. 701–706
K. Kunze, H. Kawaichi, K. Kise, K. Yoshimura, The wordometer—estimating the number of words read using document image retrieval and mobile eye tracking, in Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013) (2013), pp. 25–29
S. Ishimaru, J. Weppner, K. Kunze, A. Bulling, K. Kise, A. Dengel, P. Lukowicz, In the blink of an eye—combining head motion and eye blink frequency for activity recognition with Google Glass, in Proceedings of the 5th Augmented Human International Conference (2014), pp. 150–153
Y. Shiga, T. Toyama, Y. Utsumi, A. Dengel, K. Kise, Daily activity recognition combining gaze motion and visual features, in PETMEI 2014: 4th International Workshop on Pervasive Eye Tracking and Mobile Eye-based Interaction, Proceedings of the 16th International Conference on Ubiquitous Computing (2014), pp. 1103–1111
K. Kunze, Y. Shiga, S. Ishimaru, K. Kise, Reading activity recognition using an off-the-shelf EEG—detecting reading activities and distinguishing genres of documents, in Proceedings of the12th International Conference on Document Analysis and Recognition (ICDAR2013) (2013), pp. 96–100
Y. Utsumi, Y. Shiga, M. Iwamura, K. Kunze, K. Kise, Document type classification toward understanding reading habits, in Proceedings of the 20th Korea-Japan Joint Workshop on Frontiers of Computer Vision, 3, (2014), pp. 11–17
K. Kunze, Y. Utsumi, Y. Shiga, K. Kise, A. Bulling, I know what you are reading: recognition of document types using mobile eye tracking, in Proceedings of the 17th Annual International Symposium on Wearable Computers (2013), pp. 113–116
O. Augereau, K. Kise, K. Hoshika, A proposal of a document image reading-life log based on document image retrieval and eyetracking, in Proceedings of the 13th International Conference on Document Analysis and Recognition (ICDAR2015) (2015), pp. 246–250
K. Kunze, H. Kawaichi, K. Yoshimura, K. Kise, Towards inferring language expertise using eye tracking, in CHI’13 Extended Abstracts on Human Factors in Computing Systems (2013), p. 6
K. Yoshimura, K. Kunze, K. Kise, The eye as the window of the language ability: estimation of english skills by analyzing eye movement while reading documents, in Proceedings of the 13th International Conference on Document Analysis and Recognition (ICDAR2015) (2015), pp. 251–255
H. Fujiyoshi, K. Yoshimura, K. Kunze, K. Kise, A method of estimating English skills using eye gaze information of answering questions of English exercises, IEICE Technical Report, 115(24), PRMU2015-10, pp. 49–54 (2015) (in Japanese)
K. Kunze, K. Masai, M. Inami, Ö. Sacakli, M. Liwicki, A. Dengel, S. Ishimaru, K. Kise, Quantifying reading habits: counting how many words you read, in Presented at the UbiComp’15: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (2015), pp. 87–96
S. Ishimaru, K. Kunze, K. Tanaka, Y. Uema, K. Kise, M. Inami, Smart eyewear for interaction and activity recognition, in Presented at the CHI EA’15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (2015), pp. 307–310
T. Kimura, R. Huang, S. Uchida, M. Iwamura, S. Omachi, K. Kise, The reading-life log—technologies to recognize texts that we read, in Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013) (2013), pp. 91–95
A. Okoso, K. Kunze, K. Kise, Implicit gaze based annotations to support second language learning, in Proceedings of the 2014 ACM Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp2014) (2014), pp. 143–146
R. Biedert, G. Buscher, S. Schwarz, J. Hees, A. Dengel, Text 2.0, in Proceedings of the 28th ACM Conference on Human Factors in Computing Systems (CHI2011) (2011)
T. Toyama, W. Suzuki, A. Dengel, K. Kise, User attention oriented augmented reality on documents with document dependent dynamic overlay, in Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR2013) (2013), pp. 299–300
K. Masai, Y. Sugiura, K. Suzuki, S. Shimamura, K. Kunze, M. Ogata, M. Inami, M. Sugimoto, AffectiveWear: towards recognizing affect in real life, in Presented at the UbiComp/ISWC’15 Adjunct: Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers (2015), pp. 357–360
S. Sanchez, T. Dingler, H. Gu, K. Kunze, Embodied reading: a multisensory experience, in Presented at the CHI EA’16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (2016), pp. 1459–1466
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer Japan KK
About this chapter
Cite this chapter
Kise, K., Omachi, S., Uchida, S., Iwamura, M., Inami, M., Kunze, K. (2017). Reading-Life Log as a New Paradigm of Utilizing Character and Document Media. In: Nishida, T. (eds) Human-Harmonized Information Technology, Volume 2. Springer, Tokyo. https://doi.org/10.1007/978-4-431-56535-2_7
Download citation
DOI: https://doi.org/10.1007/978-4-431-56535-2_7
Published:
Publisher Name: Springer, Tokyo
Print ISBN: 978-4-431-56533-8
Online ISBN: 978-4-431-56535-2
eBook Packages: Computer ScienceComputer Science (R0)