Computational Learning Analytics to Estimate Location-Based Self-Regulation Process of Real-World Experiences

A learner can autonomously acquire knowledge by experiencing the world, without necessarily being explicitly taught. The contents and ways of this type of real-world learning are grounded on his/her surroundings and are self-determined by computing real-world information. However, conventional studies have not modeled, observed, or understood a learner's self-regulation mechanism of real-world learning. This study developed computational learning analytics to estimate how this mechanism works. Our analytics segmented a time series of real-world learning into units of a cognitively closed and semantically independent function by estimating the spatiotemporal clusters of a learner's concentrated stay behavior. We found that learners' intercluster moves functioned to determine whether they maintained or changed their contents and strategies of real-world learning. We also found that the spatiotemporal sizes of the estimated clusters were correlated with the activeness and diversity of strategy-based content examinations at each location. This study forms a basis for automatically estimating qualitative transitions of real-world learning and encouraging a learner to obtain a better understanding of the world.


Computational Learning Analytics to Estimate
Location-Based Self-Regulation Process of Real-World Experiences Masaya Okada , Koryu Nagata, Nanae Watanabe , and Masahiro Tada Abstract-A learner can autonomously acquire knowledge by experiencing the world, without necessarily being explicitly taught.The contents and ways of this type of real-world learning are grounded on his/her surroundings and are self-determined by computing real-world information.However, conventional studies have not modeled, observed, or understood a learner's self-regulation mechanism of real-world learning.This study developed computational learning analytics to estimate how this mechanism works.Our analytics segmented a time series of real-world learning into units of a cognitively closed and semantically independent function by estimating the spatiotemporal clusters of a learner's concentrated stay behavior.We found that learners' intercluster moves functioned to determine whether they maintained or changed their contents and strategies of real-world learning.We also found that the spatiotemporal sizes of the estimated clusters were correlated with the activeness and diversity of strategy-based content examinations at each location.This study forms a basis for automatically estimating qualitative transitions of real-world learning and encouraging a learner to obtain a better understanding of the world.Index Terms-Computational learning analytics, grounded cognition, location-based context estimation, real-world learning.

I. INTRODUCTION
A. Real-World Learning M ANY effective educational practices are traditionally performed in a classroom with textbooks [1] or electronic materials [2].On the other hand, it is claimed that a person can spontaneously learn by being involved in a real-world situation, without necessarily being explicitly taught [3].An important alternative to classroom learning is real-world learning by body-based behavior to acquire knowledge from out-of-class real-world experience [3].
In contrast to "a laboratory setting" or "a learning setting driven by artificially preprogrammed stimuli," our research defines the term "real world" as "the world where human activities occur in a natural setting as they really are without being artificially or strictly controlled" [4].The reason why we focus on the real world is that experiments in which stimuli are artificial and response options are fixed inevitably result in findings that are less ecologically valid in relation to real-world behavior [5].Ecological validity is the consistency between experimental conditions and real life [5] and is a basis of a research design aimed toward understanding natural behavior.
An example of a real-world field-study area is a natural environment where various phenomena spontaneously occur in a symbiotic relationship among various living things, e.g., plants and animals.Our model case of real-world learning is a self-directed and collaborative style of environmental learning by human-world interaction involving walking and exploring in a natural environment (see Fig. 1).Grounded cognition (i.e., cognition derived from and grounded on the world) [6] during the learning is shared on site via human-human interaction among collaborative learners.

B. Learning Analytics
Many recent studies on learning analytics have focused on assessing the process of classroom learning [7], online learning [8], and blended learning [9], rather than real-world learning.Various analytics scenarios were considered for learning at a table, classroom, or school, but it is still an unsolved issue to create learning analytics for unconstrained collaboration in large physical learning environments [10].
Multimodal learning analytics has been implemented using various data, such as behavior data (e.g., eye movement, speech, and face expression), physiological data (e.g., electroencephalogram), and environmental data (e.g., humidity, CO 2 concentration) [11].However, it is claimed that existing studies on multimodal learning analytics have shown the potential to accurately predict simple educational events/behaviors but still struggle to capture complex learning situations (e.g., collaboration quality, the state of acquiring insufficient ideas to progress) [12], [13].
How a learner purposefully self-regulates his/her learning processes to accomplish the learner's own learning goals [14], [15] is high-level assessment information hard to trace from the outside of the learner [16].For example, during real-world learning, a learner should self-regulate to reconsider where, what, and how to learn in a wide and uncontrolled field-study area when his/her grounded cognition does not work well to find useful information from the surroundings.However, no learning analytics has been proposed to assess whether a learner at a nonprospective situation can self-regulate to behave to dramatically change the state of real-world learning.

C. Experience-Based Symbolic Computation
On the basis of the discussions by Marr's [17], we consider "computation" as the operational process that a system (e.g., a human, an information system) converts the raw and primitive input of a symbol set into an output symbol set with a higher abstraction level of semantics.Computation inside a person is not directly observed from the outside but can be assumed at every information processing underlying human activities [18].A mathematical or algorithmic representation of computation can provide explicit testable predictions about learning and behavior [19], although a black-box model with an unknown mechanism can neither be proved nor disproved.
In collaborative environmental learning, each learner behaves to perform on-site symbolic computation for experientially understanding the nature of real-world information.For example, symbol grounding [20] is performed by observing real-world information based on each learner's viewpoint, verbally sharing and cooperatively interpreting the meaning of the information, and grounding learners' internal understanding of phenomena observable in the world.Furthermore, symbol emergence [21] is realized by collaboratively inferring the structure of a real-world issue (e.g., the reasons or causal relationship behind observed phenomena), and generating or reorganizing the semantic network structure of their internal knowledge space.
These symbolic computations are experientially accomplished by body-based behavior to acquire grounded cognition in the world.We assume that a learner does not necessarily passively receive grounded cognition from the world but his/her internal autonomous computation self-regulates how grounded cognition via his/her body should be acquired to better experientially understand the world.However, no study has explicitly assumed or tested a mathematical or algorithmic representation of the mechanism to self-regulate this type of computation derived from and grounded on the world.

D. Research Issue
Complex intelligent behavior can arise from surprisingly straightforward interactions between an agent and its environment [22], [23].Embodiment (i.e., the role of a body as an enabler for cognition or thinking [24]) allows agents to discover various ways to achieve their goals through sensorimotor shortcuts: by actively manipulating their own perceptual inputs through motor activity [22].A causal and constitutive relation between one's cognitive state and body state appears in a bidirectional manner [25], which creates a semantic congruency between body movements and cognition [26].
Here, an important question in the studies of grounded and embodied cognition is how specific body states and movements matter for specific forms of thinking and learning [26].However, little is known about 1) the mechanism by which a learner's internal computation is made under grounded cognition in a field-study area, 2) the mechanism by which 1's computation is embodied externally as a coordinated state of sensorimotor functions, and 3) the technique to detect sensorimotor controls that correspond to the self-regulation state of 1's computation.
A person's cognition underlying his/her behavior generation is a systematic process affected by his/her internal beliefs [27].Thus, we develop computational learning analytics of the selfregulation process of real-world learning, which assumes a learner as a computation system (e.g., a Bayesian decisionmaking system) to orchestrate sequential executions of algorithmic components under real-world situations.To be concrete, we 1) model a learner self-regulating how to compute real-world information derived from grounded cognition, 2) assume how the learner's externally observable behavior is accompanied by the self-regulation state of his/her internal computation, and 3) develop a sensor-based technique to estimate the self-regulation state of behavior-based computation in real-world learning (e.g., self-determination of where, what, and how to learn in the world in order to break the saturated state of learning with the low possibility of further knowledge acquisition).

II. RELATED WORKS
For designing learning analytics to trace a complex selfregulated process of learning, it is indispensable to integrate theoretical models and frameworks from multiple disciplines including educational and computational sciences [16].In this section, we discuss how to integrate and enhance the theories of self-regulated learning (i.e., internal autonomous computation), grounded cognition (i.e., real-world oriented cognition), computational behavior modeling (i.e., behavior-based decisionmaking system), and research design (e.g., science of natural behavior).

A. Self-Regulated Learning
Self-regulation of learning (SRL) is adaptively building and coordinating a repertoire of strategies to accomplish one's Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
Since learners often have difficulties in performing selfregulation at the SRL and SSRL levels during their learning, such self-regulation should be promoted by educational interventions [16].However, studies on multimodal learning analytics have not featured prescriptive analytics that provides educational recommendations based on sensor data [12].A general problem is that the executions of learners' SRL-level and SSRL-level self-regulation are hard to capture from the outside with a computer system [16].A domain-specific problem is that little was known about the mechanism by which a learner's strategies are self-regulated to make grounded cognition work better in a given real-world situation.Based on these global and domain-specific problems, as a basis to develop prescriptive analytics adapted to the state of each learner's self-regulation, we find traceable features to estimate the mechanism by which a series of a learner's strategy executions is self-regulated in the world.

B. Grounded Cognition
The theory of grounded cognition [6] explains that a person 1) behaves to perceive the world with his/her multimodal sensors (e.g., visionary, auditory, and tactile organs), 2) associates his/her perceptual, motor, and introspective states during direct experience, 3) internally creates a practical representation of the semantics of the world, and 4) grounds the representation on his/her direct experience in the world, not only on indirect experience (e.g., conceptual knowledge).Umwelt (i.e., the viewpoint that a person actively gets necessary information from his/her self-centered world) [35] and affordance (i.e., the viewpoint that a person behaves according to the passively received information from the environment) [36] are different perspectives of real-world cognition.However, the theory of grounded cognition integrates these two perspectives by modeling the process that multimodal sensorimotor information is self-organized in a person's brain to perform symbolic computations derived from real-world experiences [6], [21].
Within a grounded cognition framework, where concepts' representations involve partial replay of perception and action experience [37], a recent study [38] suggested that word concepts can be assessed by measuring semantic similarity that is grounded in perceptual or action resemblances (i.e., the similarity of sensorimotor experience).Another study [26] explored the cognitive relevance of particular body states (e.g., directed actions) to students' mathematical reasoning in the area of geometry.However, there was no basic assumption about how internal cognition of a learner (including the SRL by orchestrating a series of strategy executions) works and is embodied as behavior while processing real-world-oriented information during real-world learning.Although grounded cognition [6], [21] is basically a model of human-world interaction, our current study extends the discussion on how the generation state of a learner's behavior is computationally self-regulated to construct an internal cognitive representation of the semantics of the world not only by human-world interaction in the context of the real world but also by human-human interaction to collaboratively share learners' states of grounded cognition.

C. Computational Behavior Modeling
Recently, computational modeling with mathematical representations has been applied to behavioral science, in order to implement algorithmic hypotheses that explicitly explain how behavior is generated and also to gain a deeper understanding of behavioral data [39]1 .A typical example of this type of computational behavior modeling is the learning of behavior selection to maximize a human's reward in the case where the most rewarding choice is initially unknown (e.g., multiarmed bandit task).Studies on computational behavior modeling have shown that a person taking actions evaluates and learns the value of the actions internally, and this learning process can be modeled by using or modifying computational models implemented with the sigmoid function, the softmax function, and the reinforcement learning algorithm [40], [41], [42], [43].This idea has been recently applied in the field of computational psychiatry to investigate the correlation between the generation of prosocial behavior (i.e., actions that benefit others) and the characteristics of the psychiatric disorder [44].In the setting of an online game, computational behavior modeling was used to estimate the factor (e.g., learning progress maximization, perceptual novelty, or random search) that drove a learner's exploration under the conditions of minimal task instructions and no external rewards [45].
However, real-world learning has been outside the current focus of computational behavior modeling, and no method has modeled, observed, or understood the mechanism of a learner's internal computation to self-regulate his/her action selection that is grounded on and adapted to the context changed by behavior in a real-world learning environment.Thus, we propose methods for modeling, tracing, and assessing this computation mechanism to achieve better experiential understanding of the world in a given real-world situation (e.g., computation for finding and executing better behavior to increase the expected amount of real-world information acquired from the surroundings).

D. Real-World-Oriented Design Methodology
We can learn many important lessons from conventional computational modeling that investigated behavior in indoor, online, or laboratory experiments by systematically controlling or excluding parameters (e.g., explanatory variables, extraneous variables) under ideal and artificial conditions [41], [42], [43], [44], [45], and by repeatedly investigating how each individual parameter affects other parameters [40].However, we should consider that assessing cognitive processes in narrow and artificially isolated conditions with little ecological validity would not represent the actual functioning of cognition [46] and would not necessarily generalize to real-world cognition [47].
To maintain ecological validity, not all parameters (e.g., parameters related to a learner or the world) can or must be strictly controlled in an artificial manner.Thus, a wider range of possible values should be assumed for each of the internal parameters that drive a learner's computation.In fact, different from laboratory experiments allowing no or only a few self-determined behaviors [40], [41], [42], [43], [44], [45], learners in a natural and unconstrained situation in the world were able to execute far wider ranges of strategies to generate behavior (e.g., 115 kinds of strategies in a setting of our previous experiments [4]).Furthermore, in such a natural situation, different parameters can co-occur inside a learner, but mutual influence among co-occurring parameters is hard to discriminate without strict and artificial experimental controls.
We consider that, for a practical use of computational learning analytics under an ecological valid situation, a learner's computation should not be modeled at the level that is too reactive to the complexity and dynamics caused by being situated in the real world.This current study proposes and practices a realworld-oriented design methodology of computational learning analytics applicable to ecologically valid situations of learning, consisting of the policies of 1) constructing a simple and ideal model of a learner's internal computation of behavior generation in the world, 2) avoiding overfitting each of detailed computational components assumed as a learner's ideal model, 3) abstracting essential functions of series of computations in the ideal model, and 4) obtaining estimation results at a practically necessary level of granularity.

A. External and Internal Controls
This section discusses how a learner self-regulates the content of learning (i.e., what to learn by examining spatial information) and the way of learning (i.e., how to learn by executing strategies), given two types of controls: 1) external control (mainly from the viewpoint of affordance [36]) and 2) internal control (mainly from the viewpoint of Umwelt [35]).

1) External Control by Spatial Context:
The environment is the world of experience [48].Observational evidence that a real-world learner finds at various locations in the world can be seed information for acquiring knowledge [49].Importantly, the value of each location in the world is not the same; for example, the density of unique vegetation in a field-study area differs by location [50].At a place where interesting objects exist, a learner makes profound inquiries.At uninteresting places, his/her active inquiries are restricted.These facts mean that each location of the world has different spatial characteristics for encouraging or discouraging on-site cognition and spatial inquiries [49].
We consider that each location of the world has a "spatial context" [51], i.e., nonlinguistic, external, and potential situation information with which a person at a certain location is involved.Traditionally, the semantics of a space has a passive role that is used as reference information to identify the world where the target of a human's real-world action (including discourse) should be grounded [52], [53], [54].However, this article proposes an extended idea that spatial context has an active role in determining the range of real-world information that a learner at a location can potentially access, cognize, and process.This idea assumes that spatial context functions as hidden affordance to explicitly or tacitly restrict, activate, and regulate a learner's cognition, strategy executions, and behavior generation in the world.
2) Internal Control by Strategy Executions: In real-world learning, learners collaboratively execute diverse strategies during spatial inquiry behaviors in the world [4].Each learner's strategies are sequentially executed when he/she 1) autonomously selects a location to be investigated, 2) inputs the surrounding seed information to the learner's computation system, 3) finds a question about an observable phenomenon, 4) outputs his/her internal understanding to be collectively examined with other cooperative learners, and 5) updates his/her internal cognition about the semantic structure of his/her surrounding world.We consider that, a learner's internal strategy functions as the primitive and computational information to self-determine the way that his/her grounded cognition works at a certain time point in the given spatial context.

B. Hypothesis: Stay and Move as Learning Regulation
Although his/her surroundings give a learner contextual information (see Section III-A1), the learner is not a slave to his/her given situation.For example, even if a real-world situation encourages a learner to adopt a certain behavior, he/she may not necessarily output a behavior on which he/she places no value.
A person's expectancies are "beliefs about a future state of affairs" and are derived from his/her knowledge or schemas about the world [55].Expectancies are subjective estimates of the likelihood of future events ranging from merely possible to virtually certain [55].By considering that expectancies are used to guide effective behavior and to regulate single and sequential behaviors [55], the current article proposes the following hypotheses to estimate a learner's self-regulation of real-world learning under the external and internal controls.
1) Concentrated Stay Behavior: During concentrated stay behavior by remaining at a location, a learner extracts cognitive resources (e.g., the seed of a question and knowledge) embedded in the current surroundings and continuously makes intensive inquiries toward a solution.A learner behaves based on his/her local cognition of what is physically possible at his/her stay location, which encourages him/her to subjectively interpret spatial context of each location of the world.When each learner in a group has exhaustively examined all objects of interest at a Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.location, the group reaches the saturation state, which is the state in which new information, questions, hypotheses, discussion topics, or strategies are not expected to be found.Continuously staying at the same location gradually decreases the entropy of observable information.
2) Location Switch Behavior: Moving to a high-entropy location having different cognitive or semantic roles changes the contextual constraints when a learner is searching for a solution.An ecosystem has a horizontal structure into which natural living things self-organize [50] and observation from a single location is not enough to form a complete image of an ecosystem [56].Location switch behavior drastically increases a learner's expectation to access a larger amount of observable information and new questions, and to complement his/her partial understanding of the world.Switching a learner's stay location stirs up cognitive dissonance between his/her previously acquired knowledge and newly perceived information, and then triggers new strategy executions for spatial inquires.
3) Sensing Body-Based Learning Regulation: While a learner subjectively estimates a nonlow value of the likelihood to encounter new information at his/her current location (in other words, while the state of real-world learning is not saturated), concentrated stay behavior at the same location occurs to actively and diversely make on-site computation for mutually associating his/her direct and indirect experiences and for constructing an internal representation of the semantics of the world.When a learner cannot expect further experience at his/her location, location switch behavior to self-determine where to learn in the world embodies learning regulation not only to break the saturation state of on-going learning but also to make a learner's grounded cognition system work in a different spatial context.By sensing the time-series occurrence of location switch behavior from externally observable features of a learner's movement (e.g., position data), we can estimate the units into which a learner subjectively segments the semantics of the real-world spatial context.

IV. ESTIMATION ALGORITHM OF LEARNING REGULATION
We model a learner who subjectively estimates the value of his/her surrounding spatial context.Fig. 2 expresses that different cognition is made due to being situated at a different location; this type of cognition repeatedly encourages a learner to find a prospective content of learning at each location and to execute strategies applicable to the content.Based on this model, our proposal was formulated by 1) assuming the mathematical mechanism of a learner's hidden and sequential computations, 2) finding a key computational function in the mechanism, and 3) developing an approximate estimation method of the self-regulated state of real-world learning.

A. Ideal Model to Self-Evaluate Location's Value
By using the sigmoid function, the softmax function, and the reinforcement learning algorithm, the internal computation of a human can be modeled as a Bayesian decision-making system that updates his/her internal belief to generate behavior [40], [41], [42], [43], [44], [45].Assuming that real-world learning is location-oriented computation, the present study proposes an extended model to express that the degree of a learner's belief is determined by the conditional probability and conditional expectation given by his/her stay location.
Specifically, as the ideal, this research models that learning in a location's spatial context provides a learner with the expected effect E(S i |l) Here, r(s i,j |l) is the effect caused by a learner i executing a strategy s i,j at a given location l. S i is the set of possible strategies of learner i. P (S i = s i,j |l) is the prior probability that strategy s i,j occurs at location l.E(S i |l) is the potential effect achieved by learner i from executing all possible strategies S i by staying at location l P (D) defined using the sigmoid function sig(x) is the probabilistic decision about where to be situated for learning.The variable D is defined as either "remain," which indicates staying at the same location, or "switch," which means going to a different location.
Ê(S i |l) is learner i's internal belief about E(S i |l), constructed by learning at location l.Δ Ê(S i |l) is the difference in Ê(S i |l) updated by learning at location l per unit time.θ i,l is learner i's internal expectation of the effect acquired at location l per unit time.θ i,l represents learner i's personal belief about a location's potential to give him/her new experience (e.g., finding new seed information).θ i,l is given a high value for locations where various objects exist, and θ i,l is given a low value for those where few objects exist.Since a learner cannot actually execute the large number of all possible strategies, he/she cannot know the true values of r(s i,j |l) or P (s i,j |l).However, the occurrence probability of each executed strategy can be calculated as the ratio of the number of times the strategy was executed to the total number of strategy executions at location l.Ê(S i |l) can be gradually updated as learner i executes strategies at location l and makes a step-by-step estimation of r(s i,j |l).As learning proceeds at location l, the state of the learning gradually reaches the saturation state that learner i can hardly obtain any new effect.Therefore, while learning at the same location, the change rate Δ Ê(S i |l) is decreasing, and the value of Ê(S i |l) approaches that of E(S i |l).
While the learner believes that his/her estimated Δ Ê(S i |l) is greater than threshold θ i,l , he/she remains at the same location to learn.When he does not believe this, the learner switches to a different location.In short, learning is self-regulated to drastic effect by changing where to learn when a learner cannot acquire a result that fits the amount of effort that he/she actually expends by executing a series of strategies based on the estimated value of those strategies.

B. Approximate Estimation of Potential Information Existence
Section IV-A presented our ideal model of learning regulation by assessing the expected effect of executing possible strategies at a certain location.It is possible to manually list all possible strategies, but the number of strategies executed in real-world learning is large (e.g., 115 kinds of strategies in the setting of our previous experiments [4]).When trying to automatically estimate E(S i |l) of (1), we encounter the following difficulties.First of all, it is difficult to automatically estimate strategy execution at every time point because a strategy is part of high-level behavior semantics that needs to be manually annotated by experimenters from video-based observation [4].Second, it is hard to determine the value of r(s i,j |l) given independently by each individual strategy at a location, since a wide variety of strategies spontaneously occur in a real-world context, and are sometimes used in combination.As discussed in Section II-D, for our research purpose, the effect of individual strategies should not be investigated under artificial conditions without a real-world context.
Instead of directly estimating each internal value computed by a learner, in this article, we find an approximate solution to estimate the change of state dynamics of his/her internally self-regulated computation process.Since the amount of information existing at a certain location is not infinite, a long stay at the same location decreases not only the probability of obtaining new experience but also the value of staying at the location (see Section III-B).A learner's internal indicator regarding whether to continue to learn at a location is the probability that unknown important information remains in his/her surroundings.Thus, without directly measuring r(s i,j |l) or E(S i |l), we perform approximation by estimating the degree that a learner has finished accessing the information embedded at the location, for example, by evaluating the density of his/her spatial investigation at the location.

C. Estimation of Concentrated Stay Behavior
Using a local positioning system, in our previous study, we developed a time-series estimation method for the body state called "stable stay" that frequently holds during a learner's active real-world inquiries (e.g., observation, visual inspection, knowledge exchange, and cooperative thinking) [49].Stable stay was defined as the motion state that a learner stands still at a point in space in the sense of keeping his/her rate of body orientation changes within a certain range (T θ [deg/s] ≤ 60) and not moving (T v [m/s] ≤ 0.1) for at least a specified period of time (T t [s] ≥ 15).Stable stay was estimated by a thresholding technique to automatically find time series of sensor data that simultaneously satisfied all state conditions determined by the three parameters T θ , T v , T t .However, stable stay was not defined in consideration of a natural ecosystem, where observation targets with the same or similar characteristics often gather within a relatively small area.
By extending the estimation of stable stay, this study extracts concentrated stay behavior, i.e., a spatiotemporally high density of continuous stays within a certain sized location cluster (see Fig. 3).From the viewpoint of Umwelt [35], we expect that each spatiotemporal cluster of concentrated stay behavior is the result of an individual learner cognitively segmenting the semantic functions of his/her self-centered world.From the viewpoint of affordance [36], we expect that each cluster has a different functional role to afford location-oriented cognitive processing (e.g., information conversion, knowledge acquisition, and strategy execution).

D. Algorithm for Spatiotemporal Clustering
By sensing spatiotemporal density of time-series stay behavior, we extract semantic clusters where a learner's cognition, strategies, and behavior complementally function as the same or a similar role in grounded cognition in the world.Gridding a field-study area into equally sized subspaces (e.g., 10 m×10 m grid squares) has been shown to be useful for clarifying the spatial characteristics that promote behavior occurrence in each Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.subspace [51].However, subspaces should be clustered as variably sized areas, because the real-world areas of a learner's interest are not uniform in size or a particular shape.
Our first trial of kernel density estimation with position sensing data did not obtain good clustering results, because it estimated the probability density function of stay locations as nonzero even at physically impractical locations (e.g., a location in the pond).In contrast, for the current study, we propose the following set of assumptions, which are original to this study.
1) An intracluster move corresponds to continuously examining the same or similar contents with a limited range of strategies.2) An intercluster move corresponds to a change of contents and strategies of learning.3) An intercluster move is larger than an intracluster move.Based on these assumptions, we perform spatiotemporal clustering of points of learners' stays based on the density of time-series position sensing data.As an implementation, we use the algorithm of ST-DBSCAN [57] that groups together high-density points with many spatiotemporal neighbors 2 .
ST-DBSCAN does not require an assumption on the number of clusters.It works based on the parameters of spatial nearness (EPS1), temporal nearness (EPS2), and the minimum number of points to construct a cluster (minPoint).Assume that p(x, y, t) expresses a learner located at (x, y) at time t.Given points p 1 (x 1 , y 1 , t 1 ) and p 2 (x 2 , y 2 , t 2 ), the parameters of ST-DBSCAN are formulated as follows: When a learner visits the same location (small value of EPS1) twice at different time points (large value of EPS2), ST-DBSCAN estimates the possibility of assigning the visits to two different clusters.Height data (z) are not used in this study, because our experimental field-study area does not have differences in altitude.With our wearable sensor system (implemented with a single-board computer "Raspberry Pi"), we obtain each learner's position data (5 Hz) of a global navigation satellite system (GNSS).Conventional receivers of the global positioning system (GPS) have meter-order errors of position estimation [49], but the accuracy of position estimation with the GNSS sensor is improved to centimeter order by the realtime kinematic method.We observed how the spatiotemporal distribution of previous learners' stay positions was related to the time-series occurrence of semantically similar topics and strategies.Then, we experimentally determined the parameter values used in ST-DBSCAN as EPS1 = 2.5 (m), EPS2 = 300 (s), and minPoint = 50 (points).

V. GROUND TRUTH DATA OF THE SEMANTICS OF TIME-SERIES REAL-WORLD LEARNING
By extending our hybrid analysis of externally and internally observable data [4], [49], the present study generates ground truth data that code how grounded cognition works in humanworld and human-human interactions.

A. Heterogeneous Records 1) External Observation Data:
We obtain multimodal behavior data of each learner with our wearable sensor system, such as head and body posture (motion sensors with 3-axis gyroscope, 3-axis accelerometer, and 3-axis compass), utterance (microphone), and vision (first-person view camera).We also obtain data of third-person's view video of learners' behavior, which are captured by handheld video cameras operated by experimenters.Video and motion data are used to identify the content and target of each learner's behavior, collaboration, and investigation.Since conversation protocol reflects the collective cognition shared by people [58], we use the data of conversation to trace the internal understanding and strategies of each learner.
2) Internal Observation Data: By encouraging each learner to draw a network-style activity map (see Fig. 4) 3 just after finishing real-world learning (e.g., at an off-site classroom), we can extract his/her on-site activities to find, examine, and understand a relationship (i.e., arcs) among different real-world oriented concepts (i.e., nodes) [49].A learner clarifies the semantic role of each real-world activity by adding the following attribute information to each arc: 1) example of a concept, 2) general knowledge (a learner's background knowledge), 3) question, 4) hypothesis to explain a relation between concepts, 5) observation of a phenomenon in the world carefully watched for a period of time, 6) verification of behavior to examine and verify the hypothesis, and 7) discovery (new knowledge obtained through observation, discussion, and hypothesis verification).As introspective data, our current study makes extended use of an activity map to know the process and result of each learner's grounded cognition to associate on-site experiences and his/her knowledge, to hypothesize the mechanism behind his/her observable world, and to experientially understand the semantics of the world.

B. Time-Series Learning Content: What to Learn
Not all experiences are genuinely or equally educative [48].We manually represent time-series on-site activities that contributed toward achieving the summative knowledge structured in an activity map [49].For this purpose, we first extract learning topics from each learner's activity map (see Fig. 4) by carefully manually grouping semantically related contents of node(s) and arc(s) [4].Examples of topics are "rotten branches at the waterside" and "difference of growth of moss at different locations" [4].Second, we identify and annotate the time points when each topic was examined in the world (t 1 -t 2 , t 3 -t 5 in Fig. 5) by an off-site video-based manual check to find how each learner's contents of on-site conversation and behavior correspond to topic components (i.e., nodes, arcs) of his/her activity map.Time t 6 -t 8 not corresponding to any topic was the time to examine the content that a learner did not externalize in his/her activity map as his/her summative result.
For the present study, a graph representation is constructed to compute the content of learning.An activity map G is a directed labeled graph, and is expressed as G = (N, A).N is a finite set of k nodes, expressed as N = {n 1 , n 2 , . . ., n i , . . ., n k }, where element n i is the ith node in the activity map.A is a finite set of l arcs, expressed as A = {a 1 , a 2 , . . ., a j , . . ., a l }, where element a j is the jth arc in the activity map.The pth topic TP p is a subgraph of G and consists of the finite sets N p (N p ⊆ N ) and A p (A p ⊆ A), which is expressed as TP p = (N p , A p ). Specifically, TP p is expressed as a variable-length list of a fixed set of parameters, e.g., Fig. 4 showing TP 5 = ({n 1 , n 5 }, {a 4 , a 10 }).

C. Time-Series Strategy Execution: How to Learn
Using heterogeneous data sources (see Section V-A), we code the time series of how each learner is self-directed to execute a strategy for operating on a certain target (see Fig. 5).
A strategy execution at a certain time point is expressed as strategy = (operation, target), i.e., a definite combination of an operation code and a target code (see Table I).Each time point can be annotated with several strategy codes, in order to express multiplex strategy execution in the world [4].In order to list up the target and operation codes, we originally observed approximately 250 learners' strategies to perform physical actions on phenomena or objects in the world, to cognitively manipulate a learner's own or other learners' knowledge spaces, and to self-regulate a learning process [4].Many of the general operation codes we designed have been widely used in traditional learning sciences (e.g., "infer," "explain," "assess" [59], [60], [61]), but our target codes were extended to express targets of human-world and human-human interactions, e.g., real-world information, a learner's individual cognition, and others' cognition [4].Furthermore, the present study additionally observed 30 learners' behavior to update our previous code list with supplementary codes to express grounded cognition, for example, grounding real-world symbols (e.g., O24: ground) and operating the strength of personal belief (e.g., O30: intensify, O54: weaken).Combining an operation code (from 55 types) and a target code (from 44 types) can express a wide range of strategies even after excluding logically incompatible combinations of parameters.
Rather than using the concrete name of each individual object, a target code is defined as a compressed expression of the conceptual category of the operated-on targets.Thus, target expresses the substantial meaning of experience and computation derived from the world.This is our technique for expressing a strategy at the abstract level without overfitting the diverse objects and phenomena in the world.Our strategy expression is not overly sensitive to small changes of observation targets in the world, but rather traces the categorical and semantic changes of the operated-on targets.
Based on our coding scheme, multimodal data including discourse between the group members are used for annotating strategies including SRL-level strategies (e.g., "O15: criticize, T27: own behavior"; "O9: change, T18: goal of own learning") and SSRL-level strategies (e.g., "O44: re-propose, T16: goal of group's learning").The traditional studies of SRL and SSRL established behavior indicators and coding schemes of cognitive, metacognitive, and behavioral operations underlying intellectual activities in various learning settings [62], [63], [64], and our current study uses them to identify, ascertain, and annotate how operation and target are taking place in a field-study area.

VI. INTRACLUSTER AND INTERCLUSTER DYNAMICS
The semantics of time series of real-world learning is expressed as meta-level information about what was learned and how the learning was made, i.e., content examinations (see Section V-B) and strategy executions (see Section V-C).We create two types of metrics to express the dynamics of contents and strategies of real-world learning that changes through a learner's location-oriented computation.

A. Intracluster Dynamics
The following data sequences are synchronized with the time parameter t to assess how concepts and relationships were examined by strategy executions at each estimated location cluster.

1) location(t) = {x(t), y(t), locationClusterLabel(t)}
Data of position and location cluster at time point t.LocationClusterLabel(t) is the unique identifier of estimated location clusters.The number of location clusters differs between learners.

2) topic(t) = {nodeList(t), arcList(t)}
Content data of what was learned at time point t.N odeList(t) and arcList(t) are variable-length string lists of the elements of N and those of A (see Section V-B), respectively.

3) strategyList(t)
Content data of how learning was performed at time point t.StrategyList(t) is a variable-length string list consisting of one or more strategies.A strategy is expressed as a pair (operation, target), as explained in Section V-C.The number of executions of content codes (e.g., node, arc, operation, target) at a cluster is an index of activeness of learning at the cluster.The number of unique codes executed at a cluster is an index of diversity of learning at the cluster.

B. Intercluster Dynamics
The semantics of real-world learning (i.e., the contents and ways of learning) at a location is expressed by identifying time points with a designated LocationClusterLabel, and extracting the contents of nodeList, arcList, and strategyList at the time points.For verifying the semantic independence of learning at each allocated cluster, we define the intercluster similarity of the contents and strategies that a learner i examined and applied at the dth location cluster to those at the subsequent (d + f )th location cluster.
Given sets X and Y , the similarity of the elements of the two sets is calculated using the Jaccard index ) We define J i,f as the average of Jaccard index values to express the similarity of meaning of learner i's activities among all possible combinations of the dth and (d + f )th location clusters.For example, in the case of f = 1, J i,f represents the average degree of how the meaning of a learner i's activities at each cluster was similar to the meaning at the temporally following cluster: Here, M i,d is a variable-length list expressing the meaning of the activities that learner i conducted at the dth location cluster.m i,d,j is the jth value of nodeList, arcList, or strategyList that learner i executed at the dth location cluster.As an example for nodes, assume that learner 5 staying at location cluster 10 conducted an examination of not only a topic consisting of nodes n 20 , n 25 but also another topic consisting of nodes n 100 , n 101 , n 105 .This case is expressed as M 5,10 = {m 5,10,1 , m 5,10,2 }, where m 5,10,1 = {n 20 , n 25 } and m 5,10,2 = {n 100 , n 101 , n 105 }.Similarly, consider the cases of arcs and strategy executions J f (0 ≤ J f ≤ 1) is index J i,f averaged over all n learners.This average index shows the similarity of the contents or strategies of all learners' activities among possible combination of the dth and (d + f )th clusters.

A. Objective
No conventional method of low-level sensing (e.g., location sensing) has been used to automatically estimate highabstraction-level semantics of real-world learning (e.g., semantically independent units of learning regulation).The objective of our analysis is to verify 1) whether a location cluster with a high spatiotemporal density of stay locations can reflect the semantic and functional unit of real-world learning, 2) whether an intercluster move can reflect self-regulation of real-world learning, and 3) whether quantitative features (e.g., spatial size and temporal size) of estimated clusters can reflect the quality of real-world learning at locations.

B. Method
We analyzed the data of 22 adult learners (from 20 to 29 years old) who voluntarily participated in our experiments on environmental learning at a forest of Kamigamo Experimental Station, Field Science Education and Research Center, Kyoto University, Japan. 4Each experiment was conducted for 1 h in a group learning style (three learners as a group).During the experiment, GNSS position data necessary for our estimation method were automatically recorded with a learner's wearable sensor, without any learner operations.The total time amount of the analyzed data was 79 431 s (22.1 h).
No participant was acquainted with us or other group members before the experiments.Our pre-questionnaire showed that learners' prior-experience of participation in environmental learning programs was 0.3 ± 0.8 times (mean ± SD).The level of their prior-experience did not dramatically vary among the learners, and our participants were novice learners who did not have much empirical knowledge about environmental learning.The frequency of learners' computer use was 4.6 ± 2.4 times per week.
1) Task: By extending the conventional jigsaw method in classroom learning [65], we developed a learning task named the real-world jigsaw method [66].In our task, each real-world learner was given different memo fragments about the academic theories of real-world phenomena (e.g., survival strategies of a plant and the community ecology of plants).The memo fragments were shown on each learner's tablet computer by just tapping its display, similarly to the general operation of smartphones that most of the learners' generation use daily.Learners with different memo content were encouraged to complement each others' understanding and to cooperatively build and examine hypotheses about the ecosystem in their field-study area.Just after finishing real-world learning on site, each learner drew an activity map in a classroom.
2) Preinstructions: Based on the conventional jigsaw method [65], our preinstructions (including Q&A) were given to the learners so that they could understand our real-world jigsaw method and could be accustomed to necessary operations to view memo fragments on a tablet computer (for about 10 min).Before the experiment, learners had sufficient time (about 20 min) to get accustomed to moving while wearing sensors, although they did not need to operate their sensors.After real-world learning, learners were instructed how to draw an activity map with a laptop computer (for about 10 min).These experimental settings were for encouraging learners to behave naturally in the experiment, without feeling much in difficulty or much experiencing the novelty effects.In fact, 5-point scale questions (5: strongly positive, 4: positive, 3: neutral, 2: negative, 1: strongly negative) in our postquestionnaire after the experiment showed that the learners could easily use memo content on a tablet computer 4 The plan of our field study was reviewed and approved by Kamigamo Experimental Station, Field Science Education and Research Center, Kyoto University, Japan.To obtain informed consent based on the Japan Society for the Promotion of Science (JSPS) research ethics guidelines, before the experiment, we carefully explained not only the purpose and content of the experiment but also the benefits, rights, disadvantages, and privacy protection of the experimental subjects, using explanatory documents.Persons who agreed with and signed the consent document of their own free will participated in the experiment.From our 30 learners' data, this study used the data of 22 learners who did not encounter technical troubles with the single-board computer that they wore (e.g., electronic breakdown of sensors or storage modules, due to rainy weather in the forest environment).

TABLE II FREQUENCIES OF SELF-DETERMINING WHERE, WHAT, AND HOW TO LEARN
DURING THE EXPERIMENTS (4.2 ± 0.7) and an activity map (4.0 ± 1.0).The postquestionnaire also showed that the wearable sensors on a learner's body did not give the learner a sense of discomfort (4.3 ± 0.7) and did not restrict behavior in a natural environment (4.3 ± 0.9).
3) Analytical Data: GNSS data of each learner were used as the test data of our estimation method.Multimodal data records of learners (e.g., utterance, vision, behavior, and introspection) were used to generate the ground truth data of the semantics of time-series real-world learning (see Section V).Nodes and arcs of an activity map were self-coded by each learner and were not differently interpreted by the experimenters.Each learner's strategy execution was carefully hand-labeled with multimodal records by an annotator with more than four years of specialist education in the field of environmental learning.Following a method to verify the reliability of the annotation [67], we randomly selected data covering 20% of the experiment time period so that a different person could independently annotate learners' strategy executions.The κ coefficient (i.e., the degree that independent annotators could make the same judgment without negotiation) [68] was 0.854.A κ coefficient in the range 0.81-1.00indicates that the similarity of interpretations by different annotators was "almost perfect" (the highest level of consistency of different annotators' interpretations) [67].

1)
Overall Features: Since our real-world jigsaw method diversified learners' viewpoints from which to observe the world, various contents (336 topics consisting of 251 kinds of nodes and 266 kinds of arcs) were examined by various strategies (196 strategies consisting of 44 operation codes and 32 target codes) over the course of all experiments.That 80.0% (44/55) of our operation codes and 72.3% (32/44) of target codes were used for describing real-world learning indicates the good coverage of our basic coding scheme.
Table II shows the frequencies at which learners selfdetermined where, what, and how to learn during the experiment.Learners switched location clusters at the low rate of 0.172 exec./min,changed topics at the middle rate of 0.315 exec./min, and changed strategies at the high rate of 2.54 exec./min.The frequencies of strategy changes and topic changes were 14.8 times and 1.83 times higher than the frequency of location switches, respectively.As modeled in Fig. 2, learners first selected a prospective location to learn, second extracted a concrete topic of learning from potential learning contents there, and third selectively executed a sequence of strategies to examine the topic.
Table III, created by the method of Section V-C, is a typical example of sequential strategy executions to start and end realworld learning at an estimated location cluster.After identifying and sharing a learner's learning goal (i.e., the examination of survival strategies of a plant) (06:40-07:10), a learner group first visually inspected the feature of the leaves of a pine tree, and inferred their attributes (e.g., a feature, quality) from the viewpoint of photosynthesis efficiency (07:12-07:19).When grounding their knowledge (i.e., what they knew about a pine tree) on the real world (i.e., observational evidence of an actual pine tree in front of them) (07:39-08:02), the gap between their knowledge and the reality generated a question derived from the world (08:12-08:46).After this question triggered off a series of a thought experiment, hypothesis constructions, and a discussion about methods to verify their hypotheses (08:47-11:45), they finally found the limitations of staying at their location for giving them concrete answers to their questions and hypotheses (11:46-11:47), and considered the possibility of forming a better understanding at other locations (11:48-12:00), which triggered starting new learning at the next location cluster.
Outside of the estimated location clusters, we often found that few or no strategies were sequentially executed to focus on a particular real-world target for pursuing a concrete learning goal.

TABLE IV FEATURES OF ESTIMATED CLUSTERS
2) Intra-Cluster Features: Fig. 6 shows a randomly selected result of clustering a learner's location data with our method.Black dots are position data not allocated to any location cluster.Dots of other colors are position data allocated to a location cluster.Our method estimated the possibility to allocate different clusters to a revisited place (e.g., points A and B in the figure).In terms of time, 83.8% of the data were allocated to clusters.As shown in Table IV, a learner's learning (for about one hour per learner) was divided into 10.4 clusters on average.Learning at one cluster was conducted for 4.9 min over an area of 9.3 m 2 on average.As shown in Table V, on average, learning at one cluster examined 6.4 topics consisting of 3.5 nodes (i.e., concepts) and 2.9 arcs (i.e., relationships) of an activity map.On average, at one cluster, learners executed 6.3 kinds of strategies consisting of 4.1 operation codes and 4.8 target codes.Importantly, limited numbers of contents and strategies were examined at each estimated cluster.
3) Intercluster Features: On average, a learner who had finished 4.9 min of learning at a location moved to another location cluster (see Table IV).J f (defined in Section VI-B) in Table VI shows that sets of strategies (operation, target) and sets of topics (node, arc) occurring in an estimated cluster were not reused at subsequent clusters.Although sets of topics, nodes, and arcs were slightly highly reused in the next cluster (f =1), 5  J f for all possible combinations of clusters was low in all cases. 5Strategies relatively commonly used among clusters include basic ones to objectively observe and collaboratively examine the world, e.g., (O10: clarify, T30: peripheral information), (O22: explain, T6: concrete knowledge), (O22: explain, Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.At each estimated cluster, the limited range of on-site information was selected as the targets to which the limited range of strategies were applied (see Section VII-C2).Although we found some intercluster similarities of learning contents and strategies, the contents and strategies at a cluster had high semantic independence from those at other clusters (see Table VI).Our strategy expression is not overly sensitive to small changes in the observation targets in the world but rather traces the categorical and semantic changes of the operated-on targets (see Section V-C).Thus, intercluster difference of target means that the substantial meaning of real-world experience differs between the estimated clusters.
Estimation of stable stay [49] was limited, because this estimation was to find the point in space where a learner actively made observations and had discussions while standing still.Our proposed method extended this to estimate spatiotemporal focal areas that elicit different grounded cognition, and explicitly or tacitly determine the range of possible behavior and available achievements.Each estimated cluster was the functional unit of spatial context that afforded semantically independent experiences and cognitively closed symbolic computations.

4) Correlation Between Cluster Features and Learning:
We found no statistical significance between the number of estimated clusters and total achievement of learning (e.g., quantitative features of learners' activity maps).However, as shown in Table VII, stay time at an estimated cluster was positively T23: observation target), (O22: explain, T22: observation result), (O5: approve, T22: observation result), (O40: propose, T28: own idea), (O40: propose, T29: own question), (O44: re-propose, T29: own question), (O25: identify, T25: other's idea), (O5: approve, T25: other's idea), (O10: clarify, T13: difference between phenomena), and (O28: infer, T34: reason for phenomenon).Nodes correlated with the activeness and diversity of content examinations and strategy executions (0.59 ≤ r ≤ 0.69, p < 0.001).The area size of a cluster was weakly positively correlated with the activeness and diversity of content examinations and strategy executions (0.35 ≤ r ≤ 0.43, p < 0.001).The reason why correlation coefficients of area size were lower than those of stay time is considered to be that a learner in too large an area can see many phenomena (including potential concepts and relationships) but cannot necessarily examine each individual phenomenon intensively enough to organize structured knowledge.
The quality of real-world learning is hard to objectively assess from the externally observable features of a learner, and no conventional method of low-level sensing has been used to automatically estimate semantic features of real-world learning.Our location-based sensing method automatically calculated spatiotemporal features of an estimated cluster as metrics for the activeness and diversity of intellectual operations on survey targets.
5) Estimation of Body-Based Learning Regulation: Our important finding was that keeping and changing the location of learning functioned as learning regulation that was embodied in a learner's body in the spatial context of the world.A learner's intercluster move was key behavior not only to update spatial context that a learner can potentially sense but also to switch content examination and strategy execution.For this reason, our density estimation of concentrated stay behavior was able to successfully extract the time point when a learner started examining new phenomena from a different viewpoint to reveal and arcs continuously examined among clusters include concepts, phenomena, relationship, or findings on which a learner mainly focused.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
the nature of his/her self-centered world, without maintaining the old contents and strategies of learning.

D. Discussion
1) Computation Model: We assumed that a learner i at location l estimates and updates Ê(S i |l), i.e., his/her internal belief of E(S i |l), which is the potential effect that can be achieved from executing all possible strategies S i while staying at location l.Given this assumption, a learner should continue to learn at the same location while he/she can still obtain a positive effect from learning at the location.In fact, Section VII-C4 showed that during a long stay at the same location, a learner was able to intensively and repeatedly examine the same phenomenon and find relationships between concepts that were not previously known to be associated with each other.This result also supports that when a learner inappropriately gives up learning at a location where potentially useful real-world information has so far been overlooked (i.e., the case that Ê(S i |l) is underestimated), they cannot sufficiently acquire the knowledge summarized as nodes and arcs of an activity map.
Since the number of observation targets at a location is not infinite, knowledge newly acquired by staying at one location will gradually decrease with the passage of time.By continuing concentrated stay behavior at a single location, Ê(S i |l), the learner's subjective estimate, approaches the true value E(S i |l) of the potential of location l.When a learner finds that his/her learning reaches the saturation state that real-world information, strategies, and behavior are seldom updated (i.e., the state that Δ Ê(S i |l) is small), the learner should self-regulate to appropriately give up the old location where additional learning effects are not expected.In our experiments, learners at the saturated state often proposed searching and moving to locations appropriate for finding a new learning topic or observational evidence to verify their hypotheses.After conducting 4.9 min of learning on average (see Table IV), as learning regulation to break the saturation state of the information circulation system, learners did an intercluster move to start time series of relearning at a new location in a different spatial context.
When learners persist in studying in a situation where the value of Ê(S i |l) is overestimated relative to the true value of E(S i |l), too long a stay at the same single location is expected to decrease the possibility that additional contents and strategies can be generated from peripheral observation.Positive correlation among stay time and learning effect (see Section VII-C4) is considered to indicate that 1) learners in our experiments succeeded in appropriately giving up on a learning location with poor prospects, and 2) their location switch behavior allowed them to learn in a situation where the true value of E(S i |l) was higher than the estimated value of Ê(S i |l) (i.e., the situation where unfound meaning of the location still remained).
In summary, our results supported our internal regulation model of location-oriented computation in which a learner 1) sequentially evaluates the result of a preceding behavior, 2) predicts the expected results of a possible future behavior, and 3) behaves to increase the possibility of encountering new observational evidence.Based on this model, we precisely estimated an intercluster move, as a location-derived signal to start his/her sequential execution of different strategies to reveal the nature of the brand-new experience.
2) Estimating Semantics of Real-World Learning: For recognizing motion concepts, multimodal representation of human actions in a household environment has been proposed based on a probabilistic description of the kinematics of an action along with its contextual background, namely the location and the objects held during the action [69].Behavior semantics (e.g., the reason why and the way how a human is operating on a certain set of objects) was used to model the human cognitive processing [70].The multigranularity features and statistics of word use have been used not only to find appropriate sentences for knowledge-grounded conversation [71] but also to predict destructive or out-of-context phases of conversation [72], [73].Different from conventional studies, the unique feature of our estimation method is not using the following information: 1) knowledge about the role of objects existing in the world; 2) knowledge of semantics of each location; 3) data of conversation content containing a wealth of behavior semantics 6 .Conventional studies [74] have investigated time-series strategy executions changed by self-judging the strategies' effects.However, as discussed in Section I-B, learning analytics has difficulties to automatically estimate a high-level learning situation [12], [13], such as the self-regulation state of learning [16].In fact, conventional studies have not automatically assessed how a learner with grounded cognition internally self-regulates experience-based symbolic operations in the spatial context of the world.A main difficulty in estimating the regulation state of real-world learning is that a learner's hidden internal computation and his/her observable sensor data are not in one-to-one correspondence.
Our challenge is to find the features of low-level (motionlevel) behavior accompanied by strategy-based computation to construct a high abstraction level of cognitive representation of the world.We showed that a spatiotemporally high density of continuous stays within a certain sized area occurs when a learner makes spatial inquiries in the same or similar semantic context of learning.Estimation of spatiotemporal concentrated stays enabled us to extract the semantic units into which each individual learner subjectively segmented the spatial context of the world.Without knowing a learner's self-regulated strategies (e.g., s i,j ) or the actual value of his/her internal estimation of potential effect achieved by staying at a location (e.g., E(S i |l)), our location-based estimation method functioned as an approximate solution for extracting time points to change the regulation state of strategic control of a learner's internal knowledge space under grounded cognition.
3) Internal Computation Underlying Grounded Cognition: Our current study assumed that real-world learning is selfcontrolled by a learner's internal computation to make cognition (i.e., a psychological system) and body (i.e., a biological system) work as a single entity in a holistic manner.Although how sensorimotor functions work during language learning [22], word conceptualization [38], and mathematical reasoning [26] has been investigated, the discussion still remains open in many domains about how to model and trace human computation underlying coordinated systems of cognition and body.On this point, our current study found that a state pattern of a learner's embodied behavior (i.e., concentrated stay behavior) reflects his/her internal computation that controls psychological and biological systems to well achieve grounded information processing.
4) Sensor-Based Actuation Adapted to Learner's Internal Computation: Studies of multimodal learning analytics have not featured prescriptive analytics that provides educational recommendations based on sensor data [12].Furthermore, the executions of learners' SRL-level and SSRL-level self-regulation were hard to capture from the outside with a computer system [16].Our analytics can be a basis for a sensor-based actuation adapted to each learner's internal computation, by 1) automatically sensing a learner's regulation state of real-world learning, and 2) actuating him/her to well self-determine or selfregulate experience-based symbolic computations.For example, when a learner often gives up learning without enough time for concentrated investigation, the engine can give him/her behavior actuation to focus on the current context.When a learner overly persists at the same location without any learning regulation, behavior actuation can be given to promote relearning in a new prospective context.
5) Design Methodology of Computational Learning Analytics: Different from conventional computational modeling under ideal and artificial conditions [40], [41], [42], [43], [44], [45], our interest was how to model, trace, and assess the invisible and complex process of self-regulated computation by which intelligent behavior arises from learner-world interactions.By applying our real-world-oriented methodology to design computational learning analytics, we theoretically devised detailed formulas to express the probabilistic regulation process of realworld behavior under grounded cognition, and made a sensinglevel approximation of essential functions of the formulas.
Our design methodology consists of the following steps: 1) mathematically assuming the ideal (true) model of a learner's internal computation underlying behavior generation in the complex world; 2) determining which abstraction level of estimation information is required for the research purpose; 3) extracting the essential function(s) of the ideal computation model; 4) approximating human computation at a machine-traceable level to be automatically estimated with sensors.These steps are not domain-specific, and we expect that the basic ideas of our design methodology can constitute a guideline for practically adapting computational learning analytics to other learning settings.

6) Extensibility of Strategy Codes:
As explained, to create ground truth data of the semantics of time-series real-world learning, we defined strategy codes by observing approximately 280 learners.High use ratios of our operation and target codes (80.0% and 72.3%, respectively; Section VII-C1) indicate that our strategy codes well covered the learning activities in our task setting.StrategyList(t), i.e., the representation of strategies at a certain time point, is defined as a variable-length string list to be flexibly extended.When our analytics is applied to real-world learning with other tasks or in other fields, the strategy codes can be supplemented, as the present study did, by making preliminary experiments to observe potential strategies that learners can execute during that new task or in that new field.
7) Limitations and Future Work: a) Inter-personal Behavior Coordination: SRL and SSRL constitute a co-constructive and inseparate process, because the dynamic processes leading to productive engagement in a collaborative activity are regulated through a continuous individual and social process (i.e., a combination of individual and collective regulation of strategies) [64].The concentrated stay behavior effective for tracing sequential strategy executions is expected to be under both SRL and SSRL regulation, but its behavior state is defined in terms of parameters of an individual's body.
On the other hand, intercorporeality (i.e., a form of reciprocity of bodily intentionality between embodied subjects [75]) is expected to work during learners' collaboration in real-world learning.This point was outside the main focus of the current paper, but we expect that interpersonal coordination of learners' bodies (e.g., synchrony) can reflect the group-level collective regulation of learning (e.g., SSRL [16]).In fact, the positions and orientations of people's bodies are interpersonally coordinated to create, maintain, and break a particular formation for participating in a joint transactional space and are important physical information to identify a time series of not only each person's role (e.g., speaker, listener) but also their style of engagement in social interaction [76], [77].
Assuming that positions and orientations of people's bodies are shared and coregulated during SSRL-level orchestration of real-world learning, we expect that, in addition to parameters to express a learner's location (e.g., x, y in a 2-D plane, x, y, z in a 3-D space) at a certain time point t, 3-axis orientation information of his/her head and upper body (e.g., o h,x , o h,y , o h,z , o b,x , o b,y , o b,z ) at each time point can be potentially useful parameters for determining a behavior-level computation model underlying SSRL-level orchestration of real-world learning.Extending our learning analytics with a supplemental algorithm driven by the orientation parameters (including the optimization of internal parameters and hyperparameters to be used in the supplemental algorithm) will be a basis to objectively quantify the state of learners' intercorporeality and to find SSRLlevel collaboration problems occurring in real-world learning.
b) Transfer Learning in Sociocultural and Self-Directed Context: Transfer learning is learning how to apply knowledge to different tasks, problems, situations, or institutions [78], and metacognitive skills underlying transfer learning can be obtained by the process of sequential regulations of learning [79].Our estimation technique enabled automatically extracting the start and end time points of each occurrence of learning regulation in the world.However, this technique was not for identifying how the series of occurrences of learning regulation are mutually associated to achieve a higher level of knowledge, such as transferable knowledge.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
Recently, machine learning techniques have been developed to represent the mechanism of transfer learning by imitating human information processing (e.g., a brain function to construct and classify memory connection [80], an ability to describe a new category by other related knowledge in a textual and visual space [81]).However, transfer learning can occur not only in sociocultural environmental contexts (e.g., cultural artifacts, the structure of social activities) [78] but also in the context of a learner's agency (i.e., actor-oriented dynamic situations for inventing and reorganizing relations of similarity between activities) [82].Conventional studies [80], [81] did not computationally model transfer learning achieved by self-regulating, associating, and contextualizing a series of sociocultural behaviors grounded in the world.Our future study will investigate how a learner's physical body works to drive and self-regulate the process of transfer learning by making his/her computation grounded on a dynamic and socioculturally contextualized environment.
c) Prescriptive Learning Analytics: As a basis to develop prescriptive analytics adapted to the state of each learner's internal computation, our computational learning analytics precisely estimated the regulation state of real-world learning with the task of our real-world jigsaw method.Our field-study area is a forest environment that is close to human's living areas and maintains a symbiotic relationship among humans and the nature.In general, this type of forest environment is often used for the purposes of nature observations, ecological surveys, nature games, and environmental learning [4].Since our computational modeling represented a general and simple computation process underlying real-world learning, we expect that our analytics is applicable to learning settings (e.g., tasks, fields, and age groups) in which the grounded cognition of a learner can be similarly modeled.On this point, our future work should further accumulate experimental results by applying our analytics to a wider age range of learners (e.g., elementary schoolchildren, the elderly), a wide variety of field-study areas (e.g., a virgin forest), other tasks (e.g., learning subjects, field survey methods), or a larger dataset.
Our future work should also assess educational effects achieved by our new potential service, e.g., an on-site sensorbased actuation adapted to each learner's internal computation.Although hardware implementation of a sensor system is not the main focus of the current paper, improving its robustness to work in various real-world situations (including rainy weather in a forest environment) is also important work for our estimation method.

VIII. CONCLUSION
An unsolved issue in learning analytics is to capture high-level semantics of learning [12], [13].In real-world learning, a learner constructs an internal cognitive representation of knowledge by not only human-world interaction in the context of the real world but also human-human interaction to collaboratively share learners' states of grounded cognition.However, little was known about how to computationally model and analyze the selfregulation process by which real and collaborative "experience" generates learning effects derived from the world.
For automatically tracing a complex self-regulated process of real-world learning, we integrated interdisciplinary theories of self-regulated learning (i.e., internal autonomous computation), grounded cognition (i.e., real-world oriented cognition), computational behavior modeling (i.e., behavior-based decisionmaking system), and research design (e.g., science of natural behavior).Then, we formulated our basic expectation that when a learner self-directs real-world learning, he/she internally calculates and updates the meaning and effect of his/her on-site learning.Although it is difficult to automatically assess how a learner internally estimated the potential effect achieved by on-going experiences, we developed computational learning analytics to estimate the internal regulation state of behavior-based computation in the spatial context of the world.
The central point of our study is finding an automatically traceable pattern of a learner's behavior that embodies his/her internal "control" to compute how to self-regulate a series of intellectual states during real-world learning.In other words, our challenge is to find the features of low-level (motion-level) behavior accompanied by strategy-based computation to construct a high abstraction level of cognitive representation of the world.Here, a novel assumption of ours is that a spatiotemporally high density of continuous stays is accompanied by the same or similar semantic context of real-world learning.Based on this assumption, our sensor-based density estimation of a learner's concentrated stay behavior precisely extracted spatiotemporal clusters with a cognitively closed and semantically independent function to experientially compute location-oriented real-world information.By sensing the repeated process by which a learner determines where to be situated, our method estimated a time series of occurrences of learning regulation to input and compute new real-world information, and to vary and order the content and strategy of real-world learning.These results support our model that the spatial context of each location in the world functions as hidden affordance to explicitly or tacitly restrict, activate, and regulate a learner's self-directing process of realworld computation.
Learning regulation is hard for a learner to perform and also hard for a computer system to trace and support [16].The qualitative transitions of real-world learning are hard to objectively assess from the externally observable features of a learner, but spatiotemporal features of an estimated cluster could be metrics for automatically estimating the activeness and diversity of time-series strategy-based content examinations.Our computational learning analytics can be a basis of sensor-based actuation adapted to each learner's internal computation, such as an engine for encouraging learners to self-determine or selfregulate symbolic computations well by real-world experience.
We expect that basic ideas of our design methodology of computational learning analytics can constitute a guideline practically applicable to other learning settings.While improving the robustness of a sensor system to trace real-world learning, our future work will further show the potential of computational learning analytics by investigating 1) interpersonal behavior for SSRL-level coordination, 2) behavioral features underlying transfer learning in sociocultural and self-directed contexts, and 3) educational services driven by our computational learning analytics.

Fig. 1 .
Fig. 1.Self-directed and collaborative style of real-world learning by sharing the state of grounded cognition in the world.

Fig. 3 .
Fig.3.Estimating a spatiotemporal cluster of semantically similar real-world learning by sensing concentrated stay behaviors for inquiring in the world.

Fig. 4 .
Fig. 4. TP p : pth topic extracted from a learner's activity map (p = 5).Modified figure of the work in [4].Translated from the original language.

Fig. 5 .
Fig. 5. Scheme for coding what and how to learn in the world.Time: Step time.The change of either an operation code or a target code means a state change of strategy execution (e.g., t 4 , t 7 ).

Fig. 6 .
Fig. 6.Estimated result of focal areas where a learner concentrated stays to form a certain meaning of learning.

TABLE I EXPRESSION
OF STRATEGY EXECUTION (operation, target)

TABLE III SERIES
OF STRATEGY EXECUTIONS IN AN ESTIMATED CLUSTER

TABLE V CONTENT
AND WAY OF LEARNING AT ESTIMATED CLUSTERS TABLE VI J f : SIMILARITY OF CONTENTS AND WAYS OF LEARNING IN f TH SUCCEEDING CLUSTER TABLE VII COEFFICIENT BETWEEN CLUSTER FEATURES AND LEARNING ACTIVITIES