A Case Study for Measuring Informal Learning in PLEs

The technological support for learning and teaching processes is constantly changing. Information and Communication Technologies (ICT) applied to education, cause changes that affect the way in which people learn. This application introduces new software systems and solutions to carry out teaching and learning activities. Connected to ICT application, the emergence of Web 2.0 and its use in learning contexts enables an online implementation of the student-centred learning paradigm. In addition, 2.0 trends provide “new” ways to exchange, making easier for informal learning to become patent. Given this context, open and user-centered learning environments are needed to integrate such kinds of tools and trends and are commonly described as Personal Learning Environments. Such environments coexist with the institutional learning management systems and they should interact and exchange information between them. This interaction would allow the assessment of what happens in the personal environment from the institutional side. This article describes a solution to make the interoperability possible between these systems. It is based on a set of interoperability scenarios and some components and communication channels. In order to test the solution it is implemented as a proof of concept and the scenarios are validated through several pilot experiences. In this article one of such scenarios and its evaluation experiment is described to conclude that functionalities from the institutional environments and the personal ones can be combined and it is possible to assess what happens in the activities based on them.


INTRODUCTION
End-user involvement in the design of technology is not new.However, with the emergence of Web 2.0 [1] together with the perceived difficulties in the implementation of Information and Communications Technologies (ICT) in learning, a refocus on the user's role in taking ownership of their learning technology has gained wide support.However whilst the ICT application represents an important advance in other areas (such as culture, society and economy), it does not work with learning [2,3].
The main reasons for this lack of success can be due to: 1) Institutional resistance to change regarding the introduction of certain technologies in formal environments [2,4].2) The insistence on the application of technology when it is not required or seen as a solution [5].3) The need for digital literacy amongst teachers and students, many of whom are digital immigrants and the younger pupil generations which are digital natives [6][7][8].4) The lack of connection between the formal (structured and certified learning related to educational institutions), nonformal (structured learning traditionally associated to workplace training) and informal learning (non-structured learning obtained in daily life) environments [9] make it difficult to improve learning processes, as well as the centralization of the activity in only one context [4,10].5) Moreover, a lot of technological applications and tools are defined without taking into account the final user, which means that adopting and using them can be difficult [2,11].
In order to address these problems, learning institutions need to change their strategies.They must provide environments more adapted and open to the student to include the new set of Web 2.0 tools that are under the student's control [12].The rationale for the shift of this 'locus of control' is that personalization can improve learning by empowering the students to manage their learning at their own pace [13] with their own technology within the context of the activities of their daily lives which are also managed by the same technologies.Consequently, the PLE (Personal Learning Environment) seeks to unburden the learner of the need to learn new systems when they engage in formal learning.PLEs facilitate the user learning process by allowing them to use those tools they want to use and not by linking them to an specific institutional context or learning period [14,15].
However, the introduction of a PLE does not infer the demise of the LMS (Learning Management System) [14].LMSs have been highly successful in stimulating online engagement of teachers and learners and, besides, they are widespread and large amounts of money have been invested in them [16].The likely coexistence of LMSs and PLEs introduces a requirement for interoperation between the two.Tools from the LMS must be included in the PLE, and tools from the PLE can be included in the LMS; activities carried out in the PLE should be reported to the institutional environment as a way to measure the informal activity, etc.
This article describes the different interoperability scenarios focusing on this last concern in order to facilitate measuring informal learning and enrich the PLE with institutional functionalities.

PAPER A CASE STUDY FOR MEASURING INFORMAL LEARNING IN PLES
In order to do this, the context of the research is explained in section 2, which describes the PLE concept including its impact on informal learning, the integration possibilities and the interoperability specifications.Later, in section 3 some interoperability scenarios to measure informal activity are commented upon and service framework is presented.The following section describes the proof of concept implementation and a experiment related to one of those scenarios.Finally, some conclusions are exposed.

II. RESEARCH CONTEXT
The PLE concept is a recent concept defined in 2001 [17].Since then, there has been a wide contribution from different authors on the definition of PLE.
A PLE is a concept, not a thing."The PLE is not a piece of software.It is an environment where people, tools, communities and resources interact in a flexible way" [18].Attwell [13] agrees that a PLE should not be seen as a software application: "Personalized Learning Environment is not an application but a new approach to the use of new technologies in learning…The PLEs provide students their own space to develop and share their ideas, through learning environments that connect resources and contexts so far apart".
Informal Learning is understood as the learning which users acquire away from an institution in the course of daily life.There is therefore a need for recognition of this kind of learning, since it has taken special relevance in the Bologna Process [19] and the support to this type of tools provided by 2.0 tools and PLEs [13,14].
PLEs will provide freedom to the user to define what tools learners use to learn, and will not be constrained by institutional rules (for example, a specific period of time such as an academic year) as an LMS is.LMS's are not generally supportive of informal or lifelong learning.They can only be used in a formal learning setting, managed and controlled by the educational institution.A PLE, however, can connect formal, informal, and lifelong learning opportunities within a context that is centered upon the learner [20].The activity performed in such spaces can be gathered in a personal portfolio [21] and/or taken into consideration in the formal learning environments.
Also in a methodological and pedagogical sense there is a significant impact, summarized in seven factors [14,22]: 1) The learner is going to participate, to collaborate, to share… in the learning experience, that is to say, not just as a consumer but also as provider.2) The learner can personalize the learning process supported by a community of learners and experts.3) PLEs feed of contents through the Internet and the institution does not restrict them.4) The learner does not learn alone but also with other learners and experts not necessarily from the same context.5) Contents can be used, modified, shared only with the restriction of the copyrights and not under the ownership of the institution.6) The learner will be the center of the learning process; s/he is going to be the responsible of his/her learning process while managing his/her own learning.7) Technologically, PLE is based on a set of loosely coupled tools, among which social tools have special relevance, so they can evolve easily and depending on learner necessities.However, as commented above, LMS are not going to be discarded so integration initiatives between LMS and PLE must be taken into account.
A PLE can be seen as an opportunity to improve learning processes providing greater effectiveness due to the fact that they are centered on the learner.Personal Learning Environments can open the institutional ones [13], so it is essential to establish solutions to integrate institutional and non institutional worlds: formal, non formal and informal learning.
In order to do this, Wilson and others [23] proposed three possible scenarios of integration: • PLE and LMS exist in parallel, as informal and formal environments respectively.There are several initiatives along these lines but they are outside the scope of the integration problem.• The second scenario refers to the opening of the LMS through the inclusion of web services and interoperability initiatives.In this scenario the following may be included: iGoogle based initiatives [24], social networks connected with LMS [25], the LMS that offer support for implementations of interoperability specifications [26], PLE with specific communication protocols [27] or integration based on service-oriented architectures -SOA [28].Main difficulties of these initiatives are: the institutional barriers to the opening of formal environments and the fact that those initiatives focus on information exportation and not on interaction exchange.That is to say, communication is unidirectional, from the LMS towards the external tools; basically exchanging information about what happens on the platform and providing no interaction or information back to the LMS.• The third scenario is based on the integration of external tools into the LMS.In these initiatives a user might not decide which tools she is going to use and they will be limited to institutional decisions.Some initiatives in this scenario are: LMS defined for the integration of external tools [29], Google Wave Gadgets integrated into Moodle [30], PLE introducing tools based on log analysis [31], initiatives based on tools integration driven by learning design activities [32], integration architectures [33], etc.These initiatives have several problems such as: integration problems between tools, context integration difficulties and the customisation of student learning tools are constrained by the institution, etc.Those that best overcome these problems are the ones that define a learning platform starting from scratch or from a previous institutional development.They address student and institution requirements and integration in a better way, but they have problems such as the lack of adoption and the necessity of learning to use a new software [34].Taking all these solutions into account, with their problems and how they are faced, a possible solution could be based on the combination of second and third scenarios.In the present article this is done through the use of a servicebased framework and a set of interoperability scenarios.It allows the integration in the PLE of tools from the LMS and any user interaction carried out in the personal environment is recorded in the institutional one.In this way, it would be possible to measure informal learning in a way not explored in previous studies.There are studies related to informal learning evaluation such as [35], other that PAPER A CASE STUDY FOR MEASURING INFORMAL LEARNING IN PLES consider how to assess eLearning activities and propose new ones [36], projects related with PLE such as [37] that use specific ways of measuring informal learning, but none in this specific sense.
One of the ways to integrate LMS and PLE is through the interoperability specifications, as the second scenario described above.They provide standards for ways of interaction between both tools.
In learning environments different kinds of specifications and standards exist which are related to contents (i.e.: SCORM), quizzes (i.e.: IMS QTI), exchange of data (i.e.: IMS Enterprise), metadata in Learning Objects (i.e.: LOM), etc.
Among those specifications one of the most popular, open and complete specification is IMS LTI (also known as Full LTI) because it facilitates a real and full integration between tools and learning platforms.However, many LMSs and tools do not support it due to its difficult implantation.With a view to overcome this, a light version of the specification has been released, Basic LTI (BLTI).This version, implemented by the most representative LMS [26], will allow the creation of an external tool instance inside the learning platform, launching it and providing a Single Sign-on access.However BLTI presents a problem; there is no a real integration only authentication, so there is no exchange of information about the activity performed on the tool towards the LMS.IMS-GLC is working to extend BLTI in order to facilitate outcomes integration into the LMS.

III. INTEROPERABILITY SCENARIOS
Once technological and methodological context is described, some interoperability scenarios are explored, which try to describe a way to provide an informal space where the user can freely develop their learning activities without the restrictions of a close institutional solution [44].That space will communicate with the institutional environment to measure the performed activity, so it will be taken into account by the formal institution.The scenarios are: uation support into the PLE.This scenario aims to gather the students' activity in online tools included in the PLE.These tools are not necessarily educational tools so they are not going to provide an interface to assess the students' outcomes.The teacher defines an instance of the online tool into the LMS; this will create a context that only teachers can access and through which the results of the activity performed by the student can be returned or the evaluation could be facilitated.The student accesses his/her PLE and can use, among others, the online tools adapted to return information about the student's activity to the LMS.The tool in this case is not necessarily created with a learning objective, so they do not include assessment interfaces, something that is needed to grade the student's activity.This assessment interface is provided by the mediator (or proxy tool), which interacts with the online tool, the LMS and facilitates the integration between them.The implementation of these scenarios should be independent of the LMS and/or the underlying technology, which means that solutions such as web services or interoperability specifications are needed.

IV. IMPLEMENTATION AS A PROOF OF CONCEPT
Once technological and methodological context is described, some interoperability scenarios are explored, which try to describe a way to provide an informal space where the user can freely develop their learning activities without the restrictions of a close institutional solution.That space will communicate with the institutional environment to measure the performed activity, so it will be taken into account by the formal institution.The scenarios are: • Scenario 1. Exporting of functionality from the LMS to the PLE.One of the possible interoperability scenarios will be the exportation of functionality to other contexts.The exported institutional functionality is iJET -Volume 9, Issue 5: "ALICE 2013", April 2014 represented in the PLE and provides information about what happens there to the LMS.It will allow the user to introduce formal activities in informal environments, which will allow him/her to participate in the institutional activity even when s/he is learning through other means.There is a special version of this scenario that consists in the representation of such institutional functionality into a mobile device working as if it were a PLE.
• Scenario 2. Use of external tools with external access to them to measure activity.In this scenario no interoperability between the LMS and the PLE is proposed.It takes into consideration the students' activity in the PLE from the institutional environment, but the teacher should assess such activity by accessing other contexts that differ from LMS.For example, a student accesses an online tool from the PLE and performs (in agreement with the teacher) a task by using it; then, the teacher should enter into the online tool or the PLE, check her activity and perform her assessment from the LMS.This scenario is quite common in different institutions and it requires a extra effort on the teachers' part.• Scenario 3. Use of external tools (with evaluation support) in the PLE, and the return of the outcomes to the LMS.In this scenario the activity is done using the external educational tool but it is integrated in to the LMS.The teacher defines an instance of the educational tool into the LMS: this will create a context only accessible by teachers and through which the results of the task completed by the student in the PLE can be recovered; the student accesses his/her PLE and can use, among others, the educational tool adapted to return information about the student's activity to the LMS.The tool should be able to assess the activity or provide the interfaces needed by the teachers to do that (i.e. a tool to carry out quizzes, a simulator, a serious game, etc.).• Scenario 4. Use of external online tools without evaluation support into the PLE.This scenario aims to gather the students' activity in online tools included in the PLE.These tools are not necessarily educational tools so they are not going to provide an interface to assess the students' outcomes.The teacher defines an instance of the online tool into the LMS; this will create a context that only teachers can access and through which the results of the activity performed by the student can be returned or the evaluation could be facilitated.The student accesses his/her PLE and can use, among others, the online tools adapted to return information about the student's activity to the LMS.The tool in this case is not necessarily created with a learning objective, so they do not include assessment interfaces, something that is needed to grade the student's activity.This assessment interface is provided by the mediator (or proxy tool), which interacts with the online tool, the LMS and facilitates the integration between them.The implementation of these scenarios should be independent of the LMS and/or the underlying technology, which means that solutions such as web services or interoperability specifications are needed.
In Figure 2 a deployment diagram is shown with the components of the framework, the different tools involved in the defined scenarios and the web services and BLTI interfaces.

V. IMPLEMENTATION AND EVALUATION OF SCENARIO 1
In the previous sections several scenarios have been proposed and also a proof of concept has been described.In order to evaluate them, several pilot experiments have been carried out with the students of the Project Management subject at the University of Salamanca and teachers from different areas.
This section presents the specific case of scenario 1.Firstly, the scenario is described in terms of its implementation and functionality.Subsequently, the experiments and its results are presented.

A. Implementation of Interoperability Scenario 1
Scenario 1 aims to open the institutional learning system to extract functionalities to other environments.During the proof of concept, the Forum tool of the LMS (one of the most common functionalities in learning platforms) is exported.It is included in the PLE and can be used with other tools that the user uses to learn, such as Wikipedia, Youtube, Flickr, Slideshare, other expert forums and blogs, etc.
In order to implement this scenario, a W3C widget has been defined.It allows access to the forum in a specific context such as a course, a module, a platform, etc.This widget shows the forum discussions to which the authenticated learner has access.The user can read these discussions and its associated posts, define a new discussion, reply to a specific post and so on.The activity carried out by the user in the forum widget is also shown in the institutional environment, so the teacher does not need to go to other environments to evaluate the learner's activity or to provide feedback.
In order to represent such information exchange between the widget and the LMS the Moodle web service layer is used.To illustrate how this exchange is produced, Business Process Modelling Notation -BPMN [47] has been employed.An example is shown in the Figure 3, a diagram representing how to recover the Forum Discussions List from the widget forum.That diagram shows the exchange of messages between 2 participants the Widget-Forum and Moodle and more specifically how the request is sent and processed.
In the Widget the forum discussion list is shown as in Figure 4.

B. Pilot experience application and results
In this section the experiment carried out with students and teachers is described, to better understand firstly how the methodology is employed and presented and secondly the results achieved.

1) Research methodology
The idea behind this experiment is to validate the scenario by taking into account both students and teachers' perceptions about it, understanding this issue as something that can be addressed in a qualitative way.However, to generalize the conclusions it is also interesting to use quantitative techniques so during the experiment both perspectives are used.This is known as mixed research methods and provide a more complete approach to validation [48].
iJET -Volume 9, Issue 5: "ALICE 2013", April 2014 Forty students studying Project Management and 10 teachers from different academic areas (university, secondary school, and professional training) were involved in the experiment.With the students a quantitative methodology was applied and with the teachers a qualitative one, endorsed with quantitative results.
The methodology used with the students is a quasiexperimental design [50].It is used because in this experiment pre-established groups of students (class-groups) were used, so it was not possible to have a complete randomized group of people [51] and therefore a control study approach wasn't possible with either.
Quasi-experimental design implies the definition of a scientific hypothesis, from which a dependent variable is derived.Such a variable operates through several assertions that are proposed to the students of both the experimental and control group (independent variable).These asserts are graded by the students using a five-value levels scale (1=strongly disagree, 2=disagree, 3=indifferent, 4=agree, 5=strongly agree).In both groups the same tests are applied, a pre-test at the beginning of the experiment and a post-test after it, but the students from the experimental group test the forum widget in the PLE, while the people in the other group do not.After running the experiment, data is analysed by using probabilistic techniques to validate the initial hypothesis.
The scientific hypothesis will be accepted if the results of the pre-test are similar in both groups (which proves that both groups are similar and have a common knowledge and background) and the results of the post-test between the people involved in the experimental group and the control group are different (those who have tested the tool should answer in a different way).This has been checked using Mann Whitney U test, because the data used to consider the information is ordinal and unlikely to be normally distributed.The following null hypothesis is proposed with this statistic test: H 0 : !!= !!(where !refers to the average range, E refers to the experimental group and C refers to the control group).
Teachers' opinion has been evaluated by using semistructured interviews, their opinion was gathered through a form, which allowed a qualitative analysis supported with quantitative data.The qualitative analysis is based on open questions related to teachers' opinion about how the framework can facilitate students' learning and their evaluation from the LMS through the exportation of institutional functionalities and its combination with other tools.The answers of the text have been analysed, units are defined with a thematic criteria; after that the outcomes are synthesized and they are grouped according to the units.In this case the units are exportation, combination, participation and confusion.Later the results are shown in a matrix and conclusions posed from that information [52]. 2

) Data analysis and results
The results can be distributed between students and teachers.During the quantitative study of students' opinion the scientific hypothesis was "The exportation of functionalities from a learning platform and its use in other contexts facilitates learning personalization and therefore helps the student to learn".To test this hypothesis, some assertions have been proposed to the students.
In the pre-test: Moodle Forums and combine them with other tools such as: Youtube, Wikipedia, etc.; helps me to learn.The results of these assertions as commented above have been analysed by the use of the Mann Whitney U test with the null hypothesis H0: !!= !! .The results can be seen in the Table 1, with a significance of a 0.05.
From this data, in the pre-test, it can be observed that the differences among average ranges for assertions in the experimental and control groups are not significant, which means that with regards to these assertions, the experimental and control groups have a similar background.Something usual because the students are from the same context.Combining this information with the average results in the five-value scale, it is possible to say that in the opinion of the students they use Moodle forums (3.30 the average results in experimental and control group), not always because they were mandatory (2.15-2.40)and most of them use other tools to learn (4.55-4.65).Regarding post-tests results, average range difference is significant because it is under 0.05, so the null hypothesis is rejected for those assertions (I4, I5 and I6), which means that students that test the system think in a different way from those who do not use it.Combining such information with the average results of the experimental students, it is possible to conclude, that those who use the framework think that forums can be adapted to their needs (3.70), participation in forums help them to better understand the contents (3.90) and they agree with the possibility to combine it with other tools because they have been able to do this with the framework (4.30).
To support this conclusion, two opinion assertions about the experience were posed to the experimental group students.85% of the students agree or strongly agree that the exportation of functionalities from Moodle and their combination with other tools help them to learn; and 70% of the students consider that the exported functionalities help them to personalize their own learning environment.
In order to take into consideration the teachers' opinion, several semi-structured interviews have been carried out.As previously commented, in this case qualitative and quantitative techniques are used.The results of the qualitative techniques can be seen in Table 2.
From Table 2 it is possible to conclude that most of the teachers see useful exportation functionalities.They also think that the combination of such tools with others that the students use to learn are positive and enrich their learning.In addition, the participation can be increased because they can use the tools in other contexts more adapted to their needs and that have all the tools they use to learn in the same framework (although for some teachers this is indifferent because in their context participation is mandatory).Moreover, several teachers think the exportation of functionalities from the LMS and its combination with other tools can mean a loss of students' attention.
In addition, these conclusions are verified by quantitative techniques and the 100% of the teachers agrees or strongly agrees that exporting LMS functionalities and combining them with other learning tools help learners to learn; and 90% agree or strongly agree that such exportation can increase learner participation in subjects.eLearning is continuously evolving towards greater personalization putting the learning in the driving seat.The Bologna process is committed to learner work outside the institution, personalized learning, and continuous assessment, therefore beginning to attach importance to the learning developed in informal contexts.In this situation, PLE are especially relevant, but traditional LMS must not disappear.Both contexts should be integrated so as to provide a more complete solution.
This article shows some possible interoperability scenarios and proposals to implement them.The approach consists of a service-based framework based on interoperability specification.
One of such scenarios has been presented and evaluated from the perspective of students and teachers.It consists in the exportation of LMS functionalities to the PLE.From the evaluation of such experience, it can be concluded that in the students and teachers opinion the exportation of functionality from the LMS to the PLE facilitates learning processes, allows learners to define their own Personal Learning Environments and increases their participation.In addition, some teachers are worried by the loss of attention that can be linked to the use of not only institutional tools to learn.
In a future project some of the other performed experiments can be described, taking into account all the existing scenarios.Such experiences should be carried out in other not so controlled contexts (secondary schools, workplace, etc.) and they should consider in a quantitative way if students' grades improve due to the use of the framework.It could also be interesting to evaluate the existing gaps of the interoperability specifications to be able to propose new ways to extend them and to include other possibilities that allow an easier measurement, such as activity logs, feedback and new types of grades and so on.In addition, it is necessary to explore how the proxy gateway can be defined in a generic enough way to be applicable to any kind of tool.
With this interoperability framework, it is possible to merge informal and formal learning environments.This is done by allowing the user to develop their learning with the tools they want but taking into account his/her activity from the institutional environments.There is lot of work to do but the defined system is flexible enough to include new tools, interoperability specifications and actors into the learning process.

Figure 1 .Figure 2 .
Figure 1.Integration proposal.This image describes the elements of the proposal, the institutional context, the informal and personalized one and the tools connected to the LMS by using BLTI Figure 3. Forum Discussions List BPMN diagram.This diagram shows the exchange of messages between Moodle and the Widget Forum [49].

Figure 4 .
Figure 4. Forum widget image.It shows two images of the forum widget, with a discussion and the login form to have access to the forum [45] the institutional environment, but the teacher should assess such activity by accessing other contexts that differ from LMS.For example, a student accesses an online tool from the PLE and performs (in agreement with the teacher) a task by using it; then, the teacher should enter into the online tool or the PLE, check her activity and perform her assessment from the LMS.This scenario is quite common in different institutions and it requires a extra effort on the teachers' part[45].•Scenario 3. Use of external tools (with evaluation support) in the PLE, and the return of the outcomes to the LMS.In this scenario the activity is done using the external educational tool but it is integrated in to the LMS.The teacher defines an instance of the educational tool into the LMS: this will create a context only accessible by teachers and through which the results of the task completed by the student in the PLE can be recovered; the student accesses his/her PLE and can use, among others, the educational tool adapted to return information about the student's activity to the LMS.The tool should be able to assess the activity or provide the interfaces needed by the teachers to do that (i.e. a tool to carry out quizzes, a simulator, a serious game, etc.).• Scenario 4. Use of external online tools without eval- • Scenario 2. Use of external tools with external access to them to measure activity.In this scenario no interoperability between the LMS and the PLE is proposed.It takes into consideration the students' activity in the PLE • I1.I usually use Moodle forums in my subjects.• I2.I just use Moodle forums because in the institution participation in them is mandatory.• I3.I use other online tools to learn that are not included into Moodle (Youtube, Wikipedia, etc.).In the post-test: • I4.Moodle Forums are adapted to the way in which I learn and to my necessities, which increase my motivation.• I5.The participation in forums related to my subjects helps me to better understand the contents.• I6.The possibility to participate in my subjects'

TABLE I .
RESULTS OF THE MANN-WHITNEY U TEST. IT SHOWS THE AVERAGE RANGE FOR EXPERIMENTAL AND CONTROL GROUPS, THE RESULT OFCONTRAST STATISTIC AND THE SIGNIFICANCE PER EACH ITEM.

TABLE II
A CASE STUDY FOR MEASURING INFORMAL LEARNING IN PLES VI.CONCLUSIONS iJET -Volume 9, Issue 5: "ALICE 2013", April 2014 PAPER