A Formative Analysis of Resources Used to Learn Software

Abstract: A comprehensive, formal comparison of resources used to learn computer software has yet to be researched. Understanding the relative strengths and weakness of resources would provide useful guidance to teachers and students. The purpose of the current study was to explore the effectiveness of seven key resources: human assistance, the manual, the keyboard, the screen, the software (other than main menu), the software main menu, and software help. Thirty-six adults (18 male, 18 female), representing three computer ability levels (beginner, intermediate, and advanced), volunteered to think out loud while they learned, for a period of 55 minutes, the rudimentary steps (moving the cursor, using a menu, entering data) required to use a spreadsheet software package (Lotus 1-2-3). The main menu, the screen, and the manual were the most effective resources used. Human assistance produced short term gains in learning but was not significantly related to overall task performance. Searching the keyboard was frequently done, but was relatively ineffective for improving learning. Software help was the least effective learning resource. Individual differences in using resources were observed with respect to ability level and gender. Résumé : Aucune comparaison complète et officielle des ressources utilisées dans l’apprentissage des logiciels informatiques n’a encore été effectuée. La compréhension des forces et des faiblesses relatives des ressources pourrait être utile pour les enseignants et les étudiants. La présente étude avait pour objet d’examiner l’efficacité des sept ressources principales : l’aide des pairs, le manuel, le clavier, l’écran, le logiciel (autre que le menu principal), le menu principal du logiciel et l’aide sur le logiciel. Trentesix adultes (dix‐huit hommes et dix-huit femmes) de trois niveaux d’habiletés différents avec les ordinateurs (débutant, intermédiaire et avancé) se sont portés volontaires pour faire part de leurs commentaires tandis qu’ils apprenaient, pendant 55 minutes, les étapes rudimentaires (déplacer le curseur, utiliser un menu, saisir des données) nécessaires à l’utilisation d’un progiciel tableur (Lotus 1-2-3). Le menu principal, l’écran et le manuel se sont avérés être les ressources les plus efficaces. L’aide des pairs n’a permis d’effectuer que des gains à court terme en apprentissage mais elle n’était pas vraiment liée à l’exécution globale des tâches. Le clavier a été examiné régulièrement mais a peu contribué à l’amélioration de l’apprentissage. L’aide sur le logiciel a été la ressource d’apprentissage la moins efficace. On a observé des différences dans l’utilisation des ressources selon le niveau d’habileté et le genre.

approach is not particularly effective (Olfman & Bostrom, 1991;Olfman & Manviwalla, 1995;Shayo & Olfamn, 1993).It has been estimated that more than 50% of the participants in software workshops fail to use the software they were trained on (Olfman & Bostrom, 1991;Olfman & Manviwalla, 1995).Cross (2006) argues that formal training w hile providing a potentially useful start cannot properly address today's rapidly changing working environments.
Self-regulated, exploratory or informal learning is an alternative and common method that many people use to learn new software (Bartholomé, Stahl, Pieschl, & Bromme, 2006;Cross, 2006).In one report assessing over 25,000 computer users, 96% of respondents reported that they taught themselves to learn software through trial and error (Dryburg, 2002).In order to be successful at exploratory learning, though, effective help seeking behaviour or knowing how to use resources is critical (Bartholomé et al., 2006).The process of learning a new software package can become overwhelming very quickly (Draper, 1999).Ideally, a user w ould like to get the required help shortly after a specific problem arises (Modesitt, Maxim, & Akingbehin., 1999).Unfortunately many people are confused by the "modern labyrinth" of resources available.(Greif, 1994;Leutner, 2000;Mangold, 1997).According to cognitive load theory (Chandler & Sweller, 1991;Kester, Lehnen, Van Gerven, & Kirschner, 2006;Sweller, 1988, Sweller, van Merrie¨nboer, & Paas, 1998), a user wants to minimize extraneous cognitive load (engaging in processes that are not beneficial to learning) and optimize germane cognitive load (engaging in process that help to solve the problem at hand).In addition, cognitive load w ill vary according to the experience of the user.In other w ords, the extraneous cognitive load might increase more rapidly for a novice than an advanced user because the advanced user has a more developed schema to understand and adapt to new situations.It is important, then, to examine how users negotiate resources in order to maximize learning.
To date, there is a relatively small volume of research on how ICT skills are developed ( Taylor, 2003).More attention is spent on selecting content (software, version and platform) than on how to teach technology skills (Lambrecht, 1999;McEwen, 1996).Many studies focus on analysing the effectiveness of a single resource or approach (e.g., Bannert, 2000;Bartholomé et al., 2006;Belanger & Van Slyke, 2000;Carroll, 1990;Guzdial, 1999;Rieman, 1996).The purpose of this paper to examine and compare the effectiveness of a wide range of resources used to learn a new software package.

Literature Review
The paradox inherent in self-guided or exploratory learning is that to learn, one must be able to interact w ith the software, however, to interact with the software, one must already have some knowledge about how to use it (Lambrecht, 1999).New users typically overcome this paradox by consulting a variety of resources: human assistance, a manual, the software itself (e.g., searching the screen, the menu, or software help), and the keyboard (Rieman, 1996;Reimann & Neubert, 2000).The little evidence that has been gathered comparing the use of resources suggests that people prefer to "try things out" (Carroll, 1990(Carroll, , 1998;;Dryburg, 2002;Rieman, 1996), read the manuals (Rieman, 1996), ask for some form of human assistance (Dryburg, 2002;Rieman, 1996), and, in some cases, consult the software help system if they are particularly aggressive explorers (Rieman, 1996).Reimann & Neubert (2000) add that users tend to consult a hybrid of resources w hile learning instead of relying on a single support tool.

Using Software as a Resource
Even though a majority of users prefer a trial-and-error approach to learning software, limited research has been done looking at how various features on the screen including the software menus are used to advance understanding.Several researchers have examined a "training w heels" approach to learning softw are w here cognitive load is reduced by limiting the number of functions available (Bannert, 2000;Guzdial, 1999;Leutner, 2000).Users who learn with a reduced command set outperform those individuals presented with a full array of options (Leutner, 2000).Other research suggests that the successful use of an interface depends on whether the new softw are is consistent with previous software learned and how easy it is to guess at commands (Guzdial, 1999).Finally, with respect to using a software menu, users employ a label-following strategy-they select labels or words in the menu that are similar to words and concepts in the tasks they are trying to complete (Polson & Lewis 1990).

Using Manuals
Manuals can provide extensive information in the form of task oriented instructions, indexes, table of contents, pictorial representations, and specialized short-cut guides (Rettig, 1991).In spite of these seemingly w ell organized aids, most new users, regardless of ability level, begin using new softw are w ithout reading the manual (Carroll, 1990;Rettig, 1991;Simmons & Wild, 1991;Taylor, 2003).Rettig (1991) refers to computer manuals as the best sellers that no one reads, however Dryburg's (2002) extensive report on over 25,000 users noted that manuals are used 60% of the time at some point in the software learning process.
There is some evidence to suggest that using a manual actually improves learning performance.For example, Rieman (1996) noted that individuals can find out how to do tasks without manuals, but more advanced features remain untouched or unresolved.Bannert (2000) observed that acquiring new software skills with a manual w as significantly faster and more productive than tutor guided instruction.How ever, not all manuals are the same.Manuals that contain ample error information (minimal manuals) help students perform better than manuals with limited error information (Carroll, 1990;Lazonder 1994, Lazonder & Van Der Meij, 1995;Van der Meij & Carroll, 1995).Van der Meij (2000) adds that the most successful format of a manual is a two column layout w ith instructions and full screen images presented side-by-side.

Human Assistance
W hile many users w ill play w ith softw are or use a manual, Simmons and W ild (1991) noted that the majority of individuals end up asking for help from a knowledgeable person.Rieman (1996) also observed that asking another person for help is a natural strategy but there can be several barriers-availability, feeling like you are bothering an experienced user too often, time finding someone, and being too proud to ask for help.Emailing a person for help is rarely done because most people need a quick answ er to their problems (Rieman, 1996).However, instant messaging might prove to be an attractive alternative to email given that response time would no longer be an issue.Several researchers have reported that users would rather ask for help on an "as needed basis" rather than be controlled by a tutor or trainer (Bannert, 2000;Simmons & Wild, 1991).

Input Devices
The effect of hardw are on learning has not been examined in much detail.The majority of research on manmachine interaction has not focused directly on hardware issues (Baecker & Buxton, 1987;Baecker, Grudin, Buxton, & Greenburg, 1995;Carroll, 1991;Norman & Draper, 1986).Buxton (1986), though, has looked at the role of input devices on user behaviour and has noted that the choice of an input device (e.g., keyboard, mouse) can have a marked effect on the user's model of how a softw are package w orks.

Individual Differences
Ability .The effective use of resources w hile learning new softw are is partially dependent on a user's perception of specific challenges that arise (Lazonder & Van Der Meij, 1995).It is reasonable to anticipate that more able users would have a broader perspective on potential barriers to learning software, and therefore would be more efficient at selecting appropriate resources (Bannert, 2000;Lazonder & Van Der Meij, 1995).
As learners grow and develop understanding and expertise, their need for software support and functionality w ill change as w ell (Jackson, Krajcik, & Solow ay, 1998).More experienced users read manuals in greater depth (Rieman, 1999) and are more proficient in selecting and executing search strategies (Bartholomé et al., 2006;Lazonder, 2000;Wood & Wood, 1999).However, domain specific software expertise appears to be more important than general expertise (Draper, 1999).For example, specific knowledge of spreadsheet software would help an individual learn a new spreadsheet software package more than overall software expertise (Draper, 1999).In addition, the differences between novice and experts begin to disappear when tasks become more complex (Lazonder, 2000).
Gender .Gender differences in computer attitudes, use, ability, and behaviour (e.g., Kay, 1992;Kay, in press;Sanders, in press;Whitley, 1997) have consistently reported differences in favour of males.It is reasonable to speculate, then, that differences may occur with respect of the use of resources.To date, little research has been done on gender differences and the use of resources to learn new software.In one large scale study (Dryburg, 2002), men were more likely to use exploratory learning whereas women preferred facilitated methods (e.g., on the job training, help from friends, family, cow orkers).
The purpose of this paper was to examine and compare the effectiveness of seven resources used to learn a new software package: human assistance (the experimenter), the manual, the software itself (other than the menu), the software main menu, the software help system, the screen, and the keyboard.The specific research questions were as follows: 1. Is there a significant difference among resources with respect to frequency of use? 2. Is there a significant difference among resources with respect their impact on learning?3. Are there significant differences among ability levels and gender with respect to resources used?

Sample
The sample consisted of 36 adults (18 male, 18 female): 12 beginners, 12 intermediates, and 12 advanced users, ranging in age from 23 to 49 (M= 33.0 years), living in the greater metropolitan Toronto area.Subjects w ere selected on the basis of convenience.Equal numbers of males and females participated in each ability group.Sixteen of the subjects had obtained their Bachelor's degree, eighteen their Master's degree, one a Doctoral degree, and one, a community college diploma.Sixty-four percent (n= 23) of the sample were professionals; the remaining 36% were students (n=13).All subjects had one or more years experience using computer.Seventy-five percent (n=27) of the subjects had there own computers;17% (n=6) intended to buy a computer in the future.All subjects voluntarily participated in the study.

Procedure
Overv iew .Each subject was given an ethical review form, computerized survey, and interview before attempting the main task of learning the spreadsheet software package.Note that the survey and interview data w ere used to determine computer ability level (see data source below w ith respect to how computer ability levels w ere assessed).Once instructed on how to proceed, the subject w as asked to think-aloud w hile learning the spreadsheet software for a period of 55 minutes.All activities were videotaped with the camera focused on the screen.Follow ing the main task, a post-task interview w as conducted.
Software selection .The spreadsheet software selected for this study was Lotus 1-2-3 (version 5.0).The software was deliberately chosen because it was unfamiliar to all subjects.The more advanced users had a w ide range of softw are they w ere familiar w ith, so it w as necessary to select a more obscure softw are package to establish true learning situation.It is realized that this choice of softw are might affect the generalizability of the results, how ever, the range of skills attempted during the 55 minute learning session, in general, was quite limited: moving around the screen, using the menu, and entering data.These skills are pretty common to most spreadsheet software packages, regardless of the version, and are typically done in the same way.
Learning tasks.The selection and presentation of learning tasks was carefully designed to be as authentic as possible.To do this, spreadsheet software was chosen because most participants had minimal experience using this softw are.Spreadsheet software is used to create, manipulate, and present rows and columns of data.The mean pre-task score for spreadsheet skills was 13.1 (SD = 15.3)out of a total possible score of 44.Ten of the subjects (6 advanced users, 4 intermediates) reported scores of 30 or more.None of the subjects had ever used the specific spreadsheet software package used in this study (Lotus 1-2-3).
Subjects attempted a maximum of five spreadsheet activities arranged in ascending level of difficulty including (1) moving around the spreadsheet (screen), (2) using the command menu, (3) entering data, (4) deleting, copying, and moving data, and (5) editing.They were first asked to learn "in general" how to do activity one, namely moving around the spreadsheet.When they were confident that they had learned this activity, they were then asked to complete a series of specific tasks.This semi-structured exploratory approach to learning software is supported by a number of researchers (Bannert, 2000;Kester et al., 2006;Leutner, 2000;Wiedenbeck, Zavala, & Nawyn, 2000;Wiedenbeck & Zila, 1997).All general and specific activities were done in the order presented in Appendix A.
From an initial pilot study of 10 subjects, it w as determined that 50-60 minutes w as a reasonable amount of time for subjects w ith a w ide range of abilities to demonstrate their ability to learn the spreadsheet softw are package.Shorter time periods limited the range of activities that beginners and intermediate subjects could complete.
In the 55 minute time period allotted to learn the software in the current study, a majority of the subjects completed all learning tasks w ith respect to moving around the screen (100%) and using the command menu (78%).About two thirds of the subjects attempted to enter data (69%), although only one third finished (33%) all the activities in this area.Less than 15% of all subjects completed the final tasks: deleting, copying, moving, and editing data.

Data Collection
Think-aloud protocols (TAPs) .The main focus of this study was to examine the use of resources with respect to learning computer software.The use of think-aloud protocols (TAPs), where subjects verbalize w hat comes to their mind as they are doing a task, is one promising technique for examining transfer.
Essentially, the think-aloud procedure offers a w indow into the internal talk of a subject w hile he/she is learning.In a detailed critique of TAPs, Ericsson and Simon (1980) conclude that "verbal reports, elicited w ith care and interpreted with full understanding of the circumstances under which they were obtained, are a valuable and thoroughly reliable source of information about cognitive processes" (p.247).

Learning behav iours .
The analyses used in this study are based on think-aloud data.Specifically, 3,169 learning behaviours involving the use of seven resources were identified and rated according to the degree to which they influenced learning.A learning behaviour was defined as "an action that influenced learning or the completion of an assigned task".Examples of learning behaviours might include pressing a key on the keyboard, searching the keyboard, reading the manual or screen, or searching through a menu Presentation of TAPs .The following steps were carried out in the think-aloud procedure to ensure high quality data: 11/10/2010 A Formative Analysis of Resources Use… cjlt.ca/index.php/cjlt/article/…/184/18 Step 1. (Instructions) Subjects were asked to say everything they were thinking while working on the software.Subjects were told not to plan what they were going to say; Step 2. (Examples) Examples of thinking aloud were given, but no practice sessions were done; Step 3. (Prompt) Subjects were told it was important that they keep talking and that if they were silent for more than 5 sec onds, they would be reminded to "Keep talking"; Step 4. ( Reading).Subjects were permitted to read silently, but they had to indicate what they were reading and summarize when they had finished; Step 5. (Giving help).If a subjec t was really stuck, he/she could ask for help.A minor, medium, or major form of help would be given, depending on how "stuc k" a subject was; Step 6. (Recording of TAPs).Both thinking aloud and the computer screen were recorded using an 8mm video c amera.

Data Sources
Resources .There were seven resources examined: human assistance (the experimenter), the manual (an independent, best selling book), the spreadsheet software other than the menu, the software main menu, the software help system, the screen, and the keyboard.These resources were selected based on their prevalence in the literature review.Note that use of the software was broken down into three separate categories (using softw are, reading screen, using the menu) to provide more detailed information.
Operational definitions of each of the resources are provided in Table 1.A total of 3,169 learning behaviours were categorized according what resource was used.
It is important to note that human assistance, one of the resources a subject could use, was only given if a subject felt he/she w as stuck.This instruction w as deliberately included in an attempt to mimic a fairly typically software learning experience.It is assumed that most people do not have a personal or readily available expert w ho is w illing to answ er numerous questions about a new softw are package being learned.
In addition, some research suggests (e.g., Reiman, 1996), there that the typical user is reluctant to ask for help at first.

Table 1. Operational Definitions of Resources Used
Com puter ability .Three computer ability levels w ere compared in this study: beginners, intermediates, and advanced users.The criteria used to determined these levels included years of experience, previous collaboration, previous learning, software experience, number of application software packages used, number of programming languages/operating systems known, and application software and programming languages know n (78 items, reliability estimates ranged from .79 to .97).A multivariate analysis show ed that beginners had significantly low er scores that intermediate and advanced users (p<.005), and intermediates users had significantly lower scores than advanced users on all eight measures (p<.005).
Learning .After each of the 3,169 learning behaviours were categorized according to resource type, they were scored on their immediate influence on learning (a score from -3 to +3 -see Table 2 for rating criteria).
Five variables were used to evaluate the relative effectiveness of resources: how often a resource was used (frequency), mean influence the resource had on learning (Table 2), percentage of subjects who used the resource, total resource effect score, and performance score.Conceptually, the first three variables assessed prevalence (how often the behaviour was observed and by how many subjects) and intensity (mean influence of resource).The fourth variable, total resource effect score, is a composite of the first three variables and w as calculated by multiplying the frequency in w hich a resource w as used by the mean influence score of the resource by the percentage of subjects who used the resource.For example, the software main menu was used 1080 times, had a mean influence of 0.73, and was used by 83% of the subjects.The total resource effect score, then, was 657.0 (1080 x .73x .83).
Performance scores were calculated by adding up the number of subgoal scores that each subject attained during the 55 minute time period.For each task, a set of learning subgoals was rated according to difficulty and usefulness.For example, the task of "moving around the screen" had five possible subgoals that could be attained by a subject: using the cursor key (1 point), using the page keys (1 point), using the tab keys (1 point), using the GOTO key (2 points), and using the End-Home keys (2 points).If a subject met each of these subgoals successfully, a score of 7 would be given.If a subject missed the last subgoal (using the GOTO key) a score of 5 would be assigned.A sample of scoring for the first main goal, moving around the screen, is presented in Appendix B.
Note that nine performance scores were calculated for each subject including five main spreadsheet categories (moving around the screen, using the menu, entering data, editing data, and modifying data) and four resources (understanding how to use software help, the screen, the manual, and the keyboard).
Reliability of TAPs .Reliability and validity assessments were derived from the feedback given during thestudy and a post-task interview .One principle concern w as w hether the TAPs influenced learning.W hile, several subjects reported that the think-aloud procedure was "weird", "frustrating" or "difficult to do", the vast majority found the process relatively unobtrusive.Almost 70% of the subjects (n=25) felt that thinking aloud had little or no effect on their learning.

Table 2.Criteria for Scoring Influence on Learning
The accurate rating of the influence of a resource on learning (Table 2) is critical to the reliability and validity of this study.Because of the importance of the learning influence scores, six outside raters were used to assess a 10%, stratified, random sample of the total 3,169 occasions when resources were used.Inter-rater agreement was calculated using Cohen's Kappa (Cohen, 1960), a conservative and robust measure (Bakeman, 2000;Dewey, 1983).The Kappa coefficients for inter rater agreement between the experimenter and six external raters (within one point) were as follows: Rater 1: .80,Rater 2: .82,Rater 3: .95,Rater 4: .94,Rater 5: .93,Rater 6: .93.Coefficients of .90 or greater are nearly always acceptable and .80 or greater are acceptable in most situations, particularly for the more conservative Cohen's Kappa ( Lombard, Snyder-Duch, & Bracken, 2004).

Frequency of Resources Used
The frequencies of resources used by subjects in this study are presented in Table 3. Subjects used the main menu most often or about one third of the time.The manual and the keyboard, the next two most frequently used resources, were consulted about 20% of the time each.Software help, the screen, and using the spreadsheet software, were used less than 10% of the time.Finally, human assistance was asked for least often although this was not unexpected since subjects were instructed to consult this resource as a last resort.Over the 55 minute period, each subject, on average, used the software main menu 30 times, the keyboard 20 times, the manual 16 times, software help and the screen 7 times each, the software (trial and error) 4 times, and human assistance 3 times.

Mean Influence Score
Recall that mean influence score was defined as the average immediate effect that a particular resource had on learning (see Table 2 for the rubric).A one-way ANOVA, comparing mean influence scores for the seven resources was significant (F = 14.00, p < .001).Human assistance had the highest mean influence on learning (M = 1.06,S.D. = 0.86) and w as significantly more effective than using the manual (M = 0.38, S.D. = 1.04), softw are help (M = 0.37, S.D. = 1.15), or the keyboard (M = 0.57, S.D. = 1.09) (Scheffé post hoc analysis, p < .05).Reading the screen was the second most influential resource on learning (M = 0.83, S.D. = 0.74) and w as significantly more helpful than the manual or softw are help (Scheffé post hoc analysis, p < .05).Using the softw are menu (M =0.73, S.D. = 1.14) and other softw are features (M =0.73, S.D. = 1.14) were the next two highest mean influences on learning.Using the softw are menu w as significantly more effective than using the manual or software help (Scheffé post hoc analysis, p < .05)and using other software features was significantly more effective than using the manual (Scheffé post hoc analysis, p < .05).Clearly the manual and the softw are help w ere the least effective resources in terms of influencing learning.
It is interesting to note that those resources that had the most immediate impact on learning w ere not necessarily used more often.Correlations between mean influence score and frequency for all resources were non significant.

Percentage of Subjects Who Used Resources
While there was some variation with respect to the percentage of subjects who used resources, all resources w ere used frequently.The least often used resource w as human assistance, but subjects still used it over 75% of the time.Most subjects incorporated multiple resources.Forty-seven percent of all subjects (n=17) used all seven resources, 94% of all subjects (n=34) used 6 out of seven resources, and 100% of the subjects used at least five resources.Clearly subjects consulted on a full range of resources when trying to solve software problems.

Total Resource Effect Score
The total resource effect score w as determined by multiplying the frequency w ith w hich a resource w as used by the percentage of subjects who used the resource by the mean influence score (see Table 4).The software menu produced the highest total resource effect score, based on frequent use and a moderate mean influence on learning score.However fewer subjects used this resource.The keyboard had the next highest total resource effect score, even though its mean influence on learning w as relatively low .The high score was reported because all subjects used this resource frequently.Reading the screen produced a high total resource effect score, largely because of the high mean influence on learning score.Somewhat paradoxically, this resource w as used relatively infrequently.Even though the manual w as one of the least effective resources in terms of influencing learning, it ranked fourth in terms of total resource effect score.The higher ranking was primarily as result of frequent use.Trial and error use of the software produced a relative low total resource effect score, not because it was ineffective in terms of influencing learning, but because it was used infrequently.Human assistance was not used often, resulting in the second lowest total resource effect score.However, as stated previously, it had the highest mean influence on learning when it was used.Finally, softw are help w as used infrequently by relatively few subjects w ith minimal gains in learning.This combination produced a total effect score that was more than seven times lower than the highest score reported.Total resources effect scores for all resources are presented in Table 4.

Performance Scores
A series of correlations was run for mean influence scores and performance scores (Table 5).The mean influence of using the menu was significantly and positively correlated with all task performance scores.In other words, subjects performed significantly better if they used the menu effectively.Using the manual effectively was significantly and positively correlated with the first two task performance scores (moving around the screen and using the menu) and negatively correlated with the software help performance score.
Reading of the screen effectively was significantly and positively correlated with the performance scores for understanding the screen and using the menu.Using software help was significantly and positively correlated with the software help performance score, but no spreadsheet task performance scores.In other words, subjects w ho used softw are help, learned about help features, but this know ledge did not translate into improving performance on assigned tasks.Using the software (trial and error) was significantly correlated with only one performance score-software help.Finally the mean influence score for human assistance and searching the keyboard were not significantly correlated with any of the performance scores.

Individual Differences (Ability & Gender)
Frequency of Resource Use .A Pearson Chi-Square test show ed a significant association among ability and frequency of resources used (χ 2 (12) -74.31, p < .001).From Table 6, it appears that beginners search the keyboard and use software help more often than intermediate or advanced users.Intermediate and advanced users, on the other hand, seem to use the software (both main menu and trial and error of other features) more than beginners.The manual, human assistance, and the screen w ere referred to by all ability groups equally.There was a significant association between males and females with respect to frequency of resources used (χ2 (6) -38.15, p < .001).Males preferred to use the manual and read the screen, whereas females liked to ask for human assistance and use the software help (see Table 7).

Table 7.Frequency of Resource Use by Gender
Mean Influence of Resources .A 3-way ANOVA revealed three new significant effects (Table 8).First, advanced users (M= .74,S.D. =1.10) w ere significantly better at using resources than intermediate (M= .60,S.D. =1.10) or beginner users (M= .51,S.D. =1.08) (Scheffé post hoc analysis; p < .05).Second, males (M= .71,S.D. = 106) were significantly better at using resources, as a whole, than females (M= .53,S.D. =1.13) (p <.005).Third, there was an interaction effect for ability level and use of resources (p <.005).From Figure 1, it appears that beginners are less able to use the manual and the spreadsheet software (trial and error) than their more experience counterparts.Furthermore, advanced users seem to be better at using software help than intermediate or beginner users.

Discussion
The purpose of this study was to compare resources used to learn a new software package.Three research questions were asked: 1. Is there a significant difference among resources with respect to frequency of use? 2. Is there a significant difference among resources with respect their impact on learning?3. Are there significant differences among ability levels and gender with respect to resources used?

Frequency of Use
The frequent use of the softw are menu and searching the keyboard is consistent w ith a preference for a "trial and error" strategy reported in previous studies (Carroll, 1990(Carroll, , 1998;;Dryburg, 2002;Rieman, 1996;Riemann & Neubert, 2000).However, minimal use of the software (other than the main menu) and the screen indicates that subjects in this study did not rely solely on an exploratory approach.In fact, they used a hybrid of resources, a pattern that was reported by Rieman and Neubert.The manual was the third most frequent resource employed.The relatively high use the manual in this study w as predicted by Dryburg's report on over 25,000 users, but is inconsistent with a number of other studies (Carroll, 1990;Rettig, 1991;Simmons & Wild, 1991;Taylor, 2003).
One possible explanation for why a trial and error strategy was used less and the manual was used more is that a majority of the individuals, had never used spreadsheet software.Guzdial (1990) noted that successful use of new software partially depends on previous software used.Since spreadsheet software was completely unfamiliar to most subjects in this study, successful use of a trial-and-error approach may have been compromised and more conservative resources, such as a manual, may have been needed.In other words, subjects may not have had the skill to begin exploring on their own because they had no idea where to begin, therefore they had to consult a manual.
Previous research suggests that most users tend to avoid software help (e.g., Aleven & Keodinger, 2000;Bartholomé et al., 2006) and the results in this study confirmed this claim.Softw are help w as used relatively infrequently and by fewer subjects.One reason might be that software help was not particularly helpful with respect to immediate learning or overall spreadsheet task performance.

Impact on Learning
Human assistance, reading the screen, the software main menu, and using other software features (trial and error) had the highest immediate impact on learning performance.This result supports previous studies suggesting that learners rely heavily on human assistance (Simmons & Wild, 1991) and simply "trying the software out" (Carroll, 1990(Carroll, , 1998;;Dryburg, 2002;Rieman, 1996;Riemann & Neubert, 2000).However, correlations between the immediate influence that these resources had on learning and overall spreadsheet task performance revealed a more complicated pattern.Effective use of the software main menu was positively and significantly correlated with overall performance on all spreadsheet tasks.Accurate reading and searching of the screen was significantly and positively correlated with only one final task performance score, namely using the main menu.Effective use of other software features and human assistance were not correlated with any of the final performance tasks.The above result indicates that while human assistance and trying various softw are features may be helpful in the short term, they did not help improve final performance outcome scores.
It could be speculated that successful use of software menus are based on understanding the terms and language behind spreadsheets.Subjects who are good at using the menu may have a better conceptual understanding of software.Use of non menu related features (e.g., column and row labels, status bar, insert key indicator) of the software, on the other hand, may be more random, requiring less organization and concept formation.Therefore, proficiency in non-menus aspects of software use may not be related to increases in overall performance.Finally, human assistance in this study was deliberately designed to be as unobtrusive as possible, therefore vague hints were given in response to most requests for help.It is not surprising, then, that this form of help did not improve final spreadsheet task performance.
Even though subjects in this study relied on searching the keyboard and the manual for help, these two resources w ere not particularly helpful in terms of immediate learning performance.How ever, effective use of the manual was significantly correlated to final task performance with respect to moving around the screen and using the menu.Rieman (1996) and Bannert (2000) noted that manuals do actually improve learning, despite the fact that they are rarely the resource of choice.While the book used in this study was not a minimal manual with ample error information (e.g., Carroll, 1990;Lazoner & Van Der Meij, 1995), it was a best selling, third party, spreadsheet resource w ith extensive indexing.Clearly, it w as of some benefit for certain tasks.
Software help was the least effective resource with respect to immediate influence on learning and overall spreadsheet task performance.While previous research indicates that properly designed software help can foster learning (e.g., Bartholomé et al., 2006;Wood & Wood, 1999), the design for the spreadsheet software help used in this study did not appear to be particularly promising.Users became better at using the software help, but not more proficient at using the actual software.It is possible that the software help was too complicated, requiring too much time and cognitive effort, thereby compromising the final goal of getting tasks done.According to cognitive load theorists (Chandler & Sweller, 1991;Kester et al., 2006;Sweller, 1988, Sweller et al., 1998) software help appeared to be maximizing extraneous cognitive load (engaging in processes that are not beneficial to learning) and minimizing germane cognitive load (engaging in process that help to solve the problem at hand).Finally it is w orth noting that, users might be more adept and successful at using current and better designed software help features than the ones available in the software package examined at the time this study was done.

Individual Differences
Advanced and intermediate users in this study were more comfortable and more effective at using exploratory resources to learn (e.g., software menu, trial and error use of other software features).This is a well documented finding in the literature (Kamouri et al., 1986;Kluwe et al., 1990;Poloson & Lewis, 1990;Reimann & Neubert, 2000).Advanced users were also better at using the manual and softw are help than their less experienced counterparts, but not necessarily more resourceful.All subjects, regardless of ability level, used all resources, but beginners, and, to a lesser extent, intermediate users, w ere less able to benefit from these resources.
A typical beginner may not have enough knowledge to use the trial-and-error approach, so he/she relies on resources like software help, the keyboard, and to a lesser extent the manual.However, software help and manuals are arguably as complicated as the spreadsheet software itself, so beginners struggled to advance their knowledge.For beginners too much time is being spent on extraneous cognitive load (Chandler & Sweller, 1991;Kester et al., 2006;Sweller, 1988, Sweller et al., 1998) and not enough attention is being directed to learning the software.Human assistance and scaffolds in the form of minimal manuals (e.g., Carroll, 1990;Lazoner & Van Der Meij, 1995) and contextual help (e.g., Bartholomé et al., 2006;Patrick & McGurgan, 1993) may be critical in reducing cognitive load enough to help less able users make progress.
It is interesting to note that males w ere significantly better than females at using resources, as a w hole, yet they were not significantly more experienced.This difference may be partially explained by significant differences in the kind of resources that males and females liked to use.Females appeared to prefer human assistance and softw are help, both of which were minimally helpful in learning performance.Males, on the other hand, choose to use the manual and the screen for help, and these resources were significantly correlated w ith learning performance.Females, because of their preference for facilitated learning (also observed by Dryburg, 2002), may have been at a significant disadvantage in this study because the reduced effectiveness of human assistance was built into the design.Subjects were asked to do as much on their own before asking for help.One might speculate that in situations w here facilitated help is not available, females may struggle more w ith learning new softw are packages.

Summary
This study explored seven resources used to learning a new spreadsheet software package: software main menu, the screen, the manual, human assistance, the keyboard, the screen, the software (other than main menu), and software help.The software main menu was used most often and had a significant positive impact on all learning tasks.Observing or searching the screen was used infrequently, preferred by males, and had a significant, positive effect on performance of some spreadsheet learning tasks.The manual was used in moderation, also preferred by males, and had a significant, positive effect on performance of some spreadsheet learning tasks.Human assistance was used infrequently, preferred by females, and did not improve overall learning performance, however these finding should be taken with a proverbial grain of salt, since all users w ere instructed to "ask" for help only after they had tried all other resources.Searching the keyboard w as used frequently, especially by beginners, but w as not significantly correlated w ith learning performance.Trial and error use of software features other than the menu was used infrequently, but effectively by advanced users, although this strategy was not significantly related to task performance.Finally softw are help w as used infrequently, although it w as preferred by females, and had no significant effect on learning performance.Table 9 provides a comparison of resources examined in this study.

Suggestions for Educators
Based on the results of this study, several suggestions can be offered to educators of computer studies: Encourage students to look at the screen.Slowing down and observing closely might give important cues, especially to beginners w ho do not have the search skills of more advanced users; Recognize that the manual and software help can be difficult to use.Provide coaching and support in the use of these resources, or they could hinder learning; Provide students with a rich source of terms and concepts that will be used with the software being learned and make sure that the vocabulary matches that of the manual, softw are help, and the main men; It might be wise to provide guidance with respect to using software menus.Subjects in this study made considerable use of the main menu as a resource tool; No one strategy w ill w ork for all students.Beginners w ill need more help w ith using resources (handouts, minimal manuals, direct coaching), while advanced users will probably be able use trial and error strategies effectively on their own.

Opportunities for Future Research
This study was designed to examine the use of resources used to learn computer software in a semistructured, natural setting.A number of compromises had to be made with respect to capturing and analyzing data, and these decisions have to be considered when interpreting the results.
First, a specific software package had to be chosen-spreadsheets.It was chosen because the type of softw are w as unfamiliar to most subjects and the specific softw are package w as unfamiliar to all subjects.In addition, the software selected was dated compared to more advanced and popular spreadsheet packages.While using resources to learn this spreadsheet software may generalize to other software, that claim cannot be made from the results in this study.
Second, a semi-structured approach to learning where all subjects learned the same tasks in the same order may not be representative of the preferred learning scenario for some subjects.For example, if a subject preferred more a more formal learning approach with a textbook or software help, then a semi-structured approach might be a reasonable facsimile of his/her typical learning approach.On the other hand, if a subject preferred to use trial-and-error and a more non-directed learning style, the semi-structured format might be restrictive.
Third, subjects w ere allow ed to choose any of the seven resources they w anted in any order (w ith the exception of human help).This approach was used to capture "natural" selection of resources; however, interpretation of the results is messy.For example, not all resources are equal.Human assistance, the manual, and softw are help potentially provide a much larger range of solutions than searching the keyboard or trying out the softw are.Therefore, the comparison of resources in this study must be considered formative, a first step into understanding potential use and impact.
Fourth, although over 3100 learning activities were analyzed, the sample consisted of only 36 subjects, who were highly educated, and in their thirties.The number of subjects had to be restricted due to the time required to transcribe, code and analyze the data-in this study the process took over a year.Nonetheless, the results might be quite different for other populations.
Fifth, the resources selected for this study were based on an extensive review of the literature.However, more current resources such as instant messaging, online tutorials, video clips, and web based instruction should be added to the resource list of future studies.
Finally, and perhaps most importantly, the study focused on short-term gains in learning.The results and conclusions do not necessarily apply to long term gains.This is an empirical question to be studied in the

Figure 1 .
Figure 1.Interaction Effect for Mean Influence on Learning Score as a Function of Ability and Resource

Table 4
.Total Resource Effect Score s 11/10/2010 A Formative Analysis of Resources Use… cjlt.ca/index.php/cjlt/article/…/187/18 * Calculated by multiplying Frequency by % of Subjects who use resource by Mean Influence on Learning

Table 8 .
Three -way ANOVA for Mean Influence on Learning Score as a Function of Ability, Gender and Resource Category 11/10/2010 A Formative Analysis of Resources Use… cjlt.ca/index.php/cjlt/article/…/189/18