Does web-based role-play establish a high-quality learning environment? Design versus evaluation

Ludewig, A.; Ludewig-Rohwer, I.

Issues in Educational Research, 2013, Vol 23(2), 164-179
[ Contents Vol 23 ] [ IIER Home ]

Does web-based role-play establish a high quality learning environment? Design versus evaluation

Alexandra Ludewig and Iris Ludewig-Rohwer
The University of Western Australia

Online role-plays have been celebrated for providing an environment which allows for high quality learning. Innovative approaches have been embraced in foreign language studies, especially in countries where a great distance to the target country needs to be overcome, not only to expose students to the target language but also to provide them with a forum to apply and extend their newly-acquired language skills in creative applications outside the face-to-face classroom. This paper explores the design, application and evaluation of one of these innovative teaching and assessment strategies: an online role-play in German Studies at UWA. Complex educational objectives, as classified by Bloom and many others since, were the starting point for our design. However, despite all the ideal ingredients being included in the role-play, our evaluation transpired to be a corrective of sorts. It was intended to re-affirm that students appreciate the best-practice learning strategies which have guided the design and development of this role-play. Student feedback (gained through student surveys, vocabulary tests, classroom observations, as well as quantitative tracking of contributions) has been utilised to analyse the level of student engagement and their reflections on their learning in this role-play. Our findings suggest that even an ideally designed web-based role-play will not necessarily lead to a more effective way of learning, at least not from the students' perspective.

Aims and background

German Studies at The University of Western Australia (UWA) has a long tradition of incorporating innovative approaches into teaching in order to overcome a lack of exposure to the target language. Although several thriving exchange programs are on offer, only a small number of students are able to take advantage of the opportunity to immerse themselves in the language and culture of the German speaking countries. Apart from the budgetary and time constraints involved for students, many language instructors do not believe that visiting a particular country necessarily leads to a faster, better or more sustained uptake of the language spoken there (Freed, 1995; Synalski, 2009); something most tourists can vouch for.

It is challenging to optimise learning conditions for students of foreign languages in a university learning environment that is situated many thousands of miles away from the real-life target culture. This has inspired a team of language instructors and researchers at UWA to spend considerable time and effort on designing a variety of engaging and viable learning activities and environments that allow for interactive exploration and practice of newly acquired knowledge and skills within and outside the classroom. One solution has taken the form of online simulations and blended role-play scenarios, which have been utilised in various forms over the past decade.

The team has been curious to discover whether their efforts to extend the time students can spend on tasks, and thereby increase the opportunities for them to communicate in the target language, have been taken up and appreciated. We were also keen to ascertain whether these innovations have led to the desired student learning outcomes. The following case study of the latest cycle of such an online simulation offered to 47 students at intermediate and advanced levels will serve to illustrate a certain disjuncture between theory and practice, that is: between Bloom's Taxonomy (Bloom, Engelhart, Furst, Hill & Krathwohl, 1956) and a process that has been found in part to be "blooming taxing".

Planning for diversity

At UWA, the German units at intermediate and advanced levels are structured in such a way that students with greatly varying language backgrounds are taught together. Students have three compulsory face to face contact hours per week: one hour in a large-group interactive lecture and two hours in small-group language classes. In addition, one face to face conversation class per week is offered as an optional extra (which is attended by around 80% of students). In order to extend the time spent on task, assessment exercises have been redesigned so that they are potentially open-ended, student self-directed and of such complexity that they offer students seemingly infinite possibilities in which to immerse themselves for additional hours.

Given that any student cohort is diverse, such an offer is bound to attract varying responses. Time constraints experienced by some students, irrespective of whether they are full time or part time, mature age or school leavers, will affect any individual's motivation for studying. However, personal attributes such as confidence or hesitance, extroversion or reserve will also have an impact on learning preferences. In order to provide all students with as much freedom and flexibility as they need for the extra hours of commitment per week, online provision has become the learning extension of choice. To this end, we have designed a learning tool to be accessed in an environment which is engaging, allows for active learning, encourages learning to achieve academically challenging outcomes, provides room to safely explore and practise language and experience culture at all unit levels, while also offering flexibility, choice and a satisfying experience for students.

Literature review

Guided by the literature on the subject, the team has remained mindful of all the hallmarks of "ideal" teaching approaches when devising learning and assessment tasks, namely those which encourage students to engage in self-directed and peer-assisted learning, involve experiential and real-world learning, incorporate resource-based and problem-based learning, include reflective practice and critical self-awareness, utilise open learning and alternative delivery mechanisms and also allow for freedom of choice and individual learning style preferences (cf. Candy, 1991; Candy, Crebert & O'Leary, 1994). In particular, with regard to the foreign language classroom and the use of technology, past research has suggested that well-scaffolded online environments can provide effective and efficient opportunities to overcome the lack of exposure to the target language outside the classroom and to extend the limited opportunities to use a foreign language in everyday life (Pais Marden, 2008). For over a decade, educators have extolled the benefits of online role-play simulations, which include the provision of an environment that allows for reflective, deep and significant learning (Wills, 2012). Wills, Leigh and Ip (2011) suggested that role-based e-learning "provides opportunities to address all the principles of quality learning design" (p.12). By promoting communication and encouraging creativity, role-plays and simulations have also been shown to help develop the students' generic skills and to allow them to learn about each other in a safe environment (Pivec, Dziabenko & Schinnerl, 2003; Geurts, Duke & Vermeulen, 2007).

Despite positive endorsement of educational role-plays, Wills et al. (2011) argue that face to face role-play is only a short-term teaching tool, and that it is therefore not likely to cater for a research-intensive approach to learning. Indeed, many role-plays suggested by course books in foreign language teaching tend to be of a short-term nature, allowing little time for research and preparation (Schönherr, Jan & Dallapiazza, 2011; Aufderstra§e et al., 2011). Once this type of role-play has started, spontaneous interaction is required, inviting a comparison with impromptu theatre or "process drama", as it is referred to by Liu (2002). However, the type of online role-play analysed in this study is by its nature more akin to a simulation than to process drama. Only after in-depth research are participants expected to (re)act and deal with events in a historically appropriate fashion. The course of events is largely pre-determined by history (Dunn & O'Toole, 2009) and by carefully scaffolded learning outcomes.

In our study, we were specifically guided by the high-quality learning principles outlined by Boud and Prosser (2002), who, among others, have repeatedly proven that an ideal learning activity needs to "engage learners", "acknowledge learning context", "challenge learners" and "provide practice" (p.241). Furthermore, Wills, Leigh and Ip (2011) suggested that role-based learning considers the four components of all learning identified by Siemens and Tittenberger (2009), whereby a learning design needs to allow for "social", "situated", "reflective" and "multi-faceted" learning (p.9).

Working with these principles and translating them into assessment tasks has guided past efforts by the authors of this study, who have since devised several different scenarios and associated teaching modes, ranging from face to face role-plays and computer-facilitated simulations to blended online and face to face simulations. As the findings from their large-scale evaluation study have shown (2012), it is especially those approaches that blend technology and face to face interaction which may in fact lead to profound high-quality learning outcomes. Past research has demonstrated how face to face or in class role-plays allow for collaborative, problem-based, student-centred learning (Ludewig & Ludewig-Rohwer, 2012). However, the authors also caution that face to face role-plays are by their very nature bound to time and location, and offer only limited flexibility to the student. In this they agree with Freeman and Capper (1999), who found the benefits of web-based platforms for role-play scenarios to exist in increased (rather than total) freedom and ease of participation, as they allow for more flexibility with regard to time and place.

The conundrum

Specifically in the context of foreign language acquisition, online interaction adds two welcome advantages: anonymity and asynchronicity. A certain anonymity is provided by a secret identity, which liberates some students, who in such a setting are less afraid to contribute when any mistakes are made incognito. Bell (2001) suggests that anonymity might be of advantage, especially for the participation of non-native speakers. This argument was a particularly persuasive factor in supporting the use of secret identities in the UWA role-plays. As part of a multifaceted learning design, a secret identity has been shown to facilitate experimentation with the language in the students' text-based postings (Ludewig & Ludewig-Rohwer, 2012). In addition, compared to face to face role-play in class, which requires interaction on the spot, asynchronous web-based interaction allows for extra time to comprehend news and messages and for delayed responses which students have had time to reflect upon with regard to language and content. As a result of these benefits, online role-plays have become an integral part of teaching and assessing students within the intermediate and advanced units in German Studies at UWA.

As a result of these benefits, online role-plays have become an integral part of teaching and assessing students within the intermediate and advanced units in German Studies at UWA. Nevertheless, while the evaluation of the design principles in the web-based role-plays conducted in the past has proved that they have been effective in engaging students, a one-to-one causality of assessment type and high quality learning has thus far been an unproven claim. This study originally set out to prove this link by trying to close the last step of the endorsement logic. However, our findings have not supported this axiom; rather, they point towards a disjuncture between theory and practice, as the data analysis of this study suggests that the number of students who benefitted from this specific learning tool is much lower than anticipated.

The research project

This study forms part of a long term, large scale research project analysing the pedagogy around role-play in German language teaching at UWA and aims to answer the question "How is the pedagogy behind role-plays perceived by students?" In particular, we have been investigating whether this learning activity leads to improved language proficiency and an increased awareness of intercultural issues, as well as to improved research skills.

Learning by doing - optimising the role-play design

Over the last 4 years a series of online role-plays has been conducted annually with students enrolled in intermediate and advanced level German units. The aim of this initiative has been to offer a stimulating environment for the learning and assessment of German language and culture. Over this period, the role-plays conducted in German Studies have evolved based on student feedback. In 2008 (in line with Linser, Ree-Lindstad & Vold [2008]), students declared a preference for historical over fictional characters. After some students had taken part in a murder mystery ('Murder in High Society'), in which they were given roles ranging from fictitious reporters to that of a chihuahua, whereas others played real-life historical figures in another role-play entitled 'The Cold War'. Following further suggestions arising from the 2009 role-play ('The Cold War'), synchronous online meetings were introduced in 2010 to increase the opportunities for instant discussion. For the role-play 'We are the people!' more discussion boards were set up to encourage discussion at various levels and in differing group constellations. These were modelled on a real-life political talk show, a game show and a book fair. Instead of providing for more interest and interaction, many students reported feeling overwhelmed or confused by being confronted with too many discussion boards. In the 2012 role-play ('The enemy is whoever thinks differently!') the number of discussion boards was therefore reduced to a minimum of two, while the assessment value increased. Having thus responded to crucial student feedback and apparently optimising the learning environment, we were now less concerned with the fine-tuning of the role-play per se, than with a critical evaluation of this teaching, learning and assessment tool from a broader pedagogical perspective.

'The enemy is whoever thinks differently!'

This web-based role-play takes the students back to post-war Germany between the years of 1966 and 1976, when the country was divided into East and West. It aims to provide the students with the opportunity to experience the different levels of oppression which East German citizens encountered in the communist-socialist GDR. However, the title of the role-play also applies to the student protests, RAF terrorism and political unrest in West Germany during this period. The simulation is linked in context to interactive cultural lectures and forms part of a series of learning and assessment tools, including written grammar tests and oral interviews.

The roles are allocated by the participant researcher, and students are requested to keep their true identities secret and to remain in character for all interactions. Role allocation is guided by the students' personal language proficiency, with assumed personality type also taken into consideration. The choice of roles includes real-life celebrities and community leaders, among them politicians, authors, singers and writers. 'The enemy is whoever thinks differently!' follows the Design Space framework as set out by Wills, Leigh and Ip (2011), by focusing on stakeholders who have consequential relationships with one another and who demonstrate controversial points of view. In a study of several online role-plays, Linser, Ree-Lindstad and Vold (2008) come to the conclusion that conflict is more likely to trigger participation and engagement. The various characters in this role-play are grouped according to their interests, with a number of discussion boards allowing for conflict and competition within and between those groups.

Technical details

The role-play utilises the in-house software Simulation Builder (developed by the Multimedia Centre at UWA in 1999 and continuously improved until 2012), a customised, user-friendly and intuitive interface accessible with any web-browser. The players are organised into primary and secondary groups with different communication rules, and can edit their profile, interact with others in threaded discussions (using private individual messages, messages to all members in their group, and public messages to all), as well as upload Word documents, and set links to external sources such as websites as well as audio and video files. A news section allows the participants to receive updates about current events and can be utilised for daily (or less frequent) stimulus items. The software also permits the tracking of logins and postings.

In this role-play, news items are updated twice a week, with each day referring to a specific day or month of historical significance. Participants are divided into two groups: they are either West or East Germans who cannot communicate across the border, but can read each others' newspapers (BILD-Zeitung, the mass tabloid popular in the West, and Neues Deutschland, the East German daily paper which unquestionably toed the party line). Both media outlets report in the main on the same events, yet from very different perspectives, thus sensitising the students to the competing points of view during the Cold War period. To ensure speedy development, everyone is required to login at least twice a week to participate in synchronous online meetings, with every meeting referring to a specific task and year.

Information on assessment tasks is available on the online learning environment LMS and via announcements within the simulation, while students are given further reminders via external emails and during lectures. The participant researcher (in the role of administrator) publishes individual tasks (such as organisation of a terrorist plot, or a political response to an event) within the role-play. Tasks and news are posted 24 hours before the next meeting. The overall assessment mark constitutes 40% of the unit mark (5% diary entry and CV, 20% participation and interaction, 10% reflective statement "Where are you today?", and 5% presentation at the costume party [face to face]).

The participants

The participating students (n=47) are in their first, second or third year of their German studies and display a great variety of language and study skills.

Table 1: Student demographics with regard to their year at university and German language skills

Intermediate Intermediate Advanced Advanced Advanced

No. of students 16 20 3 2 6

Anticipated language level Comparable to Level A1/A2 of the CEF* Comparable to Level A1/A2 of the CEF* Comparable to Level A2/B1 of the CEF* Comparable to Level A2/B1 of the CEF* Comparable to Level A2/B1 of the CEF*

Year(s) of study/ semesters completed 1st year/ 1 semester 2nd year / 3 semesters 1st year/ 1 semester 2nd year/ 3 semesters 3rd year/ 5 semesters

Language background Completed tertiary entrance exam in German Former students of beginner level Native speaker background or equivalent Completed intermediate level Completed intermediate level as ex-beginners

* CEF: Common European Framework of Reference for Languages: Learning, teaching, assessment (CoE, 2012).

Data

The 2012 role-play 'The enemy is whoever thinks differently' has constituted a core element of the second semester in the intermediate and advanced German units. It started at the beginning of the semester and the timetable for its implementation, gaming phase, assessment and evaluation was as follows:

Week 1: introduction to the role-play;
Week 2: allocation of roles;
Week 3: introduction to Simulation Builder, research of character;
Weeks 4 to 7: interaction with other students;
Week 8: role-play party.

In order to answer the question "How is the pedagogy behind web-based role-plays perceived by students?", we obtained both qualitative and quantitative data from a variety of sources:

Qualitative feedback from the 47 participating students was gained through student surveys, vocabulary tests, discussions with the participant researchers and from focus group discussions with independent colleagues.
Colleagues and participant researchers provided field notes and feedback about their classroom observations (in both face to face and online interactions).
Quantitative tracking of contributions was utilised to analyse the anticipated level of engagement and learning within this environment.

The participating students were encouraged to reflect in writing on their learning at three crucial stages of the project, at the beginning, middle and end. As such their data corpus was the largest, consisting of three questionnaires, with the first completed before the start of the role-play, the second after two weeks of taking part, and the third at the end of the role-play. Using open and closed questions, the survey allowed for students to critically reflect on their learning progress and their participation in the role-play, and finally to evaluate the role-play setup. Questionnaires were not always completed by all 47 participants; Q1 n= 42, Q2 n= 35, Q3 n= 42. Together with the observations made by participant researchers and observers (colleagues who also taught the students in other settings), the students' answers formed the qualitative part of the data, whereas the quantitative data was mainly derived from analysing the software "reports", which allow for the tracking of logins and postings, with reports being taken twice a day, as well as at the beginning and end of synchronous online meetings (twice per week).

Desired learning outcomes of the role-play

At UWA an outcomes-based approach to teaching is used to plan student learning. The German units are designed for students to gain an increased level of competency in all four macroskills - listening to, speaking, reading, and writing German - and to sharpen their research skills. At the intermediate and advanced levels, where role-plays customarily find an application, students should gain an increased level of competency corresponding to proficiency level A2/B1 and upwards of the Common European Framework. This is compatible with other outcomes which are devised around more generic skills, such as an increased awareness of intercultural issues (using German as an example); a heightened awareness of the English language and how language works in general; improved research skills utilising a variety of sources and experiences; and the development of independent learning skills, interpersonal skills, and honed communication skills in a range of contexts in both spoken and written German and English. To verify the degree of achievement on the continuum of proficiency levels ranging from A2 to C2, assessment has been linked to every desired outcome, with clear marking guides devised and distributed to students in advance.

A common tool for analysing learning objectives was developed by Bloom et al. (1956). Bloom's Taxonomy of educational objectives is a "classification of the student behaviours which represent the intended outcomes of the educational process." (p.12). It classifies all intended educational objectives into six categories, organised in a hierarchy from the simplest to the most complex intended student behaviour, with each lower level educational objective being a component of a higher level, hence more complex, educational objective. The categories 'Knowledge', 'Comprehension', 'Application', 'Analysis', 'Synthesis' and 'Evaluation' are further divided into subclasses. Modern approaches to learning assume that the higher the level of complexity of the task, the more sustainable will be the learning and associated higher order thinking skills. This corresponds with the upper three levels of Bloom's Taxonomy (Hopson, Simms & Knezek, 2002).

The role-play addresses the three upper levels of Bloom's Taxonomy in terms of its learning objectives. Language-specific outcomes informing the conceptualisation of the role-play are related to communicative competencies, including:

writing and reading skills (extension of vocabulary);
interaction in the target language (initiating and maintaining dialogues in written German by using appropriate greetings, and posing and responding to questions);
meta-cognition (by reflecting on language use, such as choice of register, including formal and informal language, with special attention to the genre of text, e.g. diary entry, C.V. and role statement).

In addition to productive and reflective skills, receptive skills are also fostered, including reading comprehension, improvement and practice of global reading (news section), selective reading (identifying news relevant to the role, threaded discussion), and detailed reading strategies (media releases and threaded discussion). The learning objectives of this unit are to consolidate and improve the language ability of learners of German at intermediate and advanced levels using a research-intensive approach. The unit also aims to further develop cultural and linguistic awareness and to improve students' communication skills and knowledge of and about the target language and culture. Any role-play setting which allows for a comparatively free development of activities also hones students' intercultural skills, by requiring them to adopt another viewpoint and to engage with others from different cultural environments, varying social groupings, and diverse socio-economic backgrounds. In addition, this role-play has been designed to improve students' generic skills, for example, their ability to make informed decisions, solve problems creatively, present information in a variety of ways, and improve their Internet literacy by way of their required research (McLaughlan & Kirkpatrick, 2004).

Table 2: Tasks and corresponding skills

Role-play
Step 1 Write CV and Diary entry (200 words) Reading and writing German; language learning and communication strategies; research skills.

Role-play
Start - Step 2 Interaction, responding to events (2 postings, twice a week) Reading and writing German; language learning and communication strategies; research skills.

Role-play
Step 3 Write: Where are they today? (250 words) Reading and writing German; increased awareness of language use and improved competency in both English and German, research skills.

Role-play
Step 4 Presentation at role-play party Listening to and speaking German; increased awareness of intercultural understandings; language learning and communication strategies; research skills.

While many forms of assessment at university level, particularly in language learning, still encourage 'reproduction' rather than 'transformation' of the learned material, and test students' ability to recall rather than to apply what they have learned in new and creative settings, the online simulation role-play has deliberately incorporated assessment practices which require further extension of knowledge and not simply the application of familiar material.

Actual findings and discussion

In the following discussion the main learning outcomes and their associated assessment tasks will be evaluated against the available data obtained from the outlined sources.

Language proficiency and increased awareness of intercultural issues

As this assessment sits within the context of an intermediate to advanced language studies program, acquiring more knowledge about the German language and culture is pivotal. Students were asked to evaluate their perceived strengths and weaknesses in German at the beginning of the unit, partly by rating items such as their perceived knowledge of grammar and vocabulary on a 5-point Likert scale, and partly in an unguided text response. Students generally gave a lower rating to their level of fluency, ability to respond to questions, to use formal and informal language, to use new vocabulary and to try new grammatical features from the language class, but felt they were better able to write and read short texts and to ask short questions.

While any student self-assessment must be taken with a grain of salt, the fact that their self-assessment before the role-play was used primarily to compare it with their individual self-assessment after the completion of the role-play provided the desired insight with regard to the students' perception of their own learning, especially in relation to time-input.

Their initial self-assessment, compared with that undertaken at the end of the role-play sequence, was in turn contrasted with data obtained from their role-play contributions in the target language, from grammar and vocabulary tests conducted every fortnight in their face to face language classes, as well as from general classroom observation. Looking at the quantitative findings:

10/42 students felt that, with regard to learning of specific language features, little or no progress had been achieved over the course of the role-play.
15/42 students also reported, however, that the discussion boards and news were experienced as helpful in improving language skills.
20/42 students perceived a close link between lectures and the role-play, which they felt aided their understanding of German culture and history.

According to anecdotal feedback from colleagues teaching the students in the face to face components of the unit (i.e. in the interactive lecture and the language classes), no noticeable improvement in the understanding of specific vocabulary or historical events was observed in a significant minority, confirming the self-assessment of some students. Indeed, in-class observations indicated that new vocabulary and grammatical constructions - despite seemingly convincing evidence to the contrary in the role-play - were used by some students, but not necessarily transferred to long term memory by many. Consequently, this new "knowledge" could not be recalled in weekly face to face interactive lectures, or, in many instances, in tests conducted in later weeks.

By using text-based interaction online, students may have used new vocabulary in their written submissions, but did not seem to transfer that knowledge to their long term memory. The online setting may have tempted many to treat the material as something "online" and "remote", not realising that they should learn the new material for out of context recall. As such, neither face to face class interaction nor fortnightly vocabulary tests could confirm the students' self-proclaimed perception of their improvements.

Improved research skills utilising a variety of sources and experiences, and the development of independent learning skills

From the very beginning of the role-play, some of the students felt anxious about, and were reluctant to tackle, the assessment task, due mainly to the alienating factor of technology. This made them focus on the technological aspects rather than on the learning experience itself. In their reflective feedback on the assessment task, 13/42 students raised issues about their preoccupation with "dealing with the unknown". In the focus interviews, when questioned about how this problem had emerged, many stated that writing an essay would be more straightforward by comparison, and would have caused them "less confusion", in contrast to the online tasks which forced them to "use an unfamiliar format". Rather than appreciating the freedom provided by such an online role-play, relying on their independent learning skills, and utilising a variety of sources, many sought re-assurance and continual updates on their progress. Accordingly, in the questionnaires they expressed a desire to have had "more instructions" (17 students in the interim and 13 in the final questionnaire mentioned that the "assessment [was] not clear"). Instead of becoming immersed in and enjoying the role-playing to its full extent, the majority of students repeatedly broke out of the exercise to seek feedback. The vast majority of students stated that they "want to know all details". The fact that many were time-poor meant that they were primarily interested in efficient work / grades outcomes, and definitely did "not want to waste time by just playing" and being immersed in the experience.

As a result, for 18/42 students the open-endedness of the task was daunting, as they did not have the time to dedicate hours and hours to the role-play, and were conscious that this type of learning and assessment would interfere with their time-management. Their need for clear boundaries ("How long do you expect me to be online?", "How many words do I need to write?"), as well as instant feedback (to the effect that something was done to a satisfactory level, corresponding to a grade x or y) to alleviate any concern about passing or failing a task, detracted from the immersion. Indeed, the way the students perceived the online environment showed that some learners took to the medium and the concept of a largely self-directed simulation more easily than others. Those students who were both linguistically confident and competent, as well as technologically savvy, were at an advantage from the very beginning (cf. Liu, 2002). The role-play (which had been specifically designed to allow for several different learner types and abilities) ended up favouring the "usual suspects", i.e. those students whose abilities were already advanced and who displayed the qualities of the ideal student type: an enquiring mind, information literacy, a sense of personal agency and a repertoire of learning skills (cf. Candy et al., 1994). Others, who took longer to warm to the technology and online learning community, and found it difficult to overcome personal and interpersonal aspects of this new and anonymous group membership, had reservations and did not take the same level of control of their own learning as many of the stronger performers.

The fact that the medium for the assessment was more akin to social networking and private engagements than to academic essay writing may have also resulted in two - rather worrying - trends: on the one hand, tardy citation and referencing practices bordering on the unethical use of sources, and on the other, the use of non-academic sources, such as relying on Wikipedia to research an identity. As such, the transparent involvement of experiential and real-world learning was short-circuited and far removed from the deep learning anticipated by the team of researchers.

Input / output paradigm

The overall, and perhaps the most sobering, insight for everyone involved was that engagement and participation do not come naturally, but are very much triggered by the value of the assessment attached to the role-play activities. This insight was derived from measuring student engagement by the number of postings in correlation to assessment values. In general, interaction (as measured by the number of postings) doubled during the second half of the role-play. The overall number of postings increased from 346 (on average 7 postings per fortnight) after two weeks of playing, to 1111, with an average of 20 postings over the whole playing period.

This could be explained by the fact that students reported that they were "getting used to the software", that tutor feedback to students increased the pressure to post responses, and that clearer assessment instructions had been given in response to the feedback gained from the interim survey. However, the increased number of postings, and the apparent increase of engagement in the role-play, was - as students revealed in conversation with other tutors in the unit - more directly linked to a desire to obtain a certain grade than to furthering their knowledge.

Half of the students posted more than the recommended two postings per online meeting, with 66 marking the highest number of contributions and the lowest number 5 over the entire period of the role-play. As such, a certain peer pressure came to bear; realising that others were posting more frequently resulted in many students wanting to improve their participation mark. This spurt of activity was short-lived, however, with many students - once in receipt of a grade and feedback - dropping back to minimum input levels. In summing up their impressions at the end of the exercise, nearly two thirds of the students (25/42) claimed that they had enjoyed the role-play. However, when asked for a reason why they did or did not participate during the last week of the role-play, 16 out of 42 participants replied that they had done so because it was compulsory and part of the assessment. Only 10 explained that they had participated out of enjoyment. The reasons for not contributing were mainly attributed to time constraints. Despite following the design setup for an efficient and engaging role-play, student engagement seems to have been stimulated by its assessment value.

Given that the role-play was assessed, with 40% of the students' overall mark derived from this activity, it is not surprising that students did not feel that they participated "voluntarily". The reality of a possible "pass" or "fail", as well as general concerns about which grade they would achieve, obscured any enjoyable or fun aspects of the activity for many students. In order to pass the unit and receive a good mark, participation was compulsory and the varying levels and intensity of the students' contributions can therefore not be considered to constitute voluntary engagement with an open-ended learning tool.

This "lack" of intrinsic motivation and engagement was potentially exacerbated by the computer-based simulated environment itself. The software, when reduced to simple written communication, was easy to use, but was not learner-friendly per se; there was no automatic notification of new postings and students had to login and check for new activity. When posting a message, students were required to use the "refresh" function to check that nobody else had posted something important in the meantime. Students were also unable to access the role-play with their mobile phones. Moreover, the software did not allow for audiovisual stimulation other than via links to the web. This meant that students were forced to leave the role-play environment if they wanted to access these links and then login to the role-play again. More sophisticated software might have led to different levels of engagement and hence the stimulation of deeper learning for those students whose regular use of social networking software led them to have higher expectations of the capabilities of Simulation Builder.

While all students participating in this assessment ultimately gained experience in setting their own goals and boundaries, researching topics, and generally learning on their own, not all appreciated this learning environment to the same degree. The "staged withdrawal" of the teaching staff (participant teachers) as an authoritative and guiding voice, which was deliberately designed to facilitate a student-centred and self-directed learning experience, was viewed by some students as an abdication of responsibility on the instructors' part. Students were largely critical of this deliberate strategy to encourage deeper learning, student ownership of the task, its process and ultimate outcome, and the deliberate shift from judging performance to enjoying the learning experience.

On the other hand, given that the participant researchers also adopted roles within the role-play without being identifiable as instructors, some students were intimidated by these seemingly "über-keen" participants. The high-achievers in the unit, in particular, displayed a certain nervousness about how their personal contributions measured up. This anxiety overshadowed the anticipated benefits of the blended modes on offer to some degree, with the assessable aspects of the role-play simulation preferencing different learner types at different times: the role-play party encouraged oral and creative skills, the individual text-based submissions online highlighted written and visual skills, and the interactive engagement and participation in discussion played up collaborative skills.

Conclusion

This computer-based simulated environment aimed to encourage student engagement through tasks that allowed for open-ended learning, and involvement in an immersive experience. However, the responses received in qualitative surveys, as well as the analysis of vocabulary tests conducted in the face to face language class (which consisted of material covered as part of the online simulation), showed that active involvement in a learning and assessment activity is no guarantee of deeper learning and the appreciation of it. Although students reported being engaged to a certain extent, deeper learning, i.e. the long term retention of new material, did not necessarily take place, and the students' own perceptions of their levels of engagement, both self-directed and prescribed, were in the main negative.

A disjuncture between theory and practice became more and more apparent:

The assessment design, intended to encourage students to engage in self-directed and peer-assisted learning, was perceived as too liberal, too unguided, too open.
The transparent involvement of experiential and real-world learning was short circuited by the use of very limited and mostly non-academic resources for research (such as relying on Wikipedia to research an identity or seminal historical event).
The strategy of using problem-based learning was likewise viewed by many as artificially constructed and contrived.
The inclusion of reflective practice and critical self-awareness might have led on the whole to students becoming very critical of the assessment and learning environment.
Even those open learning and alternative delivery mechanisms which aimed to allow for freedom of choice and individual learning style preferences - the one factor which all previous research had determined to be indisputable - were perceived negatively, with many students reporting that they felt intimidated by "über-keen" contributors, and had developed a level of paranoia that their own contributions might not suffice in terms of word count, language level and content, or that their participation would not measure up against those who could dedicate hours and hours to the task.

Indeed, with many students being time-poor and struggling to combine study, work, family, and social commitments, their approach to assessment is such that they want to know how much is enough and what will get them which result; they do not appreciate that in an open-ended assessment the goal posts shift continuously.

Considering the enormous investment required for teaching staff to create a rich teaching and learning environment with the aim of engaging students and stimulating them towards higher achievement, the fact that most students attained the lesser goals they had set for themselves, i.e. passing the unit with an x or y grade - for a large minority with the minimum possible input of time and effort - is certainly something to be kept in mind for future initiatives and research into the application of idealistic teaching and learning paradigms.

Further research

Overall, it is fair to say that the analysis of the data, using narrative enquiry as a methodology, arrived at the insight that, while the majority of students enjoyed participating in the role-play, they would - if now given the choice - opt for other assessment types, as they were dissatisfied with the ratio of time input versus perceived learning output. When considering learning styles it remains to be determined whether this particular learning activity is only relevant to or stimulating for a specific type of learner, perhaps those who have a more sociable, spontaneous nature.

References

Aufderstrasse, H., Bock, H., Eisfeld, K.-H., Gerdes, M., Holthaus, H., Müller, H., Müller, J. & Schütze-Nöhmke, U., (2011). Themen aktuell 1. Kursbuch und Arbeitsbuch. Lektion 1-5. Ismaning: Hueber.

Bell, M. (2001). Online role-play: Anonymity, engagement and risk. Educational Media International, 38(4), 251-260. http://dx.doi.org/10.1080/09523980110105141

Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H. & Krathwohl, D. R. (1956). Taxonomy of educational objectives: Handbook I: Cognitive domain. New York: David McKay.

Boud, D. & Prosser, M. (2002). Appraising new technologies for learning: A framework for development. Educational Media International, 39(3-4), 237-245. http://dx.doi.org/10.1080/09523980210166026

Candy, P. C. (1991). Self-direction for lifelong learning: A comprehensive guide to theory and practice. San Francisco: Jossey-Bass.

Candy, P. C., Crebert, R. G. & O'Leary, J. (1994). Developing lifelong learners through undergraduate education. Canberra: Australian Government Publishing Service. http://vital.new.voced.edu.au/vital/access/services/Download/ngv:22704/SOURCE2

CoE (Council of Europe) (2012). Common European Framework of Reference for Languages: Learning, teaching, assessment. http://www.coe.int/t/dg4/linguistic/Cadre1_en.asp [viewed 6 May 2013]

Dunn, J. & O'Toole, J. (2009). When worlds collude: Exploring the relationship between the actual, the dramatic and the virtual. In M. Anderson, J. Carroll & D. Cameron (Eds.), Drama education with digital technology (pp. 20-37). London and New York: Continuum International Publishing Group.

Freed, B. (1995). What makes us think that students who study abroad become fluent? In B. Freed (Ed.), Second language acquisition in a study abroad context (pp.123-148). Amsterdam: John Benjamins Publishing Co.

Freeman, M. A. & Capper, J. M. (1999). Exploiting the web for education: An anonymous asynchronous role simulation. Australian Journal of Educational Technology, 15(1), 95-116. http://www.ascilite.org.au/ajet/ajet15/freeman.html

Geurts, J. L., Duke, R. D. & Vermeulen, P. A. (2007). Policy gaming for strategy and change. Long Range Planning, 40(6), 535-558. http://dx.doi.org/10.1016/j.lrp.2007.07.004

Hopson, M. H., Simms, R. L. & Knezek, G. A. (2002). Using a technology-enriched environment to improve higher-order thinking skills. Journal of Research on Technology in Education, 34(2), 109-119. http://www.iste.org/Store/Product?ID=1872

Linser, R., Ree-Lindstad N. & Vold, T. (2008). The magic circle - game design principles and online role-play simulations. Paper presented at ED-Media. World Conference on Educational Multimedia, Hypermedia and Telecommunications, 2008, Vienna. [viewed 10 Nov 2012]. http://www.simplay.net/papers/MagicCircle-Linser-Lindstad-Vold08.pdf

Liu, J. (2002). Process drama in second- and foreign-language classrooms. In G. Bräuer (Ed.), Body and language: Intercultural learning through drama. (pp. 51-70.) Westport, Connecticut & London: Ablex Publishing.

Ludewig, A. (2012). German Studies 4 - Studies in contemporary German. Unit outline. Perth: University of Western Australia.

Ludewig, A. & Ludewig-Rohwer, I. (2012). "We are the people!" Empowering students in German Studies. In C. Nygaard, N. Courtney & E. Leigh (Eds.), Simulations, games and role play in university education (pp. 257-276). Oxfordshire: Libri Publishing.

McLaughlan, R. G. & Kirkpatrick, D. (2004). Online roleplay: Design for active learning. European Journal of Engineering Education, 29(4), 477-490. http://dx.doi.org/10.1080/03043790410001716293

Pais Marden, M. (2008). The travel project: Designing and implementing an online community of learners using design based research. Paper presented at Emerging Technologies Conference, Wollongong. http://ro.uow.edu.au/etc08/19

Pivec, M., Dziabenko, O. & Schinnerl, I. (2003). Aspects of game-based learning. In Proceedings of I-KNOW'03, 2-4 July 2003. Graz, Austria (pp. 216-225). http://i-know.tugraz.at/wp-content/uploads/2008/11/32_aspects-of.pdf [viewed 2 Nov 2012].

Schönherr, T., Jan, E. & Dallapiazza, R. M. (2011). Tangram aktuell 1. Kursbuch und Arbeitsbuch. Lektion 5-8, Ismaning: Hueber.

Siemens, G. & Tittenberger, P. (2009). Handbook of emerging technologies for learning. Manitoba, Canada: University of Manitoba. [viewed 1 Nov 2012]. http://techcommittee.wikis.msad52.org/file/view/HETL.pdf

Synalski, T. P. (2009). Language learning: Myths and facts. Antimoon.com. http://www.antimoon.com/other/myths.htm [viewed 1 Nov 2012].

Wills, S., Leigh, E. & Ip, A. (2011). The power of role-based e-learning. New York: Routledge.

Wills, S. (2012). The simulation triad. In C. Nygaard, N. Courtney & E. Leigh (Eds.), Simulations, games and role play in university education (pp. 257-276). Oxfordshire: Libri Publishing.

The articles in this Special issue, Teaching and learning in higher education: Western Australia's TL Forum, were invited from the peer-reviewed full papers accepted for the Forum, and were subjected to a further peer review process conducted by the Editorial Subcommittee for the Special issue. Authors accepted for the Special issue were given options to make minor or major revisions (minor revisions in the case of Ludewig and Ludewig-Rohwer). The reference for the Forum version of their article is:
Ludewig, A. & Ludewig-Rohwer, I. (2013). Does web-based role-play establish a high quality learning environment? Design versus evaluation. In Design, develop, evaluate: The core of the learning environment. Proceedings of the 22nd Annual Teaching Learning Forum, 7-8 February 2013. Perth: Murdoch University. http://ctl.curtin.edu.au/professional_development/conferences/tlf/tlf2013/refereed/ludewig.html
Authors: Alexandra Ludewig is Professor of German and immediate past Associate Dean of Education (Teaching and Learning), has a keen interest in language acquisition pedagogies. She has published widely in the area.
Email: alexandra.ludewig@uwa.edu.au

Iris Ludewig-Rohwer has worked as a German language tutor at university level and other tertiary institutions for over 10 years. She is currently a PhD student at The University of Western Australia, investigating student learning in blended, web-based role-play simulations.
Email: ludewi01@student.uwa.edu.au

Please cite as: Ludewig, A. & Ludewig-Rohwer, I. (2013). Does web-based role-play establish a high quality learning environment? Design versus evaluation. In Special issue: Teaching and learning in higher education: Western Australia's TL Forum. Issues in Educational Research, 23(2), 164-179. http://www.iier.org.au/iier23/ludewig.html

[ PDF version of this article ] [ Contents Vol 23 ] [ IIER Home ]
© 2013 Issues In Educational Research. This URL: http://www.iier.org.au/iier23/ludewig.html
Created 14 Jun 2013. Last revision: 14 Jun 2013.
HTML: Roger Atkinson [rjatkinson@bigpond.com]

	Intermediate	Intermediate	Advanced	Advanced	Advanced
No. of students	16	20	3	2	6
Anticipated language level	Comparable to Level A1/A2 of the CEF*	Comparable to Level A1/A2 of the CEF*	Comparable to Level A2/B1 of the CEF*	Comparable to Level A2/B1 of the CEF*	Comparable to Level A2/B1 of the CEF*
Year(s) of study/ semesters completed	1st year/ 1 semester	2nd year / 3 semesters	1st year/ 1 semester	2nd year/ 3 semesters	3rd year/ 5 semesters
Language background	Completed tertiary entrance exam in German	Former students of beginner level	Native speaker background or equivalent	Completed intermediate level	Completed intermediate level as ex-beginners
* CEF: Common European Framework of Reference for Languages: Learning, teaching, assessment (CoE, 2012).

Role-play Step 1	Write CV and Diary entry (200 words)	Reading and writing German; language learning and communication strategies; research skills.
Role-play Start - Step 2	Interaction, responding to events (2 postings, twice a week)	Reading and writing German; language learning and communication strategies; research skills.
Role-play Step 3	Write: Where are they today? (250 words)	Reading and writing German; increased awareness of language use and improved competency in both English and German, research skills.
Role-play Step 4	Presentation at role-play party	Listening to and speaking German; increased awareness of intercultural understandings; language learning and communication strategies; research skills.