Joining the dots: Piloting the work diary as a data collection tool
Queensland University of Technology
This paper describes and analyses the pilot of one data collection tool, the work diary, for an educational research project. Before inclusion in the wider research project, the researcher developed, piloted and qualitatively assessed the feasibility of the data collection tool. As the wider research project will be conducted in and investigate inclusive education practices, the processes and design surrounding the development and pilot of the work diary were aligned to the principles of inclusivity. This paper provides reflections and lessons learnt about the importance and relevance of the pilot as worthwhile research practice. It also provides specific discussion about focused preparation by the researcher to overcome some of the limitations associated with the use of the work diary and its implementation in educational research. Specifically, the pilot was undertaken to 1) determine what research partners thought and felt about the use of the work diary; 2) determine what improvements and or changes needed to be made to the work diary and related processes before implementation. Findings of the pilot confirmed wider research commentary about the benefits and limitations about the use of the dairy under the themes of 1) purpose, 2) format and ease of use and 3) participant preparation and support. The benefits of comprehensive pre-planning and design process outweighed the limitations. The work diary was a useful data collection tool to record activities of busy school teams as required for the wider research project. The use of the diary was likened to joining dots of a children's drawing activity to reveal a complex picture of interactions.
This paper describes the design, pilot and evaluation of the diary, before it was used as a data collection tool in a wider PhD critical ethnographic research project. The design and methods employed in this research were selected to link with the social cultural theme relating to human rights and inclusion. This theme provided a theoretical and practical foundation for why and how the research was carried out and reported. To the critical ethnographer "data are where you find them, and all things are potential data" (Thomas, 1993, p. 39). Others recommend that critical ethnographic methods of data collection need to be "dialogic, dialectical and collaborative" (Angrosino, 2007, p. 12). To achieve this, the diary was included in the research methodology as a data collection tool. It was considered both a document and a tool for participant observation. Designed, planned and used in the context of an inclusive research project, the diary processes would need to be "dialogic, dialectical and collaborative" (Angrosino, 2007, p. 12). There is much academic criticism of the use of the diary as a research tool with many reported disadvantages (Bolger, Davis, Rafael, 2003; Hall, 2007; Nicholl, 2010). Reported disadvantages include those related to its purpose, format and participants (see Figure 1). A pilot was necessary to determine the reported disadvantages of its use in the research context.
The pilot or the field test is characterised by the administration of a procedure, tool or instrument to a group prior to a wider research project (Creswell, 2008; Kervin, Vialle, Herrington, Okely, 2006). For this research, data from the diary pilot provided vital information to and experience for the researcher. The pilot enabled the researcher to evaluate and refine related processes such as time taken for participants to complete (Kervin et al, 2006) and improvement to the instrument itself for the research project (Creswell, 2008). This was an important factor as the tool was new and was being used with an unfamiliar group of participants (Polit & Beck, 2004). The pilot assisted a relatively new researcher gain experience with working with participants and the tool itself (Beebe, 2007).
The process and outcomes of a pilot can provide significant information including its difficulties and benefits for the wider research community. Sapsford & Jupp (2006) recommend that a researcher look for a critical account of how a data collection tool has been piloted to provide guidance. Some research articles report the use of a pilot to test a data collection tool (Williams, Walker, Martin, et al, 2008). However, the reports rarely describe the process of the pilot or even the findings. The benefits of pilots within the research community, has been described as "undervalued and underreported in the literature" (Beebe, 2007, p. 213). Consequently, the importance of the pilot is underestimated by researchers and many have adopted a data collection tool without consideration of the issues that may impact on its effectiveness.
No specific advice about conducting a pilot of a qualitative diary in an educational setting was found in the research literature. The resulting intent of this pilot was to respond to predetermined disadvantages described in the literature about the use of the work diary's implementation, rather from any previous diary pilot recommendations.
Records small snapshot in time;|
Not widely used in ethnography;
Reactance - may cause a change in participants;
Cannot be used by itself;
Lower cost compared to wages/time of observer of multiple participants;|
Multiple observation record;
Pre-cursor to interview;
Records and reconstructs events;
Creates conscious perceptions;
Shows practices vary over time;
Rich and illuminating;
Can be used in conjunction with other tools to confirm or inform;
Access to events not easily observed or influenced by presence of observer.
Extensive training and practice;|
Time consuming to create and complete;
Onerous for diarists;
Difficult to maintain anonymity and privacy;
Restricts comparison of events;
Complex, detailed written and verbal instructions and terminology required;
Examples need to be provided.
Opportunity to compose narrative;|
Daily recordings more accurate;
Reduces recall error by diarists;
Provides choice and elucidation;
Records feelings, perceptions and behaviour immediately.
Reflection may be challenging;|
Event to be identified;
Varied literacy/articulacy skills;
Commitments and dedication to complete;
Forgetfulness to complete;
Self editing of events;
Follow up and support by researcher.
Can write about events their way;|
Become participant observers;
Opinions are valued;
|Adapted from Alaszewksi, 2006; Bolger et al, 2003; Camburn et al, 2010; Clayton and Thorne, 2000; Crosbie, 2006; Hall, 2007; Huang, 2005; Marelli, 2007; Nicholl, 2010; Wilkinson et al, 2005; and Zimmerman & Wieder, 1977.|
The researcher needed to determine how the diary 'fits' with other data collection methods before use. For example, the type of data collected for this research project was qualitative with the intention to gain insight into how and why team members engage in particular interactions. Therefore, the diary's usefulness to collect qualitative data needed to be determined. The use of a diary to collect qualitative data within school teams has been limited as many of the reports of its use are related to quantitative studies in medical or scientific contexts (Camburn, Spillane& Sebastian, 2010). In the field of education, diaries have been used in the area of linguistics and classroom language learning (Bailey & Nunan, 1996; Hall, 2007), and are consistently cited as contributing to in depth understanding of teaching and learning (Huang, 2005). The diary has been used to quantitatively determine the time spent on leadership and management activities of school principals (Camburn et al, 2010).
Diaries are commonly used in combination with other data collection methods and frequent disadvantage is noted about its use in isolation (Basit, 2010; Crosbie, 2006: Nicholl, 2010). As for most ethnographic studies, the use of a variety of data collection tool verifies or helps better understand the dairy data collected (Marelli, 2007). For example, this research used observation, and semi-structured, conversational interviews in addition to the work diary. The combination of data collection methods allowed the critical ethnographer to delve deeply to uncover "individual subjectivity and collective belonging" (Madison, 2005, p. 26).
The diary has been used in social and educational research contexts to enable the researcher to "give voice to other people" (Plummer, 1983, p.1). It is necessary for the researcher to determine if this is a relevant purpose for the diary in their particular research. For example, it was chosen for this research to provide opportunity for research participants and researchers to became both observers and informants themselves (Hall, 2007; Zimmerman & Weider, 1977). The use of the diary was seen as an inclusive and collaborative data collection method. This researcher ensured effective and open communication at every stage of the pilot and referred to participants as "partners". For example, when negotiating access to the research site the aims of the pilot and its methods were explicitly outlined to the partner school. The inclusion of the partners in the research process in itself made the process more complicated. Ultimately the researcher was 'on site' for longer to develop trust and gain input from the research partners. However, the researcher was able to gain greater insight into how partners interpreted interactions and how they assigned significance to actions and events highlighted in the diary (Alaszewski, 2006).
The use of the work diary and its relationship to the data collected and the type of setting in which the research takes place, requires analysis before its use. For example, the researcher wanted to collect qualitative data about the interactions between staff about curriculum in a naturalistic setting. This contrasts with the use of diaries for medical and social research that focuses on collection of data in structured formats about time spent in activities by its participants (Alaszewski, 2006; Basit, 2010; Wilkinson, Wells & Bushnell, 2005). Given that the researcher would collect more subjective data and wanted research partners to make decisions about what was important enough to diarise, it was vital that this be discussed with the partners . This was included in the plan to prepare partners for the diary's use. The diary's format was also designed to ensure data would be collected in an easy to use format.
One intention of the work diary was to compliment and provided the basis for questions for semi-structured, conversational interviews within the wider research project (Lewis, et. al., 2005). It was important that the diary design proactively addressed some of the known disadvantages reported with its design and ease of use before the pilot (see Figure 1). Researchers need to predict what barriers to completion they may encounter in any pilot. For example, the format of this diary needed to provide space for details, such as time of the day, and what the interactions were, to determine the patterns and purposes of interactions between members of the school team. This format needed to supply the researcher with enough data to determine questions for semi-structured, conversational interviews to follow.
Variable participant motivation to complete; and inaccuracy of recording of the details of the event (Marelli, 2007) are disadvantages noted in the literature. Researchers need to proactively apply processes that assist to overcome known barriers to completion. For example, to respond to the variable 'participant motivation to completion' noted in the literature (Crosbie, 2006), the researcher determined that the 'diary guide' (Appendix 1, pp. 1-3) needed to be clear about the purpose of diary and the wider research. Though they should not be copious, provision of written instructions were important to support partner's motivation to complete the diary accurately (Basit, 2010). The 'diary guide' also provided reminders such as avoiding the use of personal details about students, parents and colleagues. This addition was provided to respond to the caution of issues related to confidentiality and anonymity when using a diary in research (Nicholl, 2010). The section in the guide, 'filling in the daily entries', was written using jargon free language (Nicholl, 2010) and reflected the researcher's intention that the completion of the diary would not be an onerous task (Basit, 2010; Nicoll, 2010). Examples of what events could be recorded and how it could be recorded were also provided (Figure 1).
An example of a completed diary entry was provided to participants. The example page was an important element as it modelled the amount and type of data to be recorded. The diary entry page provided space for differing levels of detail regarding the interactions between partners. Figure 1 shows the type of information recorded frequently (e.g. type of interaction and people involved), that could be ticked to reduce the burden of time to complete the entry (Basit, 2010).
The researcher acknowledged that predetermining the structure of the diary entry page may not meet the needs of the research partners (Nicholl, 2010). Therefore, an electronic version was provided and partners were encouraged to modify the diary to meet their needs (e.g. size, method of data entry either electronic, audio or pen and paper). The intent of this example and instructions was to overcome variable motivation for completion linked to participant confusion about how to complete the diary entries (Bolger, Davis & Rafael, 2003; Crosbie, 2006). Extensive partner preparation before the use of the diary is commonly noted as a benefit and a disadvantage for researchers within the literature (Basit, 2010; Bolger, et al, 2003; Camburn et al, 2010). Preparation was designed, planned and delivered in conjunction with an easy to use format.
Figure 1: Examples page
Emphasis on participant preparation to overcome issues of variable participant motivation for completion and accuracy of recording of events was determined for the pilot. For example, it was important that were a similar composition (roles and numbers) to the group intended for the wider research project. A representative sample was identified through the researcher's previous professional relationship with the educational site. As for most participant recruitment, issues of diarist recruitment had to relate to the purpose of the study; build relationships and contact; and build trust (Alaszewski, 2007). For example, planning how the researcher would build trust and persuade partners to keep records was one item for consideration before the pilot began. Commonly, researchers provide incentives for research participants (Lewis et al, 2005). In this case, the ethical approval by the school's governing body was given only if the researcher did not provide incentives of any kind. Instead, relationships with partners and persuasion to maintain the diary depended upon multiple visits to the research site where the researcher engaged in informal social and professional activities with the partners.
Extensive briefing of research partners is vital before a pilot. Before invitation to join the project, all members of a school Special Education Program (SEP) and the Principal were briefed about the research pilot and whether they would be directly involved or not. The research partners who agreed to participate included the Principal of the school, Head of Special Education Services (HOSES), one SEP Teacher and two Teacher Aides. These staff members completed the work diary at the same time for five continuous days. Researchers using the diary need planned processes for their response to incomplete or not completed diaries. For example, two partners signed consent for involvement in the pilot, though did not return the diaries. To encourage these partners to comply, a letter, new diaries and a self addressed envelope were sent to the partners after the completion date. These were not returned. It is not known if these partners completed the five days of diary entries.
This staff member decided at the end of the first day of the pilot, when s/he realised that "the notebook was sitting on desk and I thought God almighty - I have not done this - and there is more to do" to change the format of the diary. When faced with the choice of writing in the diary in the car park, the research partner decided to use a digital pen to 'record' her recount of the day's events driving home.
I reflected upon the whole day from start to finish quite easy due to normal practice when driving home. The whole of day took approx half hour each time. I could also make note of things to do simultaneously without having to make a handwritten note or stop the car. This was an unexpected advantage. (Partner 1)
Comments about time spent completing the diary varied. Partner 5 found it not time consuming at all. Partner 3 commented it did take time but did not expand upon this. Partner 1 found time to complete the daily entry most challenging due to the busyness of the day's events related to student behaviour.
I found time management i.e. scheduling of time to make entries in diary was particularly difficult due to the near constant disruptions in "day to day" running of the SEP (This week was particularly challenging w. student behavioural support). (Partner 1)However, Partner 1 overcame this issue by changing the format from written to digital recording of audio and completing on the drive home from the school each day. "This was no extra work or impact on time or unexpectedly being distracted from task as during the day. This action reflected the importance of partnership in an inclusive ethnographic study. The change of the format did not interfere with the quality of diary entries. Affording partner decision making and ownership in this pilot improved the data collection tool. Partner 1 also commented that the timing, week 7 of term 1, was not the best for recording conversations about curriculum in a week where the cross country was held and staff were experiencing stress about literacy and numeracy expectations, and the number of relief teachers working in the school. Originally, consideration was made for the pilot not to be held during the week of the cross country, unfortunately the cross country was cancelled the week before and moved to the diary pilot week.
Overall, the participants provided positive comments about the way in which they were prepared and supported during the pilot. Partner 1 made comments about the preparation and support on their feedback page. This partner commented about the deliberations they had when addressing staff questions about the diary in the absence of the researcher. Reference was also made to the provision of the researcher's email address and mobile telephone number by the researcher as useful. For example:
[I] was not sure if I was to follow up with staff regarding their diaries despite being aware I could contact Mrs. Duke at any time. I chose to refer to actual communications and directions I had given by Mrs. Duke where staff had been informed they could contact her at any time (email or mobile) and as they are all adults would proceed with this if they felt required. (Partner 1)Partner 1 also commented about the way in which the researcher offered preparation and support by referring to flexibility and unobtrusive participation in the school's daily activity.
Choices of communications with self, staff and admin were flexible, which actually enabled the initial communications to be successful for all. Staff duties were not interrupted and staff found the flexible and friendly approach both supportive and not confronting. (Partner 1).Suggestions to improve the preparation and support for the use of the diary from both partner feedback and researcher's field notes, included (a) researcher defining what a 'curriculum' conversation could be (b) whether it should be recorded in the diary (c) including space in diary entry area for writing about the "Intent of actual enacted communication and intended communication" (Partner 1).
I quickly realised that if this week continued on or was representative of a "typical week" I realised the limitations on my intended leadership regarding curriculum change. (Partner 1).Partner 1, who chose to record audio diary entries, also found the daily event of diary completion as "liberating". S/he noted:
[O]nce the thoughts were captured (documented) further thoughts flowed more freely and subsequently were more constructive. I am aware due to previous analysis of my own cognitive processes that I have success after consulting with others - even if in the sense of a "listening board." I experienced a "mental check list" of the day - and procedures and processes were qualified.Though, they did not expand further, Partner 4, commented that they looked forward each day to completing the diary entry. The researcher assumed this was because of the opportunity the diary provided a tool for recounting or reflecting upon the day's conversations.
In response to the pilot findings, the researcher made the following modifications or changes to format and processes to ensure effectiveness and usefulness of the data collection tool.
Alaszewski, A. (2006). Using diaries for social research. London: SAGE.
Basit, T. (2010). Conducting research in educational contexts. London: Continuum International.
Bailey, K. & Nunan, D. (Eds.) (1996). Voices from the language classroom: Qualitative research in second language classrooms. Cambridge: Cambridge University Press.
Beebe, L. (2007). What can we learn from pilot studies? Perspectives in Psychiatric Care, 43(4), 213-218. http://dx.doi.org/10.1111/j.1744-6163.2007.00136.x
Bolger, N., Davis, A. & Rafaeli, E. (2003). Diary methods: Capturing life as it is lived. Annual Review of Psychology, 54, 579-616. http://dx.doi.org/10.1146/annurev.psych.54.101601.145030
Camburn, E., Spillane, J. & Sebastian, J. (2010). Assessing the utility of a daily log for measuring principal leadership practice. Educational Administration Quarterly, 46(5), 707-737. http://dx.doi.org/10.1177/0013161X10377345
Clayton, A. & Thorne, T. (2000). Diary data enhancing rigour: Analysis framework and verification tool. Journal of Advanced Nursing, 32(6), 1514-1521. http://dx.doi.org/10.1046/j.1365-2648.2000.01609.x
Creswell, J. (2008). Educational Research. Upper Saddle River: Pearson.
Creswell, J. (2007). Qualitative inquiry and research design. Thousand Oaks: SAGE.
Crosbie, T. (2006). Using activity diaries: Some methodological lessons. Journal of Research Practice, 2(1). http://jrp.icaap.org/index.php/jrp/article/view/22/68
Hall, G. (2007). An ethnographic diary study. ELT Journal, 62(2), 113-122. http://dx.doi.org/10.1093/elt/ccm088
Huang, J. (2005). A diary study of difficulties and constraints in EFL learning. System, 33(4), 609-621. http://dx.doi.org/10.1016/j.system.2005.04.001
Kervin, L., Vialle, W., Herrington, J. & Okely, T. (2006). Research for educators. South Melbourne: Cengage.
Lewis, K., Sligo, F. & Massey, C. (2005). Observe, record, then beyond: Facilitating participant reflection via research diaries. Qualitative Research in Accounting & Management, 2(2), 216-229. http://dx.doi.org/10.1108/11766090510635451
Madison, S. (2005). Critical ethnography. Thousand Oaks, USA: SAGE.
Marelli, A. (2007). Work diaries. Performance Improvement, 46(5), 44-48. http://dx.doi.org/10.1002/pfi.133
Nicholl, H. (2010). Diaries as a method of data collection in research. Paediatric Nursing, 22(7), 16-20. http://nursingstandard.rcnpublishing.co.uk/archive/article-diaries-as-a-method-of-data-collection-in-research/
Plummer, K. (1998). Documents of life: An introduction to the problems and literature of a humanistic method. London: Unwin Hayman.
Polit, D. & Beck, T. (2004). Nursing research: Principles and methods. (7th ed.). Philadelphia: Lippincott, Williams & Wilkins.
Sapsford, R. & Jupp, V. (2006). Data collection and analysis (2nd ed.). London: SAGE.
Sheehy, K., Nind, M., Rix, J. & Simons, K. (2005). Ethics and research in inclusive education. New York, USA: Routledge Farmer.
Thomas, J. (1993). Doing critical ethnography. Thousand Oaks, USA: SAGE.
Wilkinson, T., Wells, E. & Bushnell, J. (2005). Using a diary to quantify learning activities. Medical Education, 39, 657-664. http://dx.doi.org/10.1111/j.1365-2929.2005.02196.x
Williams, P., Walker, J., Martin, T., Northington, L., Waltman, P. Beachan, J. & Grant, L. (2008). Comparing clinical competencies between nursing students with degrees and traditional students. Issues in Educational Research, 18(1), http://www.iier.org.au/iier18/williams.html
Zimmerman, D. & Wieder, D. (1977). The diary: Diary-interview method. Urban Life, 5(4), 479-497. http://dx.doi.org/10.1177/089124167700500406
When completed, your work diary will provide me with important information about the day to day leadership practices of your team to implement curriculum policy. Curriculum policy includes what you do when planning and implementing teaching, assessment and reporting.
This phase of the research is to seek a deeper understanding of when and how staff used leadership practices when working together to implement curriculum policy. At the moment I am not sure what these are. Your diary will help me identify these important practices. Later this information will assist me to develop questions for interviews with you about this.
By looking at your completed diary, I wish to get an understanding about
I do not want this to be a tedious task. I have attempted to make the diary as easy as possible to complete. I will also be available to assist with any issues you find with filling in the daily entries via email or mobile phone.
Twelve entry pages are included in the diary. You do not need to enter more than two interactions per day if you prefer. If you do decide the make more than twelve entries, I can provide you with more pages.
Thank you so much for taking the time to complete this diary. Please don't hesitate to contact me if needed.
|Author: Jennifer Duke is a lecturer in the Faculty of Education at Queensland University of Technology. She is in the final stages of completing her PhD about inclusive education and leadership.|
Please cite as: Duke, J. (2012). Joining the dots: Piloting the work diary as a data collection tool. Issues In Educational Research, 22(2), 111-126. http://www.iier.org.au/iier22/duke.html