Collecting New Data

Introduction

By this point, you have explored what it means to formulate specific, context-driven questions and to identify and locate existing data that will help you answer these questions. You may find, however, that some of the data you need are not immediately available. Perhaps the data do not yet exist, and you will need to collect them.

In this module, you will read about Alex and Beth as they identify data needs and collect their own data. Cristina sought to answer her question with existing data sources and initially decided not to collect more data. The activities in this module encourage you to consider the information you need and how you can collect useful, valid data in your daily work without extensive time and effort.

To continue Module 3: Collecting New Data, click on Objectives & Keywords in the right-hand navigation.

Objectives & Keywords

Learning Objectives

After completing this module, you will be able to:

  1. Identify new data sources needed to answer your question
  2. Define the relevant population and sample to provide data for your question
  3. Determine which data collection methods to use
  4. Design and implement a simple data collection plan

To continue Module 3: Collecting New Data, click on Keywords in the right-hand navigation.

Keywords

Confidentiality: Proper use of information that can be tied to an individual; researchers or educators ensure identifiable data is protected to preserve individual privacy.

Convenience sample: A subset of the population that is easily available and accessible. Data gathered from this type of sample may not be entirely representative of the population and this should be acknowledged when reporting results.

Data collection: The act of obtaining information to be used to understand an issue and/or solve a problem.

Data collection plan: The step-by-step plan of action for collecting your data, including a description of the data to be collected, the instrument or method used to collect the data, a description of the sample, and the timeline for collection.

Instruments: Data collection tools, which may include tests, surveys, and interview protocols.

Participants: Individuals who consent to disclosing information about themselves for the purposes of research or evaluation.

Population: The entire group of people you are interested in examining more closely. In educational evaluation, this is typically a class, a school, a college, or a specific subgroup of students on a local, regional, or national scale.

Privacy: The protection of any information that would reveal the identity of an individual person or student.

Qualitative data: Non-numerical data that provide in-depth descriptions or narratives, such as interview transcripts, written comments from surveys, or notes from observations.

Quantitative data: Numerical data, such as test scores, class rankings, grades, or numerical ratings.

Sample: A subset of the identified population. A sample is often used in evaluation, as it is usually easier or more economical to gather data from a sample rather than the entire population.

To continue Module 3: Collecting New Data, click on Case Studies in the right-hand navigation.

Case Studies

Alex

At the Classroom Level: Alex

Read more about Alex’s question and exploration of existing data.

Looking at his list of potential factors that might influence his students’ physics problem-solving skills, Alex realized there were a number of factors that weren’t measured by existing data. No school record indicated how persistent or effortful students were in their learning, for example. Alex decided he wanted to measure these qualities in his students. After informally discussing this idea with his principal and other teachers in his school, he discovered that other teachers were similarly interested in learning about their students’ attitudes towards learning, but no one had attempted to measure it. Therefore, he needed to collect additional data.

Returning to his classroom after one of these discussions, Alex realized that students were providing him and his colleagues with information that was not reported in their grade books. Once a week, Alex collected a homework assignment from each student and recorded the assignment grade in his grade book. He always knew the score indicated the student’s level of understanding of the material, which is why each student’s score was recorded and counted towards the final grade. However, he also thought of recording a measure of the assignment’s completeness, to indicate the level of effort put forth by the student. Next year, as part of his data collection plan, he will record not just the percentage of questions that students answer correctly but also the percentage of questions that students attempt to answer. This is an efficient example of a data collection method Alex could use to show how effort and persistence play a role in student performance.

To continue Module 3: Collecting New Data, click on Beth in the right-hand navigation.

Beth

At the Department Level: Beth

Read more about Beth’s question and exploration of existing data.

Collecting historical data about students was easy for Beth because she had access to the institutional student files. However, she could not find any measurement of the outcome that was a major concern of the faculty: the quality of the students’ writing in their literature courses. Students’ grades in those courses could be a source, but not the only source of data. Although students with poor writing skills tended to earn lower grades than students who wrote well, the final grades in a literature course might also reflect other course activities, such as attendance, oral presentations, or participation in class discussions. Beth needed a new instrument to gather data that focused on students’ raw writing abilities.

Now that the semester had just ended, Beth decided that she would approach the faculty members she had talked with and ask them to make a rough evaluation of each student’s writing ability in their course on a scale of 1 to 10, with a one being an extremely poor writer, a five being an average writer, and a 10 being an outstanding writer. Some faculty members were initially resistant to participate, but Beth emphasized that she was only expecting them to make quick judgments about each student. This freed the faculty members from being overly concerned with evaluating each individual student perfectly. In this situation, Beth managed to develop a data collection plan that required minimal time and effort from the faculty.

Beth also had to consider the privacy and confidentiality of the data, so she asked the faculty members to submit only their ratings of each student, without providing individual student names or other identifying information. In addition, once she received their ratings, Beth was careful about keeping the data in a secure location where no one else would have access.

To continue Module 3: Collecting New Data, click on Cristina in the right-hand navigation.

Cristina

At the Institutional Level: Cristina

Cristina decided that she already had access to all the data required to answer her question, so she did not need to collect any new data. This module is not applicable to her case study.

To continue Module 3: Collecting New Data, click on Module in Action in the right-hand navigation.

Module in Action

Video 3.1 – Using Data in the Classroom: Part 2

In this video, Dr. Dawn Zimmaro discusses how she collected new data in her college classroom. Click here to watch part 1 of her story.

To continue Module 3: Collecting New Data, click on Data Sources in the right-hand navigation.

Data Sources

Intro

Revisit the problem statement and question you identified in Module 1. In Module 2, you may have discovered that any available data you have does not fully answer your question, or data does not already exist. If that is the case, you will need to identify the specific data to collect. Think about the type of data you are seeking. Can you manipulate information you already have in a different way? For example, Alex actually had the assignment completion data he needed. Instead of collecting new data, he just had to modify how he was recording the information for himself in his grade book. Beth, on the other hand, would have to collect the data she needed.

You may also want to consider what form of data may be most helpful to answer your question. Would it be better to have quantitative data? Examples of quantitative data include:

  • Numerical ratings
  • Grades
  • Test scores
  • Class rankings
  • Attendance records
  • Student demographics

Or would qualitative data be more useful? Qualitative data include:

  • Verbal comments from interviews
  • Written comments from surveys
  • Observations, usually visual

Your problem statement and the specific question you are trying to answer should drive the form of data you collect. To learn more about qualitative data, view Video 4.3, featuring Dr. Richard Reddick.

To continue Module 3: Collecting New Data, click on Activity 3.1 in the right-hand navigation.

Activity 3.1 – Identifying New Sources of Data

Review the instructions in the activity below, and use your workbook to record your responses.

To continue Module 3: Collecting New Data, click on Activity 3.2 in the right-hand navigation.

Activity 3.2 Data Sources at the Department Level

To continue Module 3: Collecting New Data, click on Population & Sample in the right-hand navigation.

Population & Sample

You need to decide if you want data from students, faculty, staff, parents, or some other group; do you need data from all members of your target group or just a few? Population is the term to define everyone in your target group. For Alex, whose target group is his classroom, it is easy to collect data from the entire population. However, for Beth it may not be as easy to collect data from all of the upper-division level writing instructors. She will likely use a sample instead. A sample is a subset of the population. Ideally, a sample is representative of a population, and the easiest way to achieve a representative sample is to randomly select people for your sample. Beth acknowledges the logistical constraints involved in her situation, so she will likely rely on a convenience sample, comprised of any instructors who are willing to rate their students. Although this sampling method is not ideal, it is practical under the circumstances and can still yield meaningful results.

To continue Module 3: Collecting New Data, click on Instruments & Timing in the right-hand navigation.

Instruments & Timing

How to Collect Data

Once you identify the information you need and the individual(s) who will provide it, you need to decide specifically how to collect those data. There are many, many data collection methods and instruments, including surveys, interviews, and reviews of existing documents. The question to ask yourself is, “How can I collect the best data possible in the most efficient way possible?”

This may require you to create your own data collection instruments. This may sound overwhelming, but consider how simple the process was for Alex and Beth to collect their data. Like Alex, you may only need to create an additional column in your grade book to note assignment completeness or timeliness. Alternatively, you may write a two-question survey for your colleagues to complete, or you may facilitate a structured conversation with them about learning challenges they have observed. You may ask students to write a “one-minute paper,” which could literally take one minute at the end of a lesson for students to write down one thing they learned and one question they have. Your methods don’t have to be time consuming, and your efforts may save you time in the long run. Keep your data collection simple.

When to Collect Data

It is also important to consider when you collect data. In Alex’s case, he could wait until the end of the year to collect the additional data because he would have a year’s worth of data to help him answer his question related to his students’ ability to solve problems in physics. He has already observed, however, that this year’s students are having problems even after he changed his teaching approaches. He may decide that he doesn’t want to wait and would rather collect the data now so that he can impact his current students. Alex would still need to set a specific time frame for the beginning and the end of data collection. He may decide to collect data on a weekly basis in his grade book, intermittently throughout the semester, or all at once at the end of the semester.

If you need to collect data from your colleagues, consider asking if you can use five minutes during a faculty meeting. This is a time when you have a captive audience, and it will be easier to get a good response rather than seeking people out one by one. If you need data from students, perhaps try reserving one minute at the end of a class for your students to write down one question they have about the day’s lesson; again, this is a time you have a captive audience and when the day’s learning will be freshest in your students’ minds. If you need data from school database systems, access them during times of the semester that are less hectic; don’t try to tackle a long-term data collection project in the first or last week of school. Busy times are often the downfall of data collection efforts, so plan ahead and allow yourself the extra time and space you need to follow through. It is important to familiarize yourself with the timelines when certain data are updated, particularly state-level data.

To continue Module 3: Collecting New Data, click on Permissions & Access in the right-hand navigation.

Permissions & Access

Affirm Permissions and Access

Once you have created your data collection plan, make sure you have the required permissions and access. For informal classroom data collection on course content, like Alex’s scenario, we recommend checking with your department head or principal. Department heads and principals like to know when data collection and evaluation are taking place. You may not need permissions to collect additional grading data or informal survey data from your students, but it’s critical to make sure you follow school procedure and protect your students’ privacy. If you will need to collect data from your colleagues, as Beth did, check with your department chair to make sure he or she understands the purpose and is on board. If you need to collect additional data from school databases or other sources, work directly with the administrator or technical specialist who is the gatekeeper to the data, and be sure your access has been approved by your superiors. If you need to collect additional data from students or faculty, you may need to get permission from your Institutional Review Board (IRB). An IRB helps protect the rights of participants in research. Some evaluation may be considered research. IRBs provide training in ethical research and facilitate the approval for research and evaluation.

Ask For Help

Take your ideas to a colleague who has prior experience with the type of data collection method you are using, and get his or her advice and assistance. We encourage you to bounce your ideas off other colleagues who may be interested in the same questions as you or can provide support. The feedback you receive from colleagues can help you decide if your data collection method seems wise, efficient, and/or practical. The data gatekeepers at your institution are also a great resource. They will know and understand data available to you and be able to advise whether you need an IRB review. In Module 2, you may have already identified collaborators at your own institution or peer institutions who are interested in the same problem or question. These are the very people who can also help you when you’re ready to collect new data.

To continue Module 3: Collecting New Data, click on Data Collection Plan in the right-hand navigation.

Data Collection Plan

Activity 3.3 – Creating a Data Collection Plan

Review the instructions in the activity below, and use your workbook to record your responses.

To continue Module 3: Collecting New Data, click on Conclusion & Review in the right-hand navigation.

Conclusion & Review

Conclusion

Module 3 directed you in outlining the process of designing a data collection plan. Start small, and remember that you don’t have to get it exactly right on your first try. Carrying out a simple plan like Alex’s or Beth’s is another important step toward becoming a data-driven decision maker and educator. When collecting your own data, focus on questions and problems within your control as a classroom teacher or administrator. Consider how you will be able to act on what you learn, which is the key to data-driven decision making. You may be amazed by the small amount of time and effort required to collect the kind of data that can shed new light on teaching and learning. In Module 4, we will address techniques to analyze and interpret data.

To continue Module 3: Collecting New Data, click on Review Questions in the right-hand navigation.

Review Questions
  1. What is the difference between a population and a sample?
  2. What are at least three ways you can collect data that don’t already exist at your institution?
  3. What is an example of a situation in which you should seek permission to collect new data?

To view the supplemental materials for Module 3: Collecting New Data, click on Supplemental Materials in the right-hand navigation.
To move on to Module 4: Analyzing and Interpreting Data, click on Analyzing and Interpreting Data in the left-hand navigation.

Supplemental Materials

Click the links below to download the supplemental materials.

To move on to Module 4: Analyzing and Interpreting Data, click on Analyzing and Interpreting Data in the left-hand navigation.