Evaluation is already an essential component of what we do in the field of education, whether we realize it or not. We’re used to asking questions, and, often, they are closely tied to pedagogy and its relationship to academic success. We may ask questions such as:
- Are our students succeeding and to what extent?
- How do we define “success”?
- Are our departments supporting students and educators?
- How well are our schools, community colleges, and universities fostering student learning?
- How can we, as educators, improve our teaching?
Considering these types of questions on a daily basis is an important first step toward becoming a data-driven decision maker. Next, we must challenge ourselves to be systematic when we ask and answer these questions. The basic steps in this process include:
- Scanning the environment for patterns
- Noticing a problem that needs action or decisions that need to be made
- Formulating a problem statement for decision making
- Formulating a question that will guide data collection
As an educator, you likely already understand the importance of asking questions about teaching and learning. Even so, perhaps you find the process of identifying and answering these questions problematic or difficult. In this module, you will begin the process of data-driven decision making by exploring the process of asking precise, measurable questions.
To continue Module 1: Asking Questions, click on Objectives & Keywords in the right-hand navigation.
Objectives & Keywords
After completing this module, you will be able to:
- Scan your environment for patterns
- Identify patterns that strike you as problematic, noteworthy, or that require decision making
- Prioritize observations
- Formulate a problem statement to clarify the decision that needs to be made
- Formulate a question that will help inform decision making
- Phrase precise questions that are measurable
To continue Module 1: Asking Questions, click on Keywords in the right-hand navigation.
Data-driven decision making: Using various forms of data, information, or evidence collected in a systematic way to make decisions, rather than relying on assumptions or anecdotal information. You may also hear this kind of decision making called evidence-based decision making or data-informed decision making. More often than not, evidence-based decision making refers to a program or an intervention and using data to tell if the program worked. The concept of data-driven decision making is broader by comparison, because it can apply to any inquiry, not just an inquiry about a program.
Environmental scan: The observation of various elements of your environment (i.e., your classroom, campus, or district), which should include identifying patterns, trends, and questions around you.
Pattern: Recurring traits, tendencies, or situations that are observable and can indicate a problem to be addressed.
Problem: A broad description of an issue that becomes more apparent when a pattern emerges.
Problem statement: A precise description of the problem to be addressed, which should include what the problem is, whom it affects, how it affects them, and what decision needs to be made.
Question: A specific, measurable inquiry that is derived from the problem statement.
To continue Module 1: Asking Questions, click on Case Studies in the right-hand navigation.
In Module 1, you’ll meet three data-driven decision makers—Alex, Beth, and Cristina—who use data to make decisions in their daily work as educators. You can follow their stories throughout the five-module sequence. Although they each have different roles as educators, administrators, and researchers, we hope you will find at least one case study that resonates with your experience. You may opt to focus on one case study throughout the modules or consider all three. The situations in which these educators find themselves are meant to provide you with concrete examples of what the data-driven decision-making process looks like in the real world.
To continue Module 1: Asking Questions, click on Alex in the right-hand navigation.
At the Classroom Level: Alex
Alex sighed as he finished grading the last of the final exams in his 11th-grade physics class. In some ways, this year was like any other; he noticed the same pattern of behavior as past years. Some students always seemed to excel, but most students seemed to struggle with the challenging material, which required considerably more thought and care than the science classes his students had taken in middle school. Reading through the exams confirmed his impressions from quizzes, homework assignments, and tests throughout the semester. Students could recall facts and equations with ease—the definitions of Newton’s Laws, for example—but struggled when it came to actually applying the equations in authentic situations. Higher-order thinking was a challenge for his students.
Seeing the same struggle in his students’ exams this year was a particular disappointment, as it came nearly one year to the day after a similar final exam grading session when Alex first noticed the problem as an educational issue to be addressed. In an effort to improve students’ skills, over the past year he had tried out some of the new pedagogical approaches he had learned in professional development sessions. Now it was clear that none of these approaches resulted in any improvement.
As he entered the last of the grades into his grade book, Alex realized he had approached the problem completely backwards by attempting to solve it before he had really identified its root cause. He decided that this time, instead of starting with a solution, he would begin with a question to try to identify the source of the problem.
Last year Alex asked himself:
- How can I improve my teaching to help students perform better on the final exam?
This year Alex could potentially find more useful information by asking:
- What factors are leading to students’ poor problem-solving skills in physics?
Once those factors are identified, a better solution may become apparent.
To continue Module 1: Asking Questions, click on Beth in the right-hand navigation.
At the Department Level: Beth
At Riverside Community College, Beth is chair of the English department. To Beth, it seemed like the same debate popped up in every faculty meeting. A faculty member teaching a literature course would inevitably make an offhand remark about the wildly uneven writing skills of their students, another faculty member would concur, and before Beth could bring the meeting back to order the discussion was sidetracked for good. With a pattern of discussions like this emerging, Beth became especially frustrated because there was rarely any progress. There were many theories about the cause of the problem: differences in high school preparation, differences in their first-year composition courses, lack of effort by some students, and cultural differences. However, no one had any data to back up their assertions, so the conversation never developed beyond this stage.
Walking back to her office, Beth resolved to try to use data in a more systematic way.
The group of faculty members seemed to focus on asking:
- Why are students’ writing skills so uneven?
However, Beth decided to develop a stronger question and ask instead:
- What factors contribute to students’ writing skills in their literature courses?
She placed a discussion of student writing skills on the agenda for next month’s meeting. With her new question in mind, Beth planned to investigate possible antecedents that influence students’ writing skills, hoping those data would help drive a productive discussion about what concrete steps they could take as a department to address the issue.
To continue Module 1: Asking Questions, click on Cristina in the right-hand navigation.
At the Institutional Level: Cristina
As an administrator at a four-year state university, Cristina found she was having more and more conversations with colleagues about degree attainment in the STEM disciplines (science, technology, engineering, and mathematics). After each conversation, she couldn’t help but notice a pattern. It seemed the demand for STEM disciplines in the workforce was rising much more quickly than the rate of students electing to study and eventually completing degrees in those fields, which could become a problem if left unaddressed.
One day during a discussion of the issue with a colleague, it struck Cristina that STEM degree attainment was part of many informal conversations administrators were having with colleagues at peer institutions. Cristina realized that to address the issue they had to be much more systematic in their inquiry.
Cristina and her colleagues did not have a concrete understanding of what was driving degree attainment in STEM fields at their home institution or at peer institutions, and many seemed to wonder:
- Why does our institution seem to be ineffective at encouraging and supporting students in STEM disciplines?
Cristina struggled with knowing where to start. She knew that a large percentage of students earned grades of C or lower in the gateway STEM courses, but she was not sure how to investigate this issue. Cristina decided that she might be able to gain insight by evaluating degree attainment in STEM fields at peer institutions, so she framed her question as follows:
- What trends in STEM attainment exist at our institution, and how do those trends compare to the trends at our peer institutions?
Cristina wasn’t completely satisfied that this was her ultimate question, but she felt like it would lead her in the right direction by informing her understanding of current degree attainment trends.
To continue Module 1: Asking Questions, click on Scanning Your Environment in the right-hand navigation.
Scanning Your Environment
As a teacher or administrator, you ask questions and observe problems on a regular basis. You may note that your students are not performing well on recent tests, but you are unsure where the gaps in learning occur. You may hear anecdotes, but you don’t know how to verify what you’re hearing or how you can best respond.
To become a data-driven decision maker, you must first identify patterns and focus on the actual problem. You do this informally all day long in your classroom or office, so how do you take these informal observations and identify the root problem? Once we can phrase these observations as a problem statement, we can specify the question and set out to answer it.
Observing the patterns around you, in data terms, is called an environmental scan. This is a great way to begin to define the problem. Watch this video to hear more about environmental scans from Dr. Cassandre Alvarado.
Here’s an example of the kind of information each of our stakeholders noted while performing their environmental scans.
At the classroom level: Alex
- Low student performance on exam application problems; similar trends year after year
- Lack of improvement since implementing new pedagogical strategies
At the department level: Beth
- Concerns repeatedly raised by faculty members about students’ writing skills
- Numerous theories but no agreement or progress
At the institutional level: Cristina
- Concerns about STEM degree attainment rates
- Poor student performance in STEM gateway courses
- Lack of comparative data on attainment rates at home institution vs. peer institutions
To continue Module 1: Asking Questions, click on Video 1.1 in the right-hand navigation.
Once you have made some observations about your environment, you will need to develop a sound problem statement to begin appropriately addressing the issue you deem most important. You will define the scope of your problem statement, including what the issue is, whom it affects, and how it affects them, and write the statement in 10-12 words. Another way to think about the problem statement is: If you wrote a paper about the problem you are facing, what would the title be?
After you have written a concise problem statement, you will then begin formulating a strong question, keeping in mind that what you look for is what you find. In other words, a question that is overly broad will likely result in a mountain of data that may not be completely germane to your problem. On the other hand, a question that is too narrow will likely lead to collecting insufficient data that will not address the entire scope of the problem.
Narrowing and reframing the problem statement into a specific, measurable question is similar to the earlier process of taking a large, complex problem from an environmental scan and distilling it into a short, direct problem statement.
To continue Module 1: Asking Questions, click on Activity 1.2 in the right-hand navigation.
Informing the Decision
Although you make decisions every day and are working to make data-driven decisions, you may find yourself in situations where you are not the decision maker. Instead, you may be the one who can inform the decision with the data or information you have. Cristina finds herself in that situation. In her position at the institutional level, she is working to bring the issue of STEM degree attainment, in all its complexity, into discussion and to provide valuable data to institutional leaders who can affect STEM education in their planning and decision-making processes. It is important for you to be aware of your position in the decision-making process and to determine the best way you can use that position, your knowledge, and your access to information to advise that process.
To continue Module 1: Asking Questions, click on Conclusion & Review in the right-hand navigation.
Conclusion & Review
Module 1 led you through identifying a problem, formulating a problem statement, and refining your question as the first phase in any data-driven decision-making process. Alex, Beth, and Cristina’s case studies show how this phase may begin at the classroom, department, or institutional level. Whether you are a classroom teacher, an administrator, or hold multiple roles, you will find your environment problem-rich. We say “problem-rich” because we see educational problems, regardless of context, as opportunities for improvement, rather than impediments. In Module 2, we will address finding existing data to inform your question.
To continue Module 1: Asking Questions, click on Review Questions in the right-hand navigation.
- What is an environmental scan, and how can it be used?
- What are three educational issues you have observed and would like to explore?
- Can you narrow the scope of one of these issues by developing a succinct problem statement followed by a specific question?
- What are the characteristics of a good question?
To view supplemental materials for Module 1: Asking Questions, click on Supplemental Materials in the right-hand navigation.
To move on to Module 2: Finding Existing Data, click on Finding Existing Data in the left-hand navigation.