I reviewed this document both as a Federal Government employee, immersed in the various standards that have been passed to encourage performance and evaluation as well as an online instructor. For example, why would the Dept of Ed be interested in sponsoring this? As many will recall from their civics classes, although the President suggests a budget, only Congress has the power to authorize the spending of our tax dollars. The Federal budget process is intended to provide an approved annual budget by October 1, but that has only happened four times since 1976. While a possible article unto itself, the purpose of looking at performance metrics is to determine whether or not federal dollars are being spent appropriately and effectively and therefore, it is good to know if a given methodology, especially an unproven one such as online teaching at that point in time, is worth the continued investment of millions of dollars.
The authors of this report reached a conclusion that “when used by itself, online learning appears to be as effective as conventional classroom instruction, but not more so”. As an online instructor, I was curious as to what practices are associated with more effective online learning. In order to better understand the best practices, I felt I needed to better understand the methodology. So I began my assessment there.
The meta-analysis described in this review relies on the use of the Q statistic which is not well explained within the article. Therefore, I leveraged guidance from UCONN, statsdirect.com, and NIH to better understand how this specific statistic is used. I will do my best to paraphrase a demanding topic based on the resources referenced above. Essentially, the Q statistic, also known as Cochran’s Q, is used to assess whether or not there is heterogeneity in the sample results. The key to understanding the heterogeneity suggests that if the Q statistic is larger, then the effect suggests greater heterogeneity and there is a greater possibility that the effects seen in the study are real. To interpret the results of this study will require that we look at those results with higher Q values to identify which practices of online learning have the greatest impact. Additionally, the number of studies used to generate the Q statistic will decrease if there are a smaller number of studies to evaluate.
While the Q statistic results are reported in Exhibit 5, the summary states that “the number of Category 3 studies concerning any single practice was insufficient to warrant a quantitative meta-analysis and the results varied to such an extent that only tentative, rather than firm, conclusions can be drawn about promising online learning practices.”
It would appear that the researchers set up a reasonable approach, but they lacked the data to either confirm or deny their proposed hypotheses. That would not be unexpected, given that online teaching was certainly a much newer technology than it is today. One might ask why they spent so much money on a series of analyses when the data seemed insufficient to support the request. Perhaps there would have been a better way to analyze the data given the extensive variability among the studies selected. It appears at the very least, that the researchers could have developed a proposal for the data that would be needed to complete an effective analysis such that prospectively developing the analytical framework would yield the data needed for the analysis in future years. As such, I did not find the discussion on online-best practices to be as rewarding as I would have liked.
The reality is that it is a tremendous challenge to do good research in any field. And if we don’t control for the variables, we’ll never be able to pinpoint which is actually a more important factor or not. This is a point that gets addressed in a number of COVID related studies that I read – and I apologize for not being able to find an example to share here – we’ll never be able to know if social distancing is more effective than wearing a mask since we haven’t done the hard and dull work of collecting the data in a systematic approach.
What this article didn’t address (and certainly would not have had the ability to test) is the access issue – are people who have access more likely to be successful than those who do not. As a country, we still have significant inequities in providing access for all students, such as those described here. And in the opening video to his Media Literacy course, John Green shares that access to the internet is still a barrier and millions of people are not able to even access on-line learning or event local media.
The challenge with analysis is that hindsight is always 20/20. In 2020, the Federal Government is certainly focusing a great deal more on data collection as a strategy as well as identifying the evidence to show positive outcomes. Perhaps now is the perfect time to design the data collection studies we need to evaluate these hypotheses further.