Teaching How to Write about Multivariate Analysis : Suggested Courses and Exercises

Writing effectively about multivariate analysis involves a range of skills, including those that are typically taught in graduate courses or seminars about expository writing, multivariate regression, and research methods. This paper discusses how to integrate teaching of these skills into a graduate sociology program, including suggestions for courses and informal teaching settings into which the material can be incorporated, tested classroom teaching approaches, and exercises to provide practice applying those skills to course material or ongoing research projects. INTRODUCTION Writing effectively about multivariate analysis involves a range of skills, including those that are typically taught in graduate courses or seminars about expository writing, multivariate regression, and research methods. In a related piece (Miller et al., 2009), we outlined a series of important issues to consider when writing papers involving a multivariate regression analysis, and provided a series of guidelines and examples for avoiding common pitfalls in that type of writing. This paper discusses how to integrate teaching of these skills into a graduate sociology program, including suggestions for courses and informal teaching settings into which the material can be incorporated, tested classroom teaching approaches, and exercises to provide practice applying those skills to course material or ongoing research projects. WRITING ABOUT MULTIVARIATE ANALYSIS IN A GRADUATE CURRICULUM Writing about multivariate analysis can be integrated into several types of courses and informal learning settings as part of a graduate curriculum. Formal graduate courses In a multivariate regression course, assign students short, self-contained exercises on how to report and interpret coefficients in simple sentence form. These tasks can be added to existing data analysis assignments. These exercises help get students in the habit of thinking about how to communicate clearly about multivariate results, and can also help reinforce the substantive meaning of the statistical concepts. In research methods courses, ask students to identify the purpose of a multivariate model for different topics and data (e.g., to test for confounding or mediating). These exercises help students reinforce their understanding of the reason for a particular model specification in the context of a given research question and data set. In both research methods courses and substantive courses such as sociology of the family, medical sociology, or criminology, ask students to evaluate and critique articles that use multivariate regression to analyze data from different types of study designs, assessing whether causal arguments are plausible given the data and methods at hand. In substantive courses, require students to write either the pre-results or results/conclusion part of a paper, with multiple drafts expected. This type of assignment forces students to think though how methodological and statistical issues affect the analytic plan designed to answer an underlying substantive question. In writing courses, assign students to decide among tables, graphs, and prose for specific tasks related to a paper they are writing about an application of multivariate analysis, and then to draft those materials according to guidelines provided. These tasks help students understand the strengths and weaknesses of each of the tools for presenting numbers, and to become proficient at designing effective versions of graphs and tables and writing the associated prose. In advanced graduate courses such as professional development seminars, ask students to revisit these concepts and skills while writing or revising their own papers, to provide practice identifying and explaining the big picture of how a multivariate model fits their topic and data. Such courses permit time to consider how writing about multivariate analysis fits into each section of a research paper, and an opportunity to apply standard expository writing techniques to this specific type of academic writing. Informal settings Informal teaching settings can also provide opportunities for teaching and reinforcing issues related to effective design, execution, and communication of multivariate analyses applied to sociological research questions. Such settings include brown bag or other research seminars, collaborative research involving graduate students on teams with faculty or post-doctoral or other graduate students, and individual meetings with faculty supervising courseor qualifying papers or doctoral dissertations. Seminars. Brown bag seminars provide opportunities for students to hear how experienced researchers plan, conduct, communicate, and critique multivariate analyses. Aspects of research thinking and writing that are often discussed in these settings include: Research methods issues that affect multivariate analyses, such as whether the choice of data set, sampling strategy, and analytic sample were appropriate for the research question; and whether the variables used in the model are adequate measures of the underlying concepts. Statistical issues such as suitability of the type of model to the variables involved; whether key variables were omitted from the analysis; and the choice of analytic strategy to address the particular research question (e.g., nested models to test for mediating effects). Substantive issues such as whether the research question is interesting and important to sociologists; whether results are credible (and if not, why not); whether there are alternative explanations for the findings; and whether the limitations of the data were taken into account in the interpretation of results. Communications issues such as clarity of graphs, tables, and prose used to convey the purpose, results and implications of the study; how experienced researchers organize tables to convey their analytic plan and statistical findings; and how they read (and interpret) the information in those tables to answer the underlying research question. Early in each academic year, it is useful to orient graduate students about how to listen to and participate in seminar experiences so that they don’t simply focus on the substantive conclusions. Point out the importance of also attending to the kinds of issues raised by members of the audience, how the presenter answers those questions; and how the feedback received by the presenter might affect his or her work on the topic. Encourage students to develop and ask questions during the seminar. Suggest that they think about which aspects of the presentation were effective (e.g., clear explanation of why a multivariate model for this topic and data) and which perhaps should be avoided in their own future written and oral research presentations (e.g., too many digits on the slides; overly technical descriptions of the results that obscure substantive meaning). Collaborative research settings. Meetings and written feedback on drafts of multivariate research projects by a team of researchers can also provide insight into many of the issues outlined above, while bringing the advantage of the student being an active participant in that research who is familiar with the topic and data. Hearing how a group of collaborators each contributes their own perspective, how they complement one another’s strengths, critique one another, and resolve conflicts about issues that arise during the project also provide valuable opportunities for students to learn about the wide range of tasks and skills involved in writing about multivariate research. Individual research projects. Work on individual research papers or dissertations also allows graduate students to receive feedback on their thinking process, analytic approach, interpretation, and writing related to applications of multivariate analysis to sociological research questions. The individualized nature of these meetings is invaluable for ensuring that each student receives guidance in the particular issues they are facing in their own projects at the moment and how the principles for writing about multivariate analysis apply to their specific project. Such feedback is critical to fostering their development into researchers who are capable of conducting well-conceived and executed independent research projects, and can anticipate and address the issues that are central to multivariate research projects. METHODS FOR TEACHING HOW TO WRITE ABOUT MULTIVARIATE ANALYSIS There are several steps to teaching how to write about multivariate analysis in graduate coursework or for dissertation writers. First, assign readings that cover key principles about statistical research writing, such as Miller (2005), Treiman (2009), or other books or articles on writing or professional research practice. Second, in lecture, briefly cover the principles and associated skills for writing about multivariate analysis, followed by in-class demonstration using such as the “poor/better/best” technique (shown below) to show students examples of how to translate abstract writing principles into concrete sentences or paragraphs; see Miller (2005) or Miller, England, Treiman and Wu (2009). Third, reinforce those concepts by assigning students to apply them to their own work or to evaluating existing published work, using one of several types of exercises, such as those shown below. Fourth, have the students use checklists such as those at the end of each chapter in (Miller, 2005) to plan and evaluate their work. Poor/better/best in the classroom The “poor/better/best” (“PBB”) approach can be used to give practice applying new principles about how to write about some aspect of a multivariate analysis, encouraging students to draft, evaluate, and revise sentences, tables or charts as a participatory exercise. Here is a suggested sequence of steps for applying poor/better/best in a classroom setting. 1. Introduce a general principle such as generalizing a pattern based on many