Jackie Dietz – Nuts and Bolts of Classroom Assessment
1. What is your process of creating an exam? That is, how do you decide on the topics to cover, types of questions, number of questions, use of technology, use of external aids, re-use of questions, etc.?
My first step in writing an exam is to make a list of the topics covered during the relevant time period. In the past, I just jotted down this list for my own use and didn’t share it with the students. But during this past year I’ve created what I call a Study Guide for the students, which I post on Blackboard a week or so before the exam. (Even earlier would be better, if I could get more organized!) This is a bulleted list of things the students should be able to do by the time of the exam. (“You should know how to do the following: Interpret the slope and intercept of the least-squares line in the context of a particular dataset. Distinguish between observational studies and experiments. Explain why it is important for experiments to compare two or more treatments.”) I try to think through what I’d really like the students to be able to do, and I try to avoid bullets that start with “Understand,” or “Know how to,” etc., in favor of more active verbs. Once I’ve posted the Study Guide, I use it to guide my choice of questions for the exam. I avoid putting anything on the exam that is not included in the Study Guide, and I try to include at least something on the exam from each major topic we’ve covered. Sometimes I jot down the points that I’ve allotted to different topics as I write the exam, to make sure that I don’t overemphasize a particular area.
For better or worse, the type of questions I ask depends somewhat on the size and ability of the class, because of the reality of grading. Last semester I taught Statistics I to a class of six honors students, and my exam questions were more open-ended and more challenging than in other sections of Stat I that I’ve taught. I also experimented in that class with having the students use Fathom on their laptops during exams. For larger classes, I spend more time up front trying to write questions that will be manageable to grade. I do have some multiple choice questions and other short answer questions that I like to use, but I never write an exam that is all multiple choice. I try to have a variety of different types of questions, and I try very hard to use real, or at least realistic, data in my questions. Aside from the one honors course, I haven’t used technology during my exams (other than calculators for simple arithmetic). In certain courses, I base exam questions on computer output.
Over the years, I’ve tried everything from open-book exams to exams with no resources allowed. I’ve usually been unhappy with open-book, open-note exams, because many students waste too much time frantically thumbing through their materials during the exam. I think my favorite option is to have students prepare one or more note sheets (depending on the material) to bring to the exam. I like to think that there’s some pedagogical value in making the sheets and that they help alleviate some anxiety. Another alternative I’ve tried is to have students bring the orange formula card that comes with The Basic Practice of Statistics, which seems to work well for that book.
I’ve had a problem over the years with writing exams that are too long for the students. I’ve been trying to convince myself that I don’t need to include a question on every topic – that a sampling of topics will do! I must be making process because I haven’t had any complaints about this during the past year.
Re-use of questions is an interesting issue. At NC State especially, I was aware that some students had access to files of old tests, so I was a bit hesitant to re-use questions. But coming up with all new questions every time is too time-consuming and encourages ever more obscure exams. I have some favorite exam questions that I use over and over, and I just hope that some students will not be advantaged over others. But now I’m all set, because I changed jobs a year ago and can now re-use old questions with abandon!
2. How do you grade exams? How much feedback do you give? How do you decide on partial credit? How much time do you spend grading? Any tips to reduce grading time?
I spend a huge amount of time grading exams. I grade one question at a time through the entire stack of papers. For a question that involves partial credit, I often sort the papers into piles with similar answers. I’ll first deal with the easy piles – those with essentially correct answers and those with totally wrong answers. Then I tackle the hard ones, trying very hard to be consistent and, if possible, to use the entire range of possible scores. My exam questions are usually worth a fairly small number of points – maybe one to six, depending on the complexity of the question. If I see that most students are having a tough time with a particular question, I may be more generous with the partial credit than I would be if most students were doing well on the question. I never used to give half points, but I’ve decided that sometimes giving half points makes the grading easier without really causing any problem.
I reduce the amount of feedback required on the exam papers by making a detailed solution that I post on Blackboard. Making the solution is time-consuming too, but it still seems less onerous than writing the same comments over and over. Then I can write “See solution” on an exam instead of writing a detailed comment. I put a lot of red ink on the exams as I grade them. I’ve gotten in the habit of putting a check mark next to each correct explanation or calculation – I guess so that students will know that I actually read what they write! I also feel compelled to justify somehow on the paper any deduction of points, even if only with “See solution.”
One time saver is to have students write their answers on the exam paper itself, in spaces that I provide, instead of having students answer on their own paper. I go through more paper that way, but having all the answers to a given question in the same position on the page makes it quicker to move through the stack of papers. In addition, this gives the students some guidance about how long their answers should be.
Grading AP Statistics exams has had an influence on my grading. One useful idea is the distinction between a parallel solution and extraneous material. On the AP exam, if a student provides two alternative solutions to a question, we score the weaker solution. I’m not usually quite that tough on my students, but the notion of a parallel solution does help me deal with the student who offers me multiple answers to choose from!
When I first starting teaching, I had trouble with the notion that a student could shoot him or herself in the foot by adding additional incorrect stuff to an answer. I’m now comfortable with the idea that a student who writes “Answer A” could get more points than a student who writes “Answer A” plus “Additional Wrong Stuff B.”
3. Besides exams/quizzes, what types of assessment do you use in your courses? What is your method of creating and grading these assessments (e.g., learning goals, expectations)?
This varies quite a bit from course to course. At NC State, I taught large classes, but I always had a graduate teaching assistant who graded homework. As a result, I always gave quite a lot of graded homework. For each assignment, I prepared a detailed homework solution that I provided to the grader and to the students. At Meredith College, I have smaller classes, but I grade my own homework. In my larger classes, which have 25 to 30 students, I assign “practice problems” that I do not collect and a smaller number of problems that I do collect and grade. In my smaller classes, like my honors class of six students, I collect and grade more homework because it is feasible to do so. I always post a written solution to the graded problems, and I try to select practice problems that have answers in the back of the textbook. I usually use problems from my textbook for homework assignments, but for some courses I’ve supplemented the textbook problems with problems I’ve written myself.
In recent years, I’ve tried to use a greater variety of types of assessment. This past year, I included two projects in most of my classes. These were individual projects and most involved analyzing data that the students found in EESEE or DASL or the JSE Data Archive. The project instructions included a detailed description of how I would grade the projects (Dataset and questions of interest are described clearly – 3 points; The slope and intercept of the linear model are estimated from the data, and the fitted line is drawn on a scatterplot – 3 points; The slope and intercept estimates are interpreted clearly in the context of the problem – 3 points; etc.). Providing this information about grading in advance has worked really well. I agonize less over giving low grades to students who do a minimal job because I know I have communicated my expectations clearly. The “rubric” also makes it easier to grade the projects.
At NC State, I taught a course on statistical thinking using Moore’s Concepts and Controversies. In that course I used an Article Collection Assignment in which students assembled an annotated collection of 16 articles from newspapers and magazines that related to specified course topics. Students knew in advance that each article was worth four points – two for finding an appropriate article on the assigned topic and two for a requested calculation or explanation. On many occasions, I saw students who struggled with my exams do an especially good job on the Article Collection Assignment. This assignment seems to tap different abilities and interests than some of my other assignments, which I think makes it a particularly valuable addition. (I described this assignment in the Spotlight Session on Pedagogy at USCOTS; it is included in the conference Resource Notebook.)
My honors section of Stat I last semester was designated as writing intensive, so I incorporated some writing assignments into the course. (Starting in the fall, all of our Stat I classes will be writing intensive.) I’m grateful to Joy Jordan for providing the following two assignments! In the first, the students wrote a “letter to Dad” explaining the randomized, double-blind experiment that was carried out on Grandma’s blood pressure medication. The second type of assignment was to write a “meaningful paragraph” – a coherent piece of writing that uses all of the words in a given list. My students wrote two meaningful paragraphs – one on sampling distributions and one on confidence intervals. For each of these assignments, the students were given a grading rubric in advance that described how points would be allotted to different aspects of the assignment (The explanation to your Dad convinces me that you understand the statistical concepts involved in the assignment; The explanation to your Dad is presented in non-technical terms that he will understand; etc.). The meaningful paragraph assignments were a real eye-opener for me. My six honors students were excellent students who had little difficulty with other course assignments, but some of them really struggled with these paragraphs. This assignment really revealed where confusion lay in a way that other assignments did not. This was painful, but valuable, for me to see, and I will use these assignments again. With six students, I was able to provide feedback on multiple drafts and opportunities for revision. In the fall, I’ll have a total of 50 students in Stat I, so I’ll need to figure out ways to streamline the process.
4. How much weight (toward the course grade) is each assessment piece worth in your class? Do you use any classroom assessment techniques that are not graded?
The weights have varied a lot for me over time and over different courses. At NC State, where I often taught graduate courses and where I graded more homework, I often counted homework for 25% of the grade, midterm exams for 50%, and the final exam for 25%. The number of midterm exams varied from two big tests to six small tests. Particularly in undergraduate courses, I also included a project or the Article Collection in addition to exams and homework.
At Meredith, I’ve used a greater variety of assessment methods, and I’ve assigned less graded homework. In fall 2004, for two regular sections of Stat I, I used the following grading scale: homework 10%, participation 5%, quizzes 5%, three exams 45%, two projects 15%, and final exam 20%.
I don’t have students turn in much that isn’t graded. My current “practice problems” are not turned in, and I give feedback only when students ask me questions about them. When I provided feedback on drafts of writing assignments in the writing-intensive class, those were not graded. I occasionally have students work a problem during class that is turned in but not graded, but I don’t do that very often. I like the idea of using ungraded minute papers toward the end of class, but I don’t think to do those very often.
I never included participation or attendance in my grading scheme at NC State, and I’m still trying to figure out how to do that effectively at Meredith. There’s a strong expectation at Meredith that students will attend class every day, and with small classes, it’s easier to keep track of attendance. I would like the participation grade to reflect more than just attendance, but I’m not sure what! I would appreciate suggestions on meaningful ways to quantify participation without introducing a record-keeping nightmare.
5. How do you support the students in preparation for assessment (e.g., review session, drafts, practice problems)?
I’ve mentioned most things already. Before exams, I provide the Study Guides described previously. I sometimes schedule extra review sessions before tests, particularly for larger classes where a review session is more efficient than meeting with students individually. I suggest practice problems for the students to work and then give quizzes that consist of one or two recent practice problems. I put a lot of effort into my homework and exam solutions so that those will be valuable resources for the students. Providing feedback on drafts of projects and writing assignments really does improve the final product, so ideally I’d like to do that more often. But reading drafts is time-consuming, and also the iterative process of feedback and revision adds to the total time required from beginning to end of a project.