SC 130 Physical Science fall 2007 term assessment report

SC 130 Physical Science fall 2007 featured a completely redesigned curriculum. The curriculum redesign focused the course on weekly laboratories that sought to incorporate mathematics, specifically linear relationships, and writing skills. The curriculum narrowed the content coverage substantially, providing a deeper, more exploratory approach to physical science.

Physical science tends to be traditionally been taught in a "thousand facts in forty lectures" approach. The text books for physical science lend themselves to this approach. Physical science laboratories tend to be "fill in the blank" recipes of a "do this, then do this, write down what happened: __________" format. I saw an opportunity to integrate writing deeply into the laboratory curriculum, focus on linear relationships, and require more analytical thinking on the part of the students than a typical "fill-in-the-blank" laboratory requires.

The course design was also influenced by a number of other factors. I often chose simple systems using materials familiar to the students, sometimes using those materials in new and unexpected ways. I tended to avoid complex technologies or unique materials as I felt the students would attribute whatever was happening in the system to the "magic" of the black box that technology represents, or a unique feature of the materials used in the laboratory. I am keenly aware that students can hold two contradictory world views. One view is the "scientific lenses" they put on when in the science laboratory, the other is their own personal view of how the world outside the laboratory works. By using materials from the later, I hoped to show that what happens in the laboratory is not unique to the laboratory.

I have a sense that our students are uniquely oriented to seeing the behavior of the world as changing according to specific location. There are behaviors and activities in a nahs that change outside a nahs. Why wouldn't science work like that as well? Why couldn't physical behaviors change according to the location in which they occur? While the concept that the behavior of the physical world is location dependent may initially sound naive, even within science the theory of general relativity postulates a warped space-time fabric in which behaviors are location and velocity dependent. The idea that the laboratory is a "magic" location is not as far fetched as one might think. During one laboratory a number of students attributed a particular observed behavior to "magic."

Another influencing factor was the intent that some of the laboratories should be easily and cheaply reproducible in an elementary or secondary school setting. At the college, biology and chemistry are clearly of critical importance to the health career opportunities students. Marine biology was developed primarily to serve the marine science students. These courses are unavoidably influenced by serving these constituents. Physical science has no constituency. On the contrary, it is the one course that is almost never taken by students going on in the natural sciences. Physics is the course for natural science majors such as health careers and marine science. Physical science is a grab bag of mechanics, thermodynamics, sound, light, electricity, magnetism, geology, astronomy, chemistry,and nuclear physics. About the only subjects not in physical science are biology and poetry, and I found a way to include the later in laboratory eight.

With no clear constituency, and a broad scope, the course is a logical place to put those activities that will well underpin a later science methods course. To this end many activities involved almost trivial levels of equipment. To the maximum extent possible, equipment was acquired on island. Ace Hardware was the "Carolina" and True Value was the "Wards" of the course. Given either a well-stocked hardware store or an old rail yard, physical science laboratories can be supported.

At the same time, a few laboratories featured technology such as an exploration of the colors of light using computer monitors, the use of Google Earth in conjunction with global positioning satellite receivers, and the display of sound waves using an oscilloscope.

As with all of my courses, all materials produced were made available on line. Thus students who do go on to become teachers, if provided Internet connectivity, can access these on line materials and laboratories for their own use.

Another influencing factor was my desire that the laboratories should be, if possible, fun. Learning requires motivation. Simply telling the students that they need to learn XYZ does nothing to ignite motivation. I wanted the students to enjoy the laboratories, associate the laboratory experience with pleasure and not frustration. Not only does this buy "mind time" for learning, it also is the only way to make a student who "hates" or dislikes science to change their mind and think about the possibility of a future career in science. Few students taking a typical physics course think, "Oh, this is fun, this is something I want to do for the rest of my life." Yet physics is fun, as are the rest of the physical sciences.

Writing

The laboratories integrated writing into the course. Each laboratory was written up using spreadsheet and word processing software. The students were usually, but not always, given a skeletal outline of what should be in their laboratory report. If the laboratory included a linear regression they had to use spreadsheet software outside of class time to create a graph and to calculate the slope and intercept. Then the student had to copy the graph and their results into a word processing program. Using the word processing program the student had to add other parts of the laboratory including a conclusion on the results found. The laboratories were marked with fairly complex rubrics that looked at data, content, data formatting, quality of their conclusion, grammar, vocabulary, cohesion, and organization. While each laboratory had a custom built rubric, there were general patterns for each rubric.

The rubrics remain a work in progress and the generic rubric has already been modified with an eye towards changes for next term.

The students had a week to produce each document. In the first two weeks of the term turn-in rates were low, and many students struggled to put together what is effectively a complex paper in a mere week. The turn-in rate improved after the first week and held at an average 83% for the rest of the term. Bear in mind that there is no writing prerequisite for the course. Nor is there any intent to invoke a writing prerequisite - the course can and should be a part of multiple opportunities for a student to acquire writing skills across the curriculum.

Mathematics

While some laboratories did not lend themselves to generating a linear regression, as many laboratories as could reasonably do so were designed to include a linear regression. Students used spreadsheet software to find the slope and intercept in these instances. Students also attempted to make determinations as to whether the system was linear. I regret to note that not enough was done to make use of the regression in regards making predictions or projections, there is room for expansion and improvement of the curriculum in this regard.

Other Focuses

In keeping with the cross-discipline nature of the course, one laboratory centered on a making an accurate sketch of a cloud type. Observation and the accurate recording of the natural world in images is one of the oldest parts of science. Scientists of the pre-camera age had to have a gift for art and drawing, and a keen eye for detail. The sketch provoked an interesting reaction from the students, many noting that they could not draw well. Yet the laboratory gave some who found the mathematics and science facts difficult a chance to shine and do well. My hope was that the assignment would give some who did not usually do well to experience success, and this did occur.

Assessments

With the focus on the laboratories and on the explorations of specific concepts in physical science, a multiple choice final that featured a hundred factoids seemed wholly inappropriate. The core to the assessment of the students had been the laboratories along with weekly quizzes and tests that also tended to center on the laboratory experience. By the end of the term 51% of the course points were in laboratories and 44% were in weekly quizzes and tests. The remaining points were for attendance and homework.

The outline for the course has not been updated to reflect the changes that have been made in the course. Thus an analysis of student learning as specified by the outline is inappropriate. The outline will have to be rewritten, but at this point I continue to wrestle with a way to capture the learning outcomes of the course. The course continues to have a stub of the lecture component core based in the textbook and apparently somewhat aligned with the old outline. Quite frankly I had not done more than glanced at the existing outline, proceeding with my own course design. Thus I was surprised when an observer of a lecture noted that I had covered a number of specific outcomes on that outline, citing the outcomes by number. While the observer later conveyed a sense of approval at having been able to check off specific outcomes on the extant outline, I knew from experience that the students were not doing anything more than most superficial memorization of those factoids. I was happy I had met the apparent expectations of the observer, but I had not met my own expectations.

My goal is not specific factoids conveyed in a lecture format, but broader meta-outcomes such as an understanding of the underlying mathematical nature of the physical world, and an excitement in learning about that world. The course is a course undergoing an evolution, and I expect it will be something on the order of five years before I have something I feel is doing what I want it to do. I expect that the course will always have a lecture component. Students cannot discover that which took the human race the past three thousand years to sort out. We all go to presentations and lectures and gain knowledge from them, this remains an efficient way of transmitting large amounts of known facts quickly.

Despite the lack of an outline against which to run an aggregating item analysis, I found some benefit in aggregating data by action verbs and topic areas. The former, action verbs, refers to the specific action a particular quiz or test item requires of the student. Thus the question "A 97.0 gram superball traveled 70.0 cm in 0.20 seconds. What is the speed?" was placed under the "calculate" action verb because the question could be summarized as, "Calculate the speed of a ball..." One might wonder why the question did not simply say "Calculate the speed of a ball that travels 70.0 cm in 0.20 seconds," but I had once been told that good test design asks for answers using questions instead of statements. I am not consistent in this regard, and my students undoubtedly have to become accustomed to both styles of question writing.

The action verb analysis is an aggregation of a large item analysis table. A fragment of that is reproduced below.

In the above table the percentage correct is the number of students answering that particular question correctly. The above fragment displays five questions from the pretest. The course began with n = 31 students and ended with n = 29 students.

Aggregating by the action verb yields:

Students tend to do better at recalling specific definitions and fact. Students have more difficulty with mathematical predictions, calculations, and inferences from data. Thus multiple choice factoid tests would play to a strength of the students while more complex open answer inferences are more challenging. The results well agree with Bloom's taxonomy, with student performance falling as the one climbs up Bloom's taxonomy.

Although the above also includes the pretest, minus the pretest the overall aggregate performance rises only to 43%. Overall, performance across the term on quizzes and tests was weak. Tests often repeated questions on quizzes and performance rates

A similar aggregate analysis by topic yields the following results:

Underneath high performance on topic areas such as heat are a larger percentage of questions at the bottom of Bloom's taxonomy. The students are fairly capable of memorizing specific facts.

On the other end of the scale performance on precision was so dismal that after midterm I stopped marking this item. Precision refers to rounding answers to the correct number of significant digits. The students do fine as long as the number of correct significant digits is the same as keeping two decimal places. By midterm I realized that students were unable to think in terms of units and dimensional analysis, and had fundamental difficulties with simple concepts such as speed is distance divided by time. Faced with weaknesses on all sides, I triaged and tossed out a triple-headed focus on correct calculations, correct units, and correct significant digits.

I shifted to a primary focus on being able to make basic calculations and conversions between cgs (centimeter-gram-second) and mks (meter-kilogram-second) systems along with conversions among time units. By term's end only 39% of the students could take a problem phrased, "A runner runs at a steady speed for a distance of 5.1 kilometers in 31 minutes and 30 seconds. What is the speed of the runner in meters per second?" and obtain the correct answer.

Even when simplified to the non-conversion problem, "A 56.0 gram tennis ball traveled 60.0 cm in 0.30 seconds. What is the speed?" which has a cgs answer that is found from a single simple division, only 59% of the students answered correctly.

When clued by the units of the answer, performance rises. On quiz eight the question was phrased, "On the 13th of September a 5.45 gram white duck marble covered 30.0 centimeters in 0.40 seconds. What was the speed of the duck in centimeters per second?" With the unit clue student performance rose to a 72% success rate.

Final examination notes

The final examination reflected directly the laboratory focus of the course. Grading in the course was built around a points structure. As noted above, 51% of the points were in laboratories with 44% in quizzes and tests. By term's end the course had accumulated over 500 points. I wanted to preserve the high value that had been placed on the weekly laboratory reports. These represented a tremendous, sustained effort by the students. Thus the design of the final was structured directly around the laboratories. For each laboratory, a single core concept from that laboratory was tested on the final.

The course is taught in the A101 physical science laboratory. The room consists of a series of tables. There is no realistic way to keep students from seeing each others papers. As a result, a spreadsheet was used to randomly select questions from a bank of questions, one for each laboratory. The random selections were then merged into a document to produce each individual final examination.

Performance on the final examination was poor, with an average student success rate as measured by item analysis of 35%. In all fairness, the students had little guidance on what to study for the final examination. I had wrestled with a number of different options, but had not solved the technical issues of my preferred options prior to the last day of class. It was only over the weekend after the last day of class that the design was completed. In another course I teach, statistics, prior finals provide practice for students and a focus for their studies. The first few terms that statistics was taught the average student success rate on questions ran below 60%. On the most recent final the average success rate was 75%. Some of this gain is likely due to the availability of numerous prior finals for students to use for practice.

The last question on the final examination was a short essay answer question. The intent of the question had been to elicit the student's "world view" as to the nature of physical science and the physical world. I wanted to get inside their heads and see if they were perceiving a mathematical world around them. The wording, however, tripped a different response among the students.

What have you learned and what will you remember about the nature of the physical world as a result of SC 130 Physical Science?

The course did apparently help some students from a writing perspective. One student noted, "This class make me feel comfortable pf writing laboratory reports. At first I have no idea how to do a write-up, but when I get used to it, it is much easier for me. This is one kind of thing I learned and will keep it, for the purpose of other classes that I will take in the future." [sic] Another student noted that the course helped them learn to use computers to make tables and charts.

The essays also betrayed deep confusion even among students who did well at making calculations. One student who regularly performed well on calculations and memorization questions noted, "In class I learn to measure the distance of the speed." [sic] A student who did poorly throughout the term noted:

This class seems to be interesting by doing so many things in class and also outside. I learn a lot in this class but it is too late to tell. From the beginning of this semester I know nothing. When it was on its way to the end I learn many things but it is too late. [sic]

The laboratories were perceived as beneficial. "Everything that we did as lab work I will remembered because you are the first instructor those kinds of assignments." [sic] Another student noted, "I learned a lot during every lab class. If I didn't understand the lecture on each week, I always recall to my note on my lab."

Some students perceived the core message of the underlying mathematical nature of the physical world. "By taking this class I also learned that mathematics is also helpful in knowing what our nature is mostly about." Another noted, "I also learned that for every object that moves, there's an equation for it. I learned how light is bent by water, and how every objects always have a reason of why they does do the thing they do. For everything, there is always a mathematical relationship." [sic] Other students made similar statements in their essays.

Some of the essays provided unexpected evidence that the course impacted attitudes toward science. "This class has many interesting parts that really have captured my attention." Another noted, "It was fun to work with physical science and its nature though sometimes it was hard because we are not scientists, but we try to prove what they had done by doing our own following their theories. It takes a lot of thinking learning in physical science, yet it is worth taking because now we understand things behind the physical world."

A third student commented, "Taking this class, SC 130 physical science, I never thought of liking it. My week first of this class I was stress because I had no idea what were doing. I kept coming to class even though my mind was blank. Slowly I started to know what we were doing in class. ... Later on, this class became easier to me and I began to like it more." For this student the course engendered a metamorphosis from a non-specific dislike of science to liking science. We cannot, we will not, learn what we find no motive to learn. Desire is mother of learning. The class awakened in some a desire to learn science. Future scientists are not produced by government reports demanding that schools produce more scientists. Future scientists are produced by students who possess a desire to learn science.

A fourth student wrote, "Even when I'm just sitting at home watching a movie, and I hear a thunderstorm, I tell my family members that it's the Bergeron process. I even randomly look up at the sky and try to name the different clouds, although most of the time I just stand there trying to figure out what is which. I loved being in this class."

Looking ahead

The fall 2007 run of physical science was a first draft of the course. Next spring provides an opportunity to revise the course. Every single laboratory has been altered as a result of in-class experience. Some laboratories were actually revised in the brief moments between the 8:00 and 11:00 laboratory sections, and then revised yet again after the second run of the lab. The content of the course is also being looked at - the course covered no astronomy, a lack that is a personal regret for me.

Closing note

Throughout the term assessment reports have been produced, these are available on line at:
http://www.comfsm.fm/~dleeling/assessment/index.html#physci

The course home page provides access to the laboratories, quizzes, tests, and related materials at:
http://www.comfsm.fm/~dleeling/physci/ps73/index.html

Assessment