The following information repeats information shared in two emails sent in late November. This information is shared in order to provide documentation of the November study of the August 2002 run of the math placement test. The first email was sent on Sunday, November 24, 2002 at 1:46 AM. The second was sent on Monday, November 25, 2002 at 11:37 PM. Lest the times of day look odd, I do most of my work after the children have gone to bed!
Please pardon me if you have seen this information before.
I have entered data from Ray's, Yen-ti's, and Dennis' classes, my thanks for getting that to me so quickly. Once I have the rest of the MS 100 data and the MS 098 data I will add that into the pile as well.
On Friday I looked at the data Dennis provided. I found that of the 13 students placed into his MS 100 section, 10 students or 77% were passing with a D or better, nine of these with a C or better.
Placement into MS 100 required a 6 or higher in the third column of the placement test. All three students with an F attained a 6 in that column. Seven other students with a 6 in the third column are passing the course. One student with a 5 in the third column is attaining a B in MS 100.
The Pearson product-moment correlation coefficients between a student's present score and their score in a particular column are as follows for the above 13 students:
m1: 0.669 m2: 0.512 m3: 0.242 m4: 0.028 sum of m1 to m4: 0.630
The low correlation to m4 is to be expected: doing well in this column places a student into MS 101. I was somewhat surprised in the steady decline of r with increasing column number. I was doubly surprised by the rather high correlation for the sum, especially given the lower correlations in m2, m3, and m4.
These patterns were seen by me again after entering data for MS 090 (all sections), and MS 095 (one section). I now had 108 students and the following results:
m1: 0.450 m2: 0.235 m3: 0.207 m4: -0.022 sum of m1 to m4: 0.367
An analysis of success rates for placed students looks as follows:
090 abc 46 66%
090 df 24 34%
70
095 abc 22 88%
095 df 3 12%
25
100 abcd 10 77%
100 f 3 23%
13
To read the above table note that 70 new freshmen were placed into MS 090. 46 of the seventy, or 66%, passed with an A, B, or a C and were promoted to the next course. 24 students either passed with a D or failed with an F, either way they were ineligible for promotion to the next course. A study in 1996 noted that the developmental math classes in the first term for new freshmen typically see a 50% non-promotion rate or higher. A non-promotion rate of 34% is strongly better than results seen in the early to mid 1990s.
Note that only an A, B, or C is a success in MS 090 and MS 095, while A, B, C, or D are successes in MS 100. A D in MS 100 is still sufficient to allow students in majors for which MS 100 is terminal to graduate.
In general the relatively high rates of success suggest that students are placing properly. Bear in mind that I have always contended that we lose a significant number of students in their first term at the national campus due to non-academic reasons. Adjusting to life away from home, adjusting to life in a dormitory environment, adjusting to living and working with people from other cultures, dealing with homesickness, and so on and so forth.
The numbers succeeding at MS 095 suggest that students may be underplaced. Two of the three not succeeding have D's at 68% and 69% respectively. Only one student is out-and-out failing at 56%. Ten students or 45% are obtaining a grade of A.
The difficulty with pulling general rules and patterns out of the data can be seen below:
SC 1 2 3 4 Sum 97 8 5 1 1 15 96 7 4 2 1 14 95 7 3 5 1 16 95 6 4 4 1 15 95 5 2 4 2 13 94 7 5 4 1 17 93 6 5 3 2 16 90 7 5 4 1 17 90 6 3 2 3 14 89 6 5 1 4 16 84 6 3 3 4 16 82 7 5 1 4 17 81 7 3 5 4 19 81 8 5 1 4 18 80 8 3 4 3 18 78 7 4 1 1 13 78 5 1 3 3 12 77 8 5 4 2 19 77 7 5 3 3 18 77 2 3 3 0 8 75 4 4 1 2 11 72 7 5 5 2 19 69 5 2 2 1 10 68 3 3 1 1 8 56 1 2 1 2 6
SC is the students current percentage mark in MS 095. This list is in descending order from highest mark to the lowest mark. The four columns of the test and the sum of the four columns are denoted by the first row of the above table.
The numbers shown are the number correct in that column of the placement. A six or above in the first column is necessary to place into MS 095. This was shifted down from a 7 or above as a result of what appeared to be systematically low placements for students during the Spring 2002 pilot run of the test. In both the Spring 2002 and August 2002 test runs a cut-off of 6 or higher was used. The sum is not used in any way, it is reported for informational purposes.
The only pattern that become obvious is that the three failing to succeed should have placed into MS 090. Why are they in MS 095? I cannot be certain, but it worth bearing in mind that two are borderline on passing the class despite their misplacement by one course. The one clear failure is originally from Chuuk State Campus.
Predictivity on an individual is always chancy, there are two others that were put in MS 095 although they are below the cutoff. One student with a 78 would have placed into MS 900 and one student with a 75 would have placed into MS 090. Why are these two in MS 095? Again, I do not know, but good on them they are passing. Maybe some of these are early term transfers - I know Ray and Yen-ti occasionally move a misplaced student on good grounds. A student might have a low placement score but have been sick that day, an in-class quiz often reveals the misplacement and I know that Ray and Yen-ti strive to fix this during that first week.
In December I hope to move the admissions committee to shift from our current entrance test to using the placement test. I firmly believe that a measurement of math skills made in the Spring while the student is still in school will better represent a student's knowledge of math than a measurement made after an English placement exam and after a long and "rusty" summer.
By using the placement as the entrance test, we will have an idea of course loads for the coming Fall and can redeploy our resources, ie, the developmental math faculty, accordingly.
I am encouraged by the overall pattern of student success coming out of this placement test. These patterns are similar to those I saw in the mid-1990's when we used a similar placement test. I found no predictive power in the current entrance test, which is why we had to put a placement test back into our system. My hope would be that come August 2003 we would only have one or two students who still needed to take the placement test (students coming to us from Guam, the US, or Hawaii, Micronesian returnees from high schools abroad as it were).
I did run one other look at this preliminary data. I looked for gender and state differences. There were no significant gender differences. There were state differences that my preliminary back-of-the envelope calculations show to be significant at a 95% level.
While Pohnpei performed no different than average (kind of a "doh" given that they held 44% of the scores), Chuuk performed below average while Yap and Kosrae were above average. I need to check my Kosrae and Yap results further, the n is small and I do not have the tools on my computer here at home to verify my findings. For now it appears Kosrae will certainly be significant at 95%, Yap is more dubious due to the small n.
I know I must sound like I am always picking on Chuuk, but year in and year out Chuukese students underperform significantly their counterparts in the rest of the FSM. The educational process in Chuuk remains in need of the most sustenance and nurturing.
Pop Fem Male Chuuk Kosrae Pohnpei Yap
count 108 52 56 34 8 48 7
mean 74.105 75.949 72.393 66.851 86.550 76.738 83.840
stdev 16.953 17.258 16.480 17.858 9.684 15.729 7.593
I have some final results on the success rate data I presented in an earlier preliminary analysis email. There was also a correction the MS 090 data.
Freq Rel Freq
90abc 45 64%
90df 25 36%
70
95abc 33 85%
95df 6 15%
39
98abc 19 83%
98df 4 17%
23
100abcd 16 57%
100f 12 43%
28
100x abcd 10 77%
100x f 3 23%
13
100y abcd 6 40%
100y f 9 60%
15
The new data included an additional section of MS 095, MS 098 data, and two sections of MS 100. The new MS 098 data supports the earlier results that indicated the placement test is strongly predictive of success.
The new MS 098 data looks a lot like the MS 095 data and suggests the possibility that some may have been underplaced in this course.
As noted earlier, the lower success rate in MS 090 is probably reflective of the overall ill-preparedness of this group of students. These are students who cannot add or subtract, in all likelihood their English skills are also quite low and they are probably struggling in other courses as well.
The one significant change was in the MS 100 data. The new data was significantly different from the earlier data. Due to the statistically significant difference in the two sets of data I am unable to treat them as having come from the same population. As a result I present the overall results for all 28 freshmen who were placed into MS 100 this Fall followed by breakouts for the two populations.
An analysis of the correlation coefficient showed the same pattern of decreasing correlation with increasing placement test column. After much thought I am more uncertain than ever that a correlation should necessarily result on this broad a scale. If any meaning exists, it would exist only within a particular course against its placing column.
I reran the gender and state analysis with the new data and obtained the following data:
Pop Fem Male Chuuk Kosrae Pohnpei Yap Chuuk F Chuuk M count 159 74 85 45 14 73 12 20 25 mean 74.79 77.64 72.32 68.72 80.74 76.02 79.94 76.72 62.33 stdev 18.14 17.18 18.59 17.88 15.49 18.74 9.03 16.35 16.42 tc 0.95 1.97 E 2.83 low 71.96 high 77.63 diff 2.84 -2.47 -6.07 5.95 1.23 5.15 1.92 -12.46 diff% 3.8% -3.3% -8.1% 8.0% 1.6% 6.9% 3% 17% Result High Null Low High Null High Null Low
Remember this data is based only on freshmen who took the placement test in August 2002 and are now in a math class. Their present course average is used in the analysis. The female freshmen who took the placement test have a present in-class average that is higher than the overall average. Males and Pohnpei state are not distinguishable from the population average. Kosrae and Yap are above average.
In the late 1990's I occasionally ran a quirky three-dimensional analysis of state, gender, and subject at the college. I called the resulting chart a topographic map of the academic terrain at the college. Based on what I had seen before, I opted to split the Chuuk data on gender lines. The result is that female Chuukese freshmen are not distinguishable from average but the males are strongly below average.
The good news is that the placement test is resulting in a middle C average for students who went through this placement system. The odd news is that the placement test either does not appropriately place Chuukese males or there is something else that goes horribly wrong for them at the College after placement.
If the placement test is actually functioning correctly, then the Chuukese men are being put in a course at which they have a reasonable chance of success. Their lack of success suggests that something interferes with their ability to make good on their potential. Dare I suggest that we might have a culture of non-performance that has developed among this sub-population?
All errors are mine!
At this point the approval by the Admission's Board and the Curriculum committee indicates that the mathematics entrance test will use the new format.
The high school students should be notified in advance that the use of calculators is allowed and recommended on this one hour forty question test.
The existence of a practice test should also be spread far and wide at this point, shared by whatever email and hand-carried routes are available. The practice test is at http://shark.comfsm.fm/~dleeling/math/practice.xml
That web page will display correctly only in the most recent browser versions. Microsoft Internet Explorer 6.0 can display that page, but some of the fraction bars will be too big. There may be other display problems. To fix these you need a browser helper, a type of plug-in, from: http://www.dessci.com/en/products/mathplayer/
Browsers older than MSIE 5.5 cannot display the page properly even with the browser helper. Go to the Help menu in your Internet Explorer and choose About Internet Explorer to determine what version you have if the page will not display correctly.
There are no answers on the practice test. Past experience has taught me that students tend to memorize answers and then put the answers to problems worked in class on related problems on quizzes and tests. As a result, I was concerned that making answers available would simply lead to students trying to memorize the answers.
My hope is that math instructors at the high schools will work out answers for themselves insofar as they need answers.
The practice test can and should be shared with students in ways that high school instructors deem appropriate.
The practice test problem set is set at a hair breadth's higher level than the entrance test. The ability to do the practice test should ensure the ability to do the entrance test.
I should also note that printing the tests still remains problematic. The functions do not always print as they display on PostScript and PCL printers (e.g. laserjets). My experience has been that a cheap inkjet does a better job of printing the tests.
The display and printing problems are connected with my use of the MathML web language module. For further information on MathML please see: http://www.w3.org/Math/
The existence of a practice test is a first for the mathematics side of the entrance test.