Master's Measurement Project Report Outline

The requirements for the Measurement Project can be fulfilled by successfully completing one of the four options described below.

Option A. Construct a new assessment instrument or procedure

1. The instrument or procedure to be developed can be selected from any of the following categories:
1. An achievement test (A test intended for one-time use in a classroom setting is not acceptable. To be acceptable, the test must be good enough to warrant repeated use in assessing a reasonably stable domain that includes intended outcomes beyond the recall level.)
2. A performance test for assessing learners’ ability to do some complex task that is valued in the workaday world outside of school contexts (The resulting test should include all displays, materials, equipment, and instructions necessary for actually administering the test.)
3. A diagnostic test of learners’ understanding/misunderstanding of the concepts, principles, rules, or procedures included in some domain that is a prerequisite for successful performance or further study in the subject.
4. A scale, inventory, or other instrument assessing one or more affective characteristics (attitudes, values, interests, anxieties, locus of control, academic self-esteem, etc.) related to some instructional experience or educational issue.
2. In addition to the instrument or procedures developed, the following ancillary products must be included to fulfill this option:
1. The set of written specifications used to guide the development of the instrument or procedure.
2. Evidence of an adequate review of relevant research, theory, or subject matter content.
3. Evidence of formative tryout and revision including item analysis data, reliability estimates, and at least content evidence of validity.
4. A User’s Manual including procedures for administering the instrument or procedure, scoring procedures and criteria, and guidelines for interpreting the resulting scores.

Option B. Conduct a validation study of an existing assessment instrument

1. Prepare a written summary of any previously conducted reliability and validation studies. Include an assessment of the strengths and weaknesses of these studies, and identify validity issues which have not been adequately addressed.
2. Design and conduct a study to collect additional evidence of the degree to which scores obtained from the instrument in question possess or lack validity. (Remember that establishing validity is a cumulative, ongoing process and that no single study is likely to provide conclusive answers to all validity issues. The goal here is to make substantive contribution to this ongoing process.)

Option C. Conduct a study focused on a practical measurement need or problem faced by a test-user

1. The study could include one of the following:
1. A standard setting study to empirically determine what cut-off scores should be used to make mastery-nonmastery, pass-fail, or admit-deny kinds of classification decisions based on test scores.
2. A study to detect test items that may be biased (differential item functioning) against some definable subgroup of the examinee population.
3. An equating study designed to equate scores from one test with scores from a different, but similar measure (e.g. as the ACT and the SAT).
4. A norming study that will provide relevant and representative norms for interpreting scores from a newly-created instrument or updated norms for an existing instrument.
5. Estimate the relative magnitude of different sources of error in supervising teachers’ ratings of student teachers.
6. Some other similar problem or issue approved by your project sponsor.
2. Prepare a written report for the client which describes the procedures used, presents the results, discusses the limitations of the study, and offers specific recommendations based on the results.

Option D. Conduct research on an unresolved methodological issue

1. Select some unresolved methodological issue related to educational measurement and conduct a research study designed to help resolve this issue. Some possible problems or issues include--
1. Factors influencing the reliability of scores obtained from performance tests
2. The cost-effectiveness and feasibility of using “authentic” tests
3. Problems associated with the use of partial-credit scoring
4. Problems of using formula scoring with multiple-choice tests
5. Problems of using complex multiple-choice (Type K) items
6. Effects of using different approaches to standard setting
7. Effects of violating unidimensionality in applications of item response theory
8. Effects of ‘fat’ versus ‘slim’ matching in Mantel-Haenszel studies of differential item functioning
9. Problems associated with the use of technology in assessing student knowledge or performance
2. Prepare a report that defines the purpose of the study, summarizes relevant previous research, describes the method used, presents the results, and interprets the findings in light of previous research and present practice and theory. The report should be suitable for submission to one of the following journals or an equivalent:
1. Journal of Educational Measurement
2. Applied Measurement in Education
3. Applied Psychological Measurement
4. Educational and Psychological Measurement

Written Report

Regardless of which option is selected, a written report must be prepared. This document should include at least the following information:

1. The purpose of the project and the problem, questions, hypotheses, or issues it was designed to address.
2. A rationale or justification explaining the need the study was intended to fill or the reasons why it needed to be done.
3. A critical, integrative review of relevant research literature.
4. The method used to collect data including a description of the design, sample, instrumentation, and procedures used.
5. The results of the study including any products created and statistical findings.
6. Conclusions or recommendations accompanied by a discussion of the implications of the findings for instructional or measurement theory and practice.
7. A critique specifying the strengths and weaknesses of the study in light of existing theory in the field and describing what you learned from the experience and how you would improve the project if you were to do it again.
8. The proposed schedule submitted with the prospectus, an actual schedule describing how the study was actually conducted, and an explanation for any discrepancies between the proposed and actual schedule.
9. A project budget comparing expenditures and actual expenditures plus an explanation of discrepancies.
   

If you choose option A, a copy of all instruments and ancillary products should be included as appendices to your report. Data summarizing the results of tryouts and revisions should be included either as tables in the body of the report or in the appendices.

If you choose option C or D, the report you prepared in Step 5 will substitute for items 1-6 in the above list, but must be accompanied by a written critique incorporating items 7-9 from the above list.

Close this window to return to Master's Project/Thesis Requirements.