Development and Validation of Physics Multiple-Choice Tests on the Nature of Physics Using Rasch Modelling Analysis
Abstract
Abstract: Assessment and evaluation of student learning outcomes are crucial aspects of education. Effective assessment requires well-designed evaluation test questions, one of them is multiple-choice questions. However, the quality of the multiple-choice questions made must be high. Quality questions are those that have undergone item analysis to assess the validity and reliability of the assessment results for students during the learning process. This study employed an evaluative research methodology with a descriptive quantitative approach. The research subjects involved 251 tenth grade students from two schools in Bombana Regency, Southeast Sulawesi Province. The evaluation was conducted using an instrument consisting of 10 physics questions that were created and validated with the assistance of teachers from each school. These questions covered the topics of the nature of physics, the scientific method, and work safety. The purpose of this study is to analyze the quality of the physics questions to determine whether they meet the criteria or not. Data collection was carried out using the QUEST software and analyzed using the Rasch model. The analysis results showed that the 10 questions had INFIT MNSQ values ranging from 0.77 to 1.33, which are consistent with the Rasch model. Furthermore, the OUTFIT t values for all item questions were obtained at values less than or equal to 2.00, indicating that the questions can be considered usable and passable. The reliability estimate score for the items is 0.98, indicating that this test has high consistency and can be relied upon to accurately measure students' learning outcome.
Keywords: cognitive assessment, rasch modelling, Item response theory.
DOI: http://dx.doi.org/10.23960/jpmipa/v25i2.pp671-685
Full Text:
PDFReferences
Abate, T., & Mishore, E. (2024). Alignment analysis between teacher-made tests with the learning objectives in a selected school of central regional state of Ethiopia. Heliyon, 10(11).
Achadah, A. (2019). Evaluasi dalam pendidikan sebagai alat ukur hasil belajar [Educational evaluation as a metric for student learning outcomes]. Jurnal An-Nuha, 6(1), 91-92.
Alfarisa, F., & Purnama. (2019). Analisis butir soal ulangan akhir semester mata pelajaran ekonomi sma menggunakan rasch model [A Rasch analysis of items from the high school economics end-of-semester examination]. Jurnal Pendidikan Ekonomi, 11(2), 366-368.
Ali, Mm., Hariyati, T., Yudestia Pratiwi, M., & Afifah. (2022). Metodologi penelitian kuantitatif dan penerapannya dalam penelitian [The quantitative research paradigm and its application in empirical research]. In Education Journal, 2(2).
Amelia, R. N., Sari, A. R. P., & Astuti, S. R. D. (2021). Assessment of chemistry learning: how is the quality of the tests made by the teacher. Journal of Educational Chemistry (JEC), 3(1).
Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives: complete edition. Addison Wesley Longman, Inc.
Anh, T. T. N., & Phong, N. T. (2023). The effects of socrative-based online homework on learning outcomes in vietnam: a case study. International Journal of Interactive Mobile Technologies, 17(5), 182–199.
Araujo, I., Espinosa, T., Miller, K., & Mazur, E. (2021). Innovation in the teaching of introductory physics in higher education: the Applied Physics 50 course at Harvard University. Revista Brasileira de Ensino de Fisica, 4(3), 1–18.
Ashraf, Z. A., & Author, C. (2020). Classical and modern methods in item analysis of test tools assistant professor of clinical psychology, IMHANS, Kozhikode. International Journal of Research and Review, 7(5), 5.
Baker, F. B., & Kim, S. H. (2004). Item response theory: Parameter estimation techniques. CRC press.
Benly, M., Kartowagiran, B., Sukariasih, L., & Fayanto, S. (2020). Application of cooperative learning type group investigation to improve physics learning outcomes in vocational schools. Universal Journal of Educational Research, 8(10), 4618–4627.
Bond, T. G., & Fox, C. M. (2007). Applying the rasch model: fundamental measurement in the human sciences (second edition). Hong Kong: University of Toledo.
Chweu, E. M., Mnisi, S., & Mji, A. (2024). A Curricular framework for discipline-specific acquisition, teaching, and assessment of values in higher education. International Journal of Assessment and Evaluation, 31(2), 65–84.
D Mardapi. (2008). Teknik penyusunan instrumen tes dan nontes. Mitra Cendikia Press.
Fadlilah, N., Sulisworo, D., & Maruto, G. (2020). The effectiveness of a video-based laboratory on discovery learning to enhance learning outcomes. Universal Journal of Educational Research, 8(8), 3648–3654.
Fischer, J., Bearman, M., Boud, D., & Tai, J. (2024). How does assessment drive learning? A focus on students’ development of evaluative judgement. Assessment and Evaluation in Higher Education, 49(2), 233–245.
Ghazali, N. H. M. (2016). A Reliability and validity of an instrument to evaluate the school based assessment system: a pilot study. International journal of evaluation and research in education, 5(2), 148-157.
Glamočić, D. S., & Mešić, V. (2022). A Rasch modeling approach to analyzing students’ incorrect answers on multiple-choice questions: an example from wave optics. Metodički Ogledi, 29(1), 217-240.
Gleason, P. M., Harris, J., Sheean, P. M., Boushey, C. J., & Bruemmer. (2010). Publishing nutrition research: validity, reliability, and diagnostic test assessment in nutrition-related research. Journal of the American Dietetic Association, 110(3), 409–419.
Goodyear, P., Carvalho, L., & Yeoman, P. (2021). Activity-Centred analysis and design (ACAD): Core purposes, distinctive qualities and current developments. Educational Technology Research and Development, 69(2), 445–464.
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Sage Publications.
Hanna, W. F., & Retnawati, H. (2022). Analisis kualitas butir soal matematika menggunakan model rasch dengan bantuan software quest [A Rasch analysis of mathematics test items using QUEST software to assess item quality]. Aksioma: Jurnal Program Studi Pendidikan Matematika, 11(4), 36-95.
Hedgeland, H., Dawkins, H., & Jordan, S. (2018). Investigating male bias in multiple choice questions: Contrasting formative and summative settings. European Journal of Physics, 39(5), 1-6.
Heru, M., & Suparno S. (2019). The development of reasoned multiple choice test in interactive physics mobile learning media (PMLM) of work and energy material to measure high school students’ HOTS. Formatif: Jurnal Ilmiah Pendidikan MIPA, 9(2), 2-5.
Hidayatullah, H., Safitri, R., & Suyanto, S. (2022). Model analisis butir instrumen tes biologi untuk penilaian akhir tahun menggunakan item response theory [An Item Response Theory-based item analysis model for biology end-of-year assessments]. Measurement In Educational Research (Meter), 2(1), 1.
Hofer, S. I., Schumacher, R., & Rubin, H. (2017). The test of basic Mechanics Conceptual Understanding (bMCU): using Rasch analysis to develop and evaluate an efficient multiple choice test on Newton’s mechanics. International journal of STEM education, 4, 1-20.
Hu, P., Li, Y., & Singh, C. (2023). Challenges in addressing student difficulties with measurement uncertainty of two-state quantum systems using a multiple-choice question sequence in online and in-person classes. European Journal of Physics, 44(1), 2-5.
Ibnu, M., Indriyani, B., Inayatullah, H., & Guntara, Y. (2019). Aplikasi Rasch Model: Pengembangan instrumen tes untuk mengukur miskonsepsi mahasiswa pada materi mekanika [The application of the Rasch model in developing a test instrument to measure student misconceptions in mechanics]. Jurnal Pendidikan FKIP, 2(1), 205–210.
Istiyono, E. (2020). Pengembangan instrumen penilaian dan analisis hasil belajar fisika dengan teori tes klasik dan modern. Yogyakarta : UNY Press
Justice, P., Marshman, E., & Singh, C. (2019). Improving student understanding of quantum mechanics underlying the Stern-Gerlach experiment using a research-validated multiple-choice question sequence. European Journal of Physics, 40(5), 2-3.
Khairani, A. Z., & Abd Razak, N. (2015). Modeling a multiple choice mathematics test with the Rasch model. Indian Journal of Science and Technology, 8(12), 1.
Kolomos, A., & J.E. Holgaard. (2007). Alignment of PBL and assessment. International Conference on Research in Higher Education. Honolulu : American Educational Research Association, 4(2), 1-9.
Kurniawan, A., Istiyono, E., & Daeng Naba, S. (2024). Item quality analysis of physics concept understanding test with rasch model. JIPF (Jurnal Ilmu Pendidikan Fisika), 9(3), 474-477.
Land, T. (2013). Conceptual understanding: The case of electricity and magnetism. European Journal of Science and Mathematics Education, 1(1), 13–28.
Langenfeld, T., Thomas, J., Zhu, R., & Morris, C. A. (2020). Integrating multiple sources of validity evidence for an assessment-based cognitive model. Journal of Educational Measurement, 57(2), 159–184.
Mafudi, I., & Istiyono, E. (2024). Development and validation of the relativity concept inventory test using item response theory generalized partial credit model 3PL. Jurnal Pendidikan MIPA, 25(1), 142–154.
Marambaawang, D. N., Oktoviana Bano, V., Rambu, R., & Enda, H. (2023). Analisis kualitas butir soal penilaian akhir semester gasal tahun 2021/2022 menggunakan iteman di smp negeri 1 kambera [An item analysis of the first semester final examination at SMP Negeri 1 Kambera in 2021/2022, utilizing the Iteman software to assess item quality] . Jurnal Undhari, 4(1), 233.
Matore, M. E. E. M., Maat, S. M., Affandi, H. M., & Khairani, A. Z. (2018). Assessment of psychometric properties for Raven Advanced Progressive Matrices in measuring intellectual quotient (IQ) using Rasch model. Asian Journal of Scientific Research, 11(3), 393-400.
Maulana, S., Rusilowati, A., Nugroho, S. E., & Susilaningsih, E. (2023). Implementasi rasch model dalam pengembangan instrumen tes diagnostik [The implementation of the Rasch model for the development of diagnostic test instruments]. In Prosiding Seminar Nasional Pascasarjana, 6(1), 748-757.
Meyer, J. P., & Zhu, S. (2013). Fair and equitable measurement of student learning in MOOCs: an introduction to item response theory, scale linking, and score equating. Journal Research & Practice in Assessment, 8(1).
Nisa, K., & Suprapto, N. (2023). Deteksi bias gender dan domisili menggunakan DIF (differential item functioning): analisis instrumen tes keterampilan pemecahan masalah terintegrasi etnofisika [An analysis of gender and domicile bias in an integrated ethno-physics problem-solving skills test using Differential Item Functioning]. Inovasi Pendidikan Fisika, 12(1), 30-35.
Nugraha, D. A., Cari, C., Suparmi, A., & Sunarno, W. (2019). Analysis of undergraduate student concept understanding-three-tier test: Simple harmonic motion on mass-spring system. AIP Conference Proceedings American Institute of Physics, 2(1), 1.
Osterlind, S. J. (2006). Modern measurement: Theory, principles, and applications of mental appraisal. Prentice Hall.
Ozarslan, M., & Çetin, G. (2018). Biology students' cognitive structures about basic components of living organisms. Science Education International, 29(2).
Pennington, M. C., & Black, S. L. (2010). Conceptual Understanding and Scientific Reasoning of High School Students in Physics. International Journal of Science Education, 32(5), 567–589.
Popham, W. J. (2008). Classroom assessment: What teachers need to know. Pearson.
Purnami, W., Fauzi, A., & Naingalis, M. L. P. (2023). Computational thinking skills identification among students of physics education department using Rasch model analysis. In AIP Conference Proceedings, 2751(1).
Putri, F. S., Istiyono, E., & Nurcahyanto, E. (2016). Instrumen pengembangan berpikir kritis [Critical Thinking Enhancement Instrument]. Unnes Physics Education Journal In UPEJ, 5(2).
Pratama, D. (2020). Analisis kualitas tes buatan guru melalui pendekatan item response theory (IRT) model rasch [An application of the Item Response Theory (IRT) Rasch model to analyze the quality of teacher-made tests]. Tarbawy : Jurnal Pendidikan Islam, 7(1), 61–70.
Rahman, A., Liliawati, W., & Rusdiana, D. (2023). Performance assessment with multiple intelligence differentiation to measure communication skills: application of many facet rasch model. Jurnal Pendidikan MIPA, 24(4), 932–943.
Rash, G. (1960). Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research.
Retnawati, H. (2016). Analisis kuantitatif instrumen penilaian. Yogyakarta: Pramana Publishing.
Safitri, A., & Retnawati, H. (2020). The estimation of mathematics literacy ability of junior high school students with partial credit model (pcm) scoring on quantity. Journal of Physics: Conference Series, 1581(1), 1-2.
Samila, A. P., Lastivka, G. I., & Tanasyuk, Y. V. (2019). Actual problems of computer parametric identification of the NMR and NQR spectra: A review. Journal of Nano- and Electronic Physics, 11(5), 1-2.
Setyawarno, D. (2017). Penggunaan aplikasi software iteman (item and test analysis) untuk analisis butir soal pilihan ganda berdasarkan teori tes klasik [An application of the Iteman software (Item and Test Analysis) in analyzing multiple-choice items based on the principles of Classical Test Theory]. Jurnal Ilmu Fisika dan Pembelajarannya, 1(1).
Simonetto, A. (2011). Using Structural Equation And Item Response Models To Assess Relationship Between Latent Traits. Journal of Apllied Quantitative Methods, 6(4), 44-45.
Sinta, T., Aprilia, N., Susilaningsih, E., & Priatmoko, S. (2020). Desain Instrumen Tes Pemahaman Konsep Berbasis Hot Dengan Analisis Model Rasch [The design and development of a test instrument to assess higher-order thinking skills related to conceptual understanding, with data analysis using the Rasch model]. Journal Chemistry in Education CiE, 9(2).
Stojanović, M., & Maksimović, B. (2022). Scientific concepts related to physics from the perspective of students of biology. Journal of Physics: Conference Series, 2288(1).
Sumintono, B., & Widhiarso, W. (2015). Aplikasi pemodelan rasch pada assessment pendidikan. Trim komunikata.
Suryani, Y. E. (2018). Aplikasi rasch model dalam mengevaluasi Intelligenz Structure Test (IST) [An application of the Rasch model for evaluating the Intelligenz Structure Test (IST)]. Psikohumaniora: Jurnal Penelitian Psikologi, 3(1), 73-100.
Suyata, P., Hidayanto, N., & Widyantoro, A. (2014). Standarisasi instrumen integrated assessment hasil belajar bahasa dengan program quest [The standardization of integrated assessment instruments for language learning outcomes using the quest program]. LITERA, 13(2), 364-367.
Syadiah, A. N., & Hamdu, G. (2020). Analisis rasch untuk soal tes berpikir kritis pada pembelajaran STEM di sekolah dasar [An application of the Rasch model for analyzing critical thinking test items in primary school STEM education]. Premiere Educandum: Jurnal Pendidikan Dasar dan Pembelajaran, 10(2), 138-148.
Tarigan, E. F., Nilmarito, S., Islamiyah, K., Darmana, A., & Suyanti, R. D. (2022). Analisis instrumen tes menggunakan rasch model dan software SPSS 22.0 [An application of the rasch model and SPSS 22.0 software for test instrument analysis]. Jurnal Inovasi Pendidikan Kimia, 16(2), 92-96.
Vera, R., Kaniawati, I., Judhistira, &, & Utama, A. (n.d.). Using the rasch model to develop a measure of students’ problem solving ability in optical instruments. Jurnal Pendidikan MIPA, 24(2), 419–431.
Yilmaz, H. B. (2019). A comparison of IRT model combinations for assessing fit in a mixed format elementary school science test. International Electronic Journal of Elementary Education, 11(5), 539–545.
Zaidi, M., Amiruddin, B., Samsudin, A., Suhandi, A., Kaniawati, I., Coştu, B., … Kuniawan, F. (n.d.). Validity and reliability of the global warming instrument: a pilot study using rasch model analysis. Jurnal Pendidikan MIPA, 24(4), 911–922.
Refbacks
- There are currently no refbacks.
Copyright (c) 2024 Jurnal Pendidikan MIPA
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
The copyright is reserved to The Jurnal Pendidikan MIPA that is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.