Browsing by Subject "http://www.yso.fi/onto/yso/p9970"

Objectives. Progressive Feedback in Early Childhood Education is a project for development of early childhood education in 12 Finnish municipalities. The data in the project is collected through child observation and staff and manager self-evaluation. The objective of this study was to assess the inter-rater agreement of the project’s observation method. Inter-rater agreement is a common way of assessing the reliability and validity of a method that is based on observational data collection. Methods. This study used paired observation data that was collected through the project. The data consisted of observations conducted by 9 different pairs of observers, generating 411 paired observations in total. The data was coded as quantitative data and quantitative methods were used for the reliability analysis. Cohen’s kappa was used to calculate the level of agreement for the rating system scales and for all the paired observers using the rating system. Cohen’s kappa calculates the level of agreement among raters and accounts for agreement based on chance. Cross-tabulation was used to examine the distribution of agreement and disagreement of the observational data on the rating scales. Results. Based on labels assigned by Landis and Koch (1977) the level of agreement for one rating scale was almost perfect, for two rating scales substantial, for three rating scales moderate and for three rating scales fair. The disagreement was notable on certain scales and there were clear differences between scales and raters. Based on the results and sources the reliability of the observational method should be improved in the future. The improvement should focus on increasing the training of the observers, evaluating the objectives of the observation and calibrating the observation rating scales. Using Cohen’s kappa was not problem-free, so in future research multiple methods should be used to ensure balanced results. Further studies include ways to follow up on the reliability of the project’s observation method and the possibilities of generalizability theory and argument based validity as part of the research on reliability and validity of observation based quality assessment methods.