I have a few questions: For my study i have 3 raters: 1. For 3 raters for continuous variables, which one should be use - Fleiss Kappa (> more than 2 raters) or ICC (more than 2 raters) ? In this website : www.statisticshowto.datasciencecentral.com/fleiss-kappa/ says: the assumption with Cohen’s kappa is that your raters are deliberately chosen and fixed. If the raters are fixed, but i have 3 raters with continuous data should i use ICC or Cohen 2. Is it a must to use ICC for 3 raters? or in some situations, Cohen's kappa can be used?
Hi, if there are 4 observers amounting to 6 crosstabs (kappa scores), do I take the average of the 6 kappa scores? Or do I use the reliability to achieve the average measure? Sorry if my question doesnt make sense, I'm of no statistics background.
Hi, I have sequence data in which I have categories that have certain values rated by two raters. As a simplified example, assume that I have a sequence of colors; each color appears on the screen for a specific duration. The rater task is to make a note of the appearing color and the duration in a list form. Here, I have 3 things to compare in the same time: the categories, the sequence and the duration. Is there anyway to compare the result of two raters?
Thank you for the video. According to Gisev, N., et al. (2013). "Interrater agreement and interrater reliability: key concepts, approaches, and applications." Res Social Adm Pharm 9(3): 330-338. - ICC can be used for categorical, ordinal or continuous data, which is different to what you suggest here. Can you elaborate please ? Thanks
I did a bit of reading on it, and it seems that some people say you can and some people say you can't. Perhaps you can try using both techniques and post the result? I would suspect that there would be less variance when using categorical variables in the ICC compared to interval/ratio type variables, whereas Kappa and Fleiss made their formulae specifically for categorical variables.
Is "Inter-rater Reliability" a synonym for "Inter-coder" or "observer"-reliability?? Definitely a point of confusion for my Research Methods class and I!
I am assessing how participants' self-report of change in substance use (on a 5 point scale from low - no change - high), corresponds to their self-report of the same change measured at a different time point. Could I use cohen's kappa? I guess it is still intra-rater reliability. But a little confused as to whether cohen's kappa is appropriate when seeing agreement between two ratings by the same rater
@@PsychEDD thank you, why is that? Would a spearmans rho rank order correlation be appropriate then? Cant really find any examples or resources where this is explained
@@ishaandolli792 Perhaps I misunderstood initially, but i thought you wanted to measure the relationship between the two time points -- if it wasn't ordinal i would suggest a repeated measured test, but it is ordinal, so that wouldn't be appropriate. But yes I would say a spearmans rho would be most appropriate if you're taking a correlational approach.
I'm doing content analysis but I've already documented the items in Microsoft Word. I only have one rater and 500 samples with 7 categories of items. Can you suggest any ways that can calculate the reliability?
It was 2000, not 2009 - apologies. Portney LG & Watkins MP (2000) Foundations of clinical research Applications to practice. Prentice Hall Inc. New Jersey ISBN 0-8385-2695-0 p 560-567
Hello and thank you for the video, Do you know how to calculate a single inter-rater reliability ICC value when the 2 raters have both measured 2 or more trials? So rater1trial1 + rater2trial1, rater1trial2 + rater2trial2. Do you average the ICC values of those trials or is there another way?
I'm not sure about averaging the ICC values individually, but it seems that 2 separate interrater values would be more appropriate for your design, or even 4 if you want to determine the intrarater reliability between two trials. Additionally, you may want to try ICC(3, k) as it gives the average reliability for a set number of raters.
Thank you for the response. I think i need to clarify a bit. The 2 trials i am referring to have occured on the same day. So both my raters have tested the same patient twice, with some time (15 minutes) between. Articles with a similar setup allways refer to a single inter-rater reliability value, even if both raters (as in my case) have performed two measurements. I figured i had to analyse these measurements seperately in SPSS. Perhaps this is not the case? So in your example: Suppose rater 1 and rater 2 both performed 2 BRG trials (trial 1 and 2). Would you calculate the inter-rater reliability seperately for each trial? Or would you analyse the data simultaneously?
When I did the analysis for the data set included in the video, I was asked to do separate ICC scores for each trial (rater 1 trial 1, rater 2 trail 1, and so on). However, if the papers you are referring to reported just one score, I'm quite sure they used the ICC(3, k) model, which gives the average scores for a set number of raters. So you would add all of your variables into the ICC box and choose that model. Hope this helps.
As far as I am aware, Cohen's kappa is limited to 2 raters. However, if you are using ordinal data, check out en.wikipedia.org/wiki/Krippendorff%27s_alpha
hi, i have 41 raters from a psychology class who rated 19 different images of mens torsos for muscularity using a likert scale of 1 (low muscularity) and 5 (hypermusculrity). I thought ICC was appropriate but now I ma not so sure as each model has confused me. thanks
Perhaps have a read on the following: www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/ "The intra-class correlation (ICC) is one of the most commonly-used statistics for assessing IRR for ordinal, interval, and ratio variables. ICCs are suitable for studies with two or more coders, and may be used when all subjects in a study are rated by multiple coders, or when only a subset of subjects is rated by multiple coders and the rest are rated by one coder."
Thank you for the fast reply! I don't really get it. So, if we rate a certain characteristic with a scoring system from 1 to 4, where 1 is poor and 4 is excellent; how can I calculate it separately? Or did I explain it badly before and it is rather one variable with 4 categories?
