We propose adjusted inference procedures for evaluating the agreement/disagreement of two raters in a clustered setting involving twins or paired body parts. These procedures include the construction of a confidence interval for the kappa statistic, a related test of statistical significance and a formula that facilitates sample size estimation. The results of a simulation study suggest that a simple adjustment using an estimated design effect will provide valid inferences. The methods proposed are illustrated using an example from the literature.