The purpose of this study was to develop a valid and reliable rating scale to assess jazz big band performance and to evaluate the psychometric properties of jazz big bands at three performance achievement levels (e.g., low, moderate, high). The pool of initial scale items (N = 22) was gleaned from jazz big band research and instructional literature. Using a four-point Likert scale, rating responses (N = 102) were gathered for jazz ensemble performances (N = 102) from volunteer raters (N = 102). A factor analysis produced a reduced 18-item scale with a four-factor structure: blend/balance, time-feel, idiomatic nuance, and expression. The four factors accounted for 63.32% of the variance and had a total alpha reliability of .84. Discriminant function analyses revealed that four specific items contributed most to identifying ensembles with low and moderate performance ratings. The factor structure was able to predict group membership with 88.5% accuracy.