Rubrics Development on Product Design Innovation: Validity and Reliability Tests

Abstract

The skills in producing ideas of innovation at early stages of product design or concept are subjective in nature. Therefore, evaluation in gauging the best ideas for product innovation could be conducted through lecturers’ evaluation by using instruments based certain innovation criteria definitions. This study produces one instrument on evaluating ideas of product design innovation by independent raters (lecturers) using a rubric scale. This instrument consists of four main constructs as the basis of product innovation ideas evaluation which is novelty, quantity, variety and quality. In order to obtain the validity and reliability of instrument developed, Many-Facets Rasch Measurement (MFRM) was used with the following three facets: ideas of product design innovation, items assessed and raters (lecturers) involved.  These were later analysed with Facets 3.71.4 software. Raters were two product design lecturers at Ministry of Education (MoE) community college and subject assessed was thirty ideas of product design innovation that use Computer-Aided Design (CAD) and traditional method (sketching) in one of MoE community colleges. Findings from the pilot study show individual reliability for the four constructs were within the range 0.82-0.90. Validity of items assessed was also investigated through data fitting of the Rasch Measurement Model by observing the values of outfit and infit mean-squares. Two items were dropped and another two items were modified based on results obtained. Inter-rater agreements among raters were between 60.6% and 75.0% and all the percentages indicate values beyond expected percentage by Rasch Measurement Model. Findings from this pilot study indicate that Rubric instrument for evaluating ideas of product design innovation based on rubrics scale is appropriate to be used to assess ideas on product design innovation generated by design students.