Item subset selection
First we calculate the cosine similarity for an item minus the maximum cosine similarity of that item with any other construct definition, and the observed cosine similarity between a given item and the mean similarity of that item with all other construct definitions. Based on human evaluations of the best performing items we can make subset selections for further AI psychometric analyses at the level of proposed scales. For example, in cases where transparency is desirable, it may be okay that the AI generated items mention the scale name in items as occurs in our examples. However, it may also be that greater diversity in items is preferred, in which case alternative items can be selected from the longer lists, or the prompt can be refined to introduce more diversity.
Pseudo discrimination density plots
In figure 1 we present density plots of the pseudo discrimination (semantic alignment) parameters where the peak of the distributions represent the most frequent pseudo discrimination values. What we see in these plots is that for each foundation, the pseudo discrimination values for the items from a given foundation are higher than the pseudo discrimination values for all other items. This is clear from the pattern where the distribution furthest to the right is the distribution of items for the given scale. We interpret these this as evidence that the A.I generated items are aligned to their construct definitions.
Checking semantic alignment validity
The data structure for this plot is a rows by columns data file where rows are items and columns contain the pseudo discrimination parameter between every item and every construct definition. We augment this data file with six binary item membership variables, one for each moral foundation. The augmented membership variables can be either the actual scale assignments or human rater allocations of items to scales.
The values in scale membership columns are 1 if the item is part of the scale and 0 otherwise. We present the the correlation matrix of the association between membership variables and the pseudo discrimination parameters. The largest correlations are between each scale’s membership variable and the pseudo that scales pseudo discrimination parameters, indicating strong item construct alignment.
Next section
Last section
Return home
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).