Semantic convergent and discriminant validity

Prior to scale development, we recommend checking that the moral foundations are distinct from personality and values, if they aren’t they’ll have limited incremental utility. We use language models to encode the scale names and definitions and check the similarity with encodings of dimensions from prominent personality and values frameworks.

We’ll compare our proposed moral foundations definitions to the big five personality traits, the dimensions of the Hogan Development Survey (HDS) and the Career Anchors model of Edgar Schein. We’ll encode scale name-definition pairs using the MiniLM language model via sentence transformers from Hugging Face.

To begin we’ll need a file of definitions, you can download the file here for all of the analyses we are about to undertake. Here we choose Python which will be more flexible when we move to later sections of the book, although we will also use R for statistical analyses too. If Jupyter Notebook is not already installed, install it by running this line from the command line.

pip install notebook

Use the command line to navigate to the folder with your definitions file and type the following code and open a new Jupyter notebook (or select an existing notebook if you’re returning to this).

jupyter notebook

Now that we have switched from the command like to Jupyter we will import the libraries that we need for the analyses and import the definitions file. We show only the code for the first plot of the relationships between moral foundations and the big five, but the full Jupyter notebook can be downloaded here.

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
from sentence_transformers import SentenceTransformer

# Read the CSV file into a DataFrame
df = pd.read_csv('aipsych_moralfoundations.csv')

# Display first few rows to verify import
print(df.head())

Generate construct embeddings

Next we need to generate embeddings using a sentence transformer. This turns the construct definitions into numerical vectors. We will choose the MiniLM sentence transformer, which we expect will work well on these data, which we expect should perform well on short, psychological definitions. Note that there is a wide variety of types of sentence transformer and decisions would ideally be made based on conceptual criteria or benchmark comparisons of different models. It is often important that each numerical value of the embedding for construct definitions we create is accessible for the analyses we will undertake in this book. The following code will achieve this and store the embeddings alongside the construct definitions.

# Load model on CPU explicitly
model = SentenceTransformer('all-MiniLM-L6-v2', device='cpu')

# Generate embeddings
embeddings = model.encode(df['definition'].tolist(), batch_size=8, show_progress_bar=True)

# Add to DataFrame
embed_cols = pd.DataFrame(embeddings, index=df.index, columns=[f"emb_dim_{i}" for i in range(embeddings.shape[1])])
df = pd.concat([df, embed_cols], axis=1)

print(df.head())

Semantic validity analyses

In our next step, we calculate cosine similarities between construct definitions. While cosine similarity theoretically ranges from -1 (opposite meaning) to 1 (perfect alignment), in practice, similarities between embeddings typically range from 0 to 1 because negative similarities with embeddings are uncommon. Using the code below, we can create heat plots of the associations between moral foundations dimensions and each comparison construct as well as a heat plot amongst the moral foundations themselves.


# Filter rows based on 'framework'
maladaptive = df[df['framework'] == 'maladaptive']
moral = df[df['framework'] == 'moral']

# Extract labels
row_labels = maladaptive['label'].values

# Insert line breaks at "/" in the moral labels
col_labels = [label.replace('/', '/\n') for label in moral['label'].values]

# Identify embedding columns (assumes they start with 'emb_dim_')
embed_cols = [col for col in df.columns if col.startswith('emb_dim_')]

# Get embeddings for each group
maladaptive_embeddings = maladaptive[embed_cols].values
moral_embeddings = moral[embed_cols].values

# Compute cosine similarity: rows (maladaptive) vs. columns (moral)
cos_sim = cosine_similarity(maladaptive_embeddings, moral_embeddings)

# Create a high resolution figure
fig, ax = plt.subplots(figsize=(12, 10), dpi=300)
cax = ax.imshow(cos_sim, aspect='auto', cmap='bwr', interpolation='nearest', vmin=0, vmax=1)
plt.colorbar(cax)

# Set tick labels
ax.set_xticks(np.arange(len(col_labels)))
ax.set_xticklabels(col_labels, rotation=0)  # horizontal x-axis labels
ax.set_yticks(np.arange(len(row_labels)))
ax.set_yticklabels(row_labels)

ax.set_xlabel('Moral Foundations')
ax.set_ylabel('Maladaptive Traits')
ax.set_title('Cosine Similarity Heatmap')

plt.tight_layout()

# Save the figure as a JPEG image
plt.savefig('darktraits_heatmap.jpeg', format='jpeg', dpi=300)

plt.show()

Results

Figures 1 to 4 each contain the labels of moral foundations theory dimensions in the columns and the labels of either the big five model, the Hogan Development Survey, or Schein’s career anchors in the columns. In the cells themselves are shadings representing the strength of the relationship. Cosine similarity values are rarely negative for embeddings, and we see the scale of values for each chart begins at zero and stretches to one.

The absence of any red hue in the first three tables indicates that none of the moral foundation dimension associations with other constructs are greater than .50 and in fact are closer to the region of .20 to .40. This indicates low to moderate similarity. The closest relation between a moral foundations dimension and any other dimension have is between the moral foundations themselves (table 4). This indicates convergent and discriminant validity for moral foundations against these other constructs.

Figure 1. Moral foundations overlap with the Big Five

Figure 2. Moral foundations overlap with HDS

Figure 3. Moral foundations overlap with Schein’s Career Anchors

Figure 4. Moral foundations overlap within dimensions

Next section

AI item generation strategies

Last section

Do moral foundations matter in business?

Return home

Psychometrics.ai

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).