## Meaning beyond lexicality This repository contains the scripts and data sources necessary to reproduce and/or expand on our paper, "Meaning beyond lexicality: Capturing Pseudoword Definitions with Language Models", accepted for publication in Computational Linguistics. In the paper, we adopt a exploratory-confirmatory approach to assess whether humans are able to assign non-random explicit definitions to **pseudowords**. We use transformer-based **language models** to measure the semantic similarity between (pseudo)word and definition embeddings, and evaluate if the similarity between them is higher for matching vs. non-matching (pseudo)word-definition pairs. We redirect the readers to the article for additional details. ### Data The data to replicate our study and to support other analyses into how humans interpret (pseudo)words is provided in the files `data/pseudoword_interpretations_v2.csv` (Exp. 1, exploratory) and `confirmatory/pseudoint_EXP2_all_newpart.csv` (Exp. 2, confirmatory). The data used in Exp. 1 comes from a previous experiment by [Gatti et al. (2024)][1]. Both datasets include free definitions for words and pseudowords from large samples of participants. ### Code The code for the exploratory experiment (Exp. 1) can be found in the main folder. The folder includes: - The code to extract the embeddings from the models and calculate the similarity between (pseudo)word and definition embeddings (`embedding_compare_models.py`) - The code to fit mixed effects models in Julia through R (`MixedModels.R`) - The code to perform the power analysis to establish the required sample size for Exp. 2 (`power_analysis.R`) The code for Exp. 2 can be found in the file `confirmatory/confirmatory_analysis.py`; it replicates the main analysis of Exp. 1 on new data and with a single model (GPT2-xl). [1]: https://link.springer.com/article/10.3758/s13423-024-02487-3