Nuancing Conjoint Experiments: Using Natural Language Processing to Analyze Decision-Reasoning

Abstract

Understanding public preferences on policy issues is paramount, particularly in the realm of controversial policy areas like urban planning and housing. Existing survey techniques, however, may fall short of capturing the complexities of individual decision-making processes. This paper aims to bridge this gap by introducing a mixed-method approach for analyzing survey data, which combines survey experiments with the analysis of open-text responses using Natural Language Processing (NLP). This study demonstrates the potential of this approach through a large-scale survey involving 8,688 respondents across four major cities: Chicago, London, Los Angeles, and New York, focusing on housing densification. In detail, the paper compares revealed preferences, calculated through Individual Marginal Component Effects (IMCE) from a conjoint experiment, with stated preference questions and compares using open-text responses analyzed using NLP techniques. This approach yields various insights. For instance, we observed that participants who place a high value on resident participation tend to support housing densification initiatives more, often discussing specific project attributes, highlighting the advantages when residents are actively involved in urban planning. Moreover, the research reveals varying local factors driving opposition and concerns regarding densification projects in different urban contexts. Notably, there is an observed trend: as the preference for resident involvement increases, the resistance to projects, influenced by local contextual factors, tends to decrease. This suggests that those with a strong preference for involvement are potentially more open to the development of such projects. For example, individuals deriving high utility from resident participation in these projects often focused on specific project details rather than the broader concept of participation. These revelations not only augment our comprehension of public preferences but also establish the mixed-method approach as a novel, supplementary, and more comprehensive tool for unravelling the complexity of opinion formation in survey-based research.