Evaluating Design Choices: Conjoint Analysis and Experience Profiles

Originally designed to study consumer product preferences, conjoint analysis is a versatile methodology that is less commonly used in user experience design. This methodology involves presenting randomized “experience profiles” — combinations of design variations for different components in an experience. This lets us gauge how these variations influence user preference holistically. With this technique, design teams can explore diverse directions and identify the choices with the greatest potential impact.

What is Conjoint Analysis?

Think of the conjoint survey method as creating a “recipe” for a product people would love. Imagine you’re making a special dish with different ingredients to choose from.

Instead of asking people directly what they want, which can be confusing, you show them different versions of your dish with various ingredients mixed in. Then, you ask them to tell you which versions they like the most and which ones they like the least.

Doing this lets you determine which ingredients are the most important to people and which combinations they prefer.

From product profiles to experience profiles:

Conjoint analysis has traditionally been applied to physical products and quantifying consumer preferences. For example, for a new canned food product, attributes like flavor, size, price, and packaging would be tested. Different variations of each are tested, such as three flavors, two sizes, and so on.

For user experience design, conjoint analysis works similarly, but the “attributes” are interface or experience components instead of physical properties. For each component, different variations are defined that enhance or change the user experience in that area.

For an e-commerce product page, key experience attributes could be:

An example of how a product listing experience on an e-commerce website can be deconstructed into attributes and levels

Designing effective conjoint surveys for experience design can be challenging, as there are often no clear dividing lines between different components of the user experience. When deconstructing an experience into attributes to test, it’s important to consider how users actually interact with and perceive the product. Rather than arbitrary divisions, the components in a conjoint survey should correspond to key user tasks and flows.

In addition, the choice of where to demarcate different experience elements may be driven by practical considerations around testing and implementation. Components that are highly interdependent may need to be bundled together, while ones that can be more easily switched out can be separated. Additionally, if certain experience metrics need to be isolated, like the performance of specific CTAs, it can make sense to define attributes aligned with those goals.

The key is mapping attributes to meaningful differentiation points from the user’s perspective. While abstraction is necessary for a conjoint exercise, the attributes should still reflect realistic variants and priorities within the overall experience. Keeping the user journey in mind while deconstructing the experience can lead to more insightful results from conjoint analysis and better-informed experience optimization.

A meaningful deconstruction of the experience is crucial to a conjoint study

By mixing and matching variations of these attributes, different experience profiles can be generated and tested by survey platforms such as Qualtrics or Conjointly. This reveals insights like whether an accordion info layout or snapshots improve the product page experience over the control, or if AR imagery has more impact than image zoom.

An illustration of how randomized experience profiles can be generated by survey software such as Qualtrics based on the attributes and levels. In this illustration, the experience is deconstructed into 3 components, with different colors representing different levels of each attribute.

Results and Insights

Conjoint analysis provides a quantitative approach to understanding the drivers of customer preferences within an experience. Statistical techniques can be applied to survey data to uncover insights. First, the relative importance of each attribute is revealed, showing which elements have an outsized influence on the overall experience. This indicates where to prioritize design and development efforts for maximum impact.

Additionally, analyzing the preferences within each attribute exposes which variant users favor. For example, with navigation designs, statistical analysis could reveal the preferred placement and labeling of menu options. The inclusion of a control variant allows direct comparison to an existing experience to determine if new options will actually improve user preference.

An illustration of the relative utility value between levels of an attribute

For more information on the analysis of a conjoint study, you can find a more thorough explanation here.

Pros and Cons of This Approach

✅ Key advantages of conjoint analysis for experience design:

• Broader Design Exploration: Experience profiles enable testing of both safe and bold design directions in one setting. Ideally for the initial design phase without fixating on usability concerns.
• Efficient Testing: Randomized combinations in experience profiles reduce the need for numerous prototypes, saving time and resources.
• Assess Holistic Preferences: User preferences might not solely be driven by individual features but by the synergy of various elements in the overall experience. Experience profiles capture these interdependencies effectively.

❌ However, some limitations exist:

• Survey Length: Testing multiple combinations of experience profiles can increase survey length, especially when visual stimuli is included.
• Lack of Context: User experience is often more subjective and context-dependent than physical product attributes. Participants may make seemingly arbitrary choices based on visual appeal rather than true utility.
• Difficult to Pin Point Differences in Variations: It can be more difficult to clearly convey interface element levels compared to physical products. Participants may struggle to grasp the implications of variations like dropdowns versus sliders, unlike more concrete attributes like phone size

🛠 These can be mitigated by:

• ︎Enhance context: Participants can be selected from an intercept survey displayed on the experience you wanted to survey. Images can be shown in the order of the actual page layout to mimic the actual user experience. Usage scenarios and tasks to frame the ask can also be provided, i.e., provide feedback on the experience they were just using.
• Reduce Survey Complexity: Limit the number of experience profiles presented at a time to reduce the overwhelming effect of presenting complex experience profiles with multiple visuals. Give participants the option to select neither of the experience profiles presented.︎
• Pinpoint Variations: Include clear and concise descriptions to explain the differences between each variation.

In summary, conjoint analysis enables expansive design exploration and quantifies potential improvements. When applied thoughtfully, it yields valuable insights for optimizing user experience.

For more case studies, like this, please visit my website.