01 — Overview
The 30-second version
An organization was redesigning its platform for both mobile and desktop. Eight designers each created two versions — 16 total. The Lead Designer needed a clear winner, an understanding of why it won, and alignment to a set of locked brand attributes. The deadline was two weeks. There was no robust survey platform available. I had to invent the methodology.
I partnered with another researcher to design a hybrid approach that collected data from 200 participants across 8 groups, using word association and direct comparison surveys. The result: a defensible winner for both desktop and mobile, specific design elements mapped to specific brand attributes, and 12 designers with direct access to real user voices for the first time.
02 — The Problem
A clear question with no obvious answer
The Lead Designer's need was deceptively simple: identify the best design from 16 options and explain why it won. But the constraints made every standard approach unworkable.
The designs needed to be evaluated against specific new brand attributes — modern, bold, visually driven — that had been set by senior stakeholders and were not up for negotiation. This wasn't just about which design looked nicest. It was about which design best embodied a specific strategic direction.
The brief was: find the winner. But the real challenge was designing a methodology rigorous enough to stand behind.
The additional constraint was a hands-off Lead Designer who needed answers, not collaboration. That shifted the expectation: I needed to deliver a clear, well-supported recommendation rather than a facilitated decision-making process.
03 — Users & Audience
Who was in the room
Research participants: 200 people recruited to evaluate designs from a user perspective, responding to visual preference and brand attribute alignment questions. Each participant saw 4 designs — enough to make meaningful comparisons without overwhelming them.
Primary stakeholder: The Lead Designer, who needed a recommendation they could confidently act on.
Secondary audience: The 8 workshop designers plus 4 additional interested designers (12 total), who attended a feedback review workshop and interpreted the affinity map directly. This turned a standard research readout into a learning experience for the design team.
04 — Role & Responsibilities
What I owned
I served as lead researcher, partnering with a second researcher to design and execute the methodology. My responsibilities spanned every phase: evaluating methodology options, designing the hybrid approach, building and launching the surveys, coding and organizing responses, creating the FigJam affinity map, briefing the Lead Designer, and facilitating the feedback review workshop with 12 designers.
05 — Scope & Constraints
The walls we were working within
The constraints shaped every decision:
06 — Process
Ruling out the obvious, then building something new
We began by systematically evaluating every established approach — not to perform due diligence, but because each one taught us something about the solution space.
| Approach considered | Why we ruled it out |
|---|---|
| Semantic differential survey | Even showing only 4 designs per participant, the survey would have been too long and cognitively taxing to yield quality data. Ruled out |
| Balanced Incomplete Block Design | Statistically sound, but the data analysis would have taken longer than the entire timeline allowed. Ruled out |
| Bracket-style voting | Too many elimination rounds needed across 16 designs, each requiring separate fielding. Ruled out |
| MaxDiff analysis | Ideal for this type of question — but the tool wasn't available. (Would use in future.) Not available |
| Hybrid approach (our solution) | Word association + direct comparison + overall pick, split across 8 participant groups of 25. Feasible, fast, and rigorous enough to defend. Selected |
The final methodology worked as follows:
Color-coded affinity map in FigJam — organized by design, coded by sentiment
Rather than simply delivering a readout, I first walked the Lead Designer through the findings one-on-one, then facilitated a feedback review workshop with all 12 designers. In the workshop, the team reviewed the affinity map and generated their own takeaways from participant responses — giving them direct access to user voices and building their ability to interpret research firsthand.
07 — Outcomes & Impact
What the data delivered
08 — What I Learned
What I'd do differently — and what I'd keep
Running 8 separate surveys was operationally complex. The setup, management, and data consolidation required significant manual effort that wouldn't have been necessary with the right tooling.
If I had access to a tool that could run MaxDiff, I would have used it. It would have reduced 8 surveys to 1 and made analysis significantly faster. But it wasn't available — so we built something that worked instead.
What I would keep: the affinity map approach and the workshop format. Giving designers direct access to user feedback — rather than summarizing it for them — was unexpectedly powerful. It built intuition in a way a traditional readout never could.
The broader takeaway is one I carry into every constrained project: sometimes the best methodology isn't available, and you have to get creative with what you have. The measure of rigorous research isn't adherence to a single standard approach. It's designing something defensible given the actual conditions you're working in.