Keller Williams Realty International (KWRI) · Case study
AI tool research and vendor evaluation
In a newly created AI research role, I established a process for testing tools against real agent tasks and turning the findings into recommendations for KW teams. The work extended from model comparisons to vendor evaluation and post-integration guidance.
Corporate headquarters · Industry & Learning. Research, prototypes, and enablement for the Keller Williams network of agents, Market Centers, and regional teams.
The starting point
Agents and Market Centers needed useful recommendations in a crowded AI market. Integrations and partnerships also needed a researched basis for deciding which vendors to pursue. A tool’s advertised features were not enough to establish whether it could complete the intended work.
My responsibility
I led hands-on evaluation, developed the reference library and benchmarks, usually initiated vendor contact, and supplied Industry & Learning recommendations. I also created and distributed guidance for using tools after integration.
Tools & methods
Arena (LMArena) · ChatHub · Platform testing · Agent interviews · Google Docs · HubSpot
Defining what a useful answer looked like
My first assignment included building a prompt library and pressure-testing prompts. I gathered expectations from real estate agents: what they would want the AI to do, what a valuable answer should contain, and how clearly it needed to explain the task.
I used Arena (LMArena) and ChatHub to compare responses across leading models. I checked task adherence, accuracy, and whether an agent could understand and use the answer, then refined the prompts and made task-specific platform recommendations. The prompt library and 10 recommended platforms were shared with KWRI-connected agents and through Jason Abrams’ roadshow.
Testing products against complete workflows
I researched platforms and tested whether they could carry out the specific task, rather than judging them only by a feature list. I considered reliability, compliance needs, evidence of actual usage, vendor maturity, and whether agents could depend on the tool over time.
I documented the findings in one-pagers, including useful options and tools with better alternatives available. The library indexed 108 evaluations: 71 marked Useful and 37 marked Better options available. A selected benchmark organized 40 tools into 13 job-to-be-done groups; a separate nine-tool social-content comparison examined local grounding, voice, and output quality.
Turning research into integration and partnership handoffs
I placed the evaluations in Google Docs for the integrations team to identify tools worth pursuing. The documents were then entered into HubSpot for partnerships and sponsorship outreach. My recommendations represented Industry & Learning endorsement of the tool’s fit.
I usually established the initial vendor relationship and supplied the research context. Integrations and partnerships took the relevant technical and commercial work forward. This process contributed to the Twin and RemyAI integrations and the rejig.AI partnership.
Addressing the adoption gap after a tool was recommended
During a Market Center visit, I heard that teams were paying for recommended platforms without knowing how to set them up or get value from them. That exposed a second problem: recommending a tool did not give the customer a usable workflow.
I created role-specific one-pagers for Agents, Market Centers, and brokerages, explaining what the platform did, how to set it up, and how to use it. I delivered the guides directly, followed up, and partnered with KW administrators to extend distribution. Approximately 15 Market Centers received them and gave positive feedback.
Outcome
Research connected to decisions and adoption
The research library and handoff process informed integration and partnership decisions. Completed organizational outcomes included the Twin and RemyAI integrations and the rejig.AI partnership. The adoption guides reached approximately 15 Market Centers through direct distribution and administrator collaboration.
Let’s connect
What could work better?
I’m interested in AI innovation and solutions roles where research becomes useful systems.