Most vendor evaluations focus on the wrong things
CTOs evaluate development partners by looking at tech stack expertise, team size, and hourly rates. These matter, but they're the easiest things to fake in a sales process. The factors that actually predict project success are harder to assess and rarely appear in RFPs.
After 20 years of being on both sides of this evaluation, here's what I'd look at.
The questions that actually matter
Do they push back on your requirements?
A vendor who agrees with everything you say is a vendor who will build exactly what you asked for, even when what you asked for is wrong. The best partners challenge assumptions early. They ask "why" before "how."
In our first meeting with IBI Investment House, we told them the mobile app should mirror their desktop platform. We pushed back. Mobile users check portfolios in 30-second bursts on the train. They need three screens, not thirty. IBI's app now has a 4.9-star rating because we designed for actual behavior, not assumed behavior.
Can they show you production code, not just portfolios?
Portfolios show screenshots. Ask to see their deployment pipeline, their monitoring dashboards, their incident response process. How do they handle a production outage at 2am? If they can't show you these, they're selling design, not engineering.
What's their attrition rate?
This is the metric vendors hate sharing. If their annual attrition is above 25%, you'll lose your senior developers mid-project. Ask directly, and ask to speak with engineers who've been there more than two years.
How do they handle scope changes?
Every project's scope changes. The question is whether your partner has a process for it or whether every change triggers a renegotiation. Look for partners who use iterative approaches where scope adjustment is built into the cadence, not treated as an exception.
Red flags that should end the conversation
No existing clients willing to speak with you. If they can't produce a reference from the last 12 months, ask why.
Technical decisions made by salespeople. If the person selling you the engagement is also architecting the solution, the architecture will be optimized for the sale, not the product.
No skin in the game. Partners who work exclusively time-and-materials have no incentive to deliver efficiently. Look for partners willing to commit to outcomes, not just hours.
Vague answers about security. Ask about their security practices, compliance certifications, and data handling policies. If the answer is "we take security seriously" without specifics, keep looking.
What a good partner engagement looks like
First month: Discovery and architecture. The partner should invest time understanding your business, not just your technical requirements.
Months 2-3: Working software in staging. Not wireframes, not architecture documents. Deployable code.
Ongoing: Shared dashboards, weekly demos, direct access to the development team. You should never have to ask for a status update.
Frequently asked questions
How many vendors should we evaluate? Three is enough. Five is too many. The evaluation process itself costs time. Interview deeply rather than broadly.
What should an RFP include? Skip the 40-page RFP. Send a 2-page brief describing the problem, the users, and the constraints. How vendors respond to ambiguity tells you more than how they respond to specifications.
How do we know if a partner is the right size? Your project should be meaningful to them but not their only client. If you're their biggest client, they'll over-accommodate you and under-challenge you.
Looking for a development partner who pushes back? Start a conversation with us.