Pre-plan your RFP evaluation

 In blog, Commentary, Technology

Tools to help your RFP to set realistic expectations and deliver a focused review.

Over our history, ETA Transit has submitted hundreds of proposals for mass transit technology RFPs. We’ve won some, we’ve lost some, and we’ve seen RFPs pulled without warning. Win, lose, or draw our practice is to reach out to transit agencies and ask for feedback about the RFP—our proposal, where we excelled, where we came up short, and what struggles the agency had during the process. That’s the nature of our industry.

This last point serves as the subject for this article; what are the common challenges for proposal evaluation?

Our 2018 transit agency survey discovered that comparing offerings was the “biggest source of stress” for those on proposal decision committees, followed by “determining value versus price” and “determining the importance of features.” These are all valid concerns—but easily fixed.

The key to solving these issues starts with preparing the RFP itself. The same 2018 survey revealed that “preparation,” “creating technical requirements,” and “determining desired outcomes” were the top three headache-inducing tasks.

For the most part, an RFP includes the following components:

  • Cover letter or statement of interest
  • Vendor information
  • Proposed solution
  • Strategy for implementation
  • Warranty & support information
  • Cost proposal
  • Required forms

Each section should ultimately contribute to the decision-making process. It’s not uncommon for an evaluation structure included in the RFP which looks something like this:

  • Technical response: 20 points
  • Price: 40 points
  • Vendor qualifications: 20 points
  • Organization of the proposal: 10 points
  • Situational understanding: 10 points

And that’s about it in terms of the breakdown and weighting of the proposal response. That’s all a decision committee relies on to score and rank their evaluation. The highest score wins, and to the victor goes the spoils. But what if the evaluation criteria are wrong? What if it’s weighted incorrectly? Why does price weigh more than the solution itself (40% of transit agencies use it as their primary decision factor) than the vendor’s track record (vendor experience is the primary factor for just 11% of transit agencies)?

What mechanisms exist in your RFP for establishing value versus performance? Our 2019 survey revealed long-term performance (70.2%) must be part of the evaluation process. That certainly isn’t listed in the standard scoring criteria? Just 52 percent of transit agencies say that the intelligent transit system (ITS) they purchased lived up to expectations in our 2020 agency poll.

The picture I’m painting here is that the defined evaluation criteria baked into RFP documentation are inherently flawed; they are not up to the task and are highly subjective. Why do so many agencies value long-term performance, but fail to include any sort of measurement tool within their evaluation structure? Why are nearly half dissatisfied with their ITS purchase?

How do we change the evaluation so that it most accurately reflects the needs of the transit agency and delivers on the expected outcomes?

Here are a few ideas to consider before you put together your RFP:

Pre-proposal polling: Conduct an anonymous internal poll of your transit agency. If possible, include everybody. Everybody is drivers, mechanics, dispatchers, planners, customer support, and administration. Use this effort to discover hidden truths about your agency and potential issues. Look for potential areas of resistance. Find pain points in potential solutions. Include the ability for respondents to rank their answers to gauge priority and level of severity.

We recommend Survey Monkey, DooPoll, or even use Microsoft Sharepoint to create your surveys and leverage these tools automatically tally the results.

If you’d prefer a manual survey process, download this sample to get a jumpstart. Customize it as needed.

Conduct pre-RFP pricing research: We know that transit agencies (ETA Transit included) keep a tight lid on their pricing structure. It’s a competitive industry, after all, and any advantage helps. This practice doesn’t mean they won’t provide you with ballpark figures for planning purposes. Use this information to A) understand the approximate cost for each component (ITS, APC, AVA, etc.), and B) evaluate cost against the amount of funding you have available.

If possible, ask for an estimated ‘per vehicle cost’ to help account for hardware, installation, and recurring fees.

This effort will help you set expectations before writing your RFP. It is better to know what you’re asking for and compare it to the final pricing. With this baseline pricing in mind, it will help establish expectations for what your agency cannot afford. It will also help you derive a better sense of value from your proposals. It may even cause you to think about how much weight you place on pricing in the overall evaluation hierarchy.

> Use this spreadsheet to help plan costs versus features.

Create a compliance matrix with priority values

We’ve written before about the importance of including a compliance matrix as part of your RFP documentation. This approach is an excellent way to get answers about what a proposed system can and cannot deliver. What’s missing from the matrix is a correlation between features and their importance to the final solution. A compliance matrix evens the value across all items but does nothing to tell your evaluation committee how to weigh capabilities against price and competition. Including this information will help the vendor present prioritized solutions based on needs. For example, if your internal survey efforts reveal the following priorities:

  1. Cloud-based ITS
  2. Route and schedule management
  3. NTP reporting
  4. Automatic passenger counters
  5. Onboard Wi-Fi

Then assign value to those features within your matrix (a scale of 1–10, perhaps?). The vendors responding to your proposal will understand what you value the most. They can also construct their proposal to prioritize some aspects over others and deliver more up-front value.

A well-constructed spreadsheet may even include auto-calculation to help score each entry in advance. An entry in a compliance matrix theoretically might look like this, where “yes” answers to compliance earn full marks, where “no” responses don’t earn points, and “extra cost” applies a multiplier value to adjust the score to account for additional charges:

Example of weighted compliance matrix

> Download a sample value-driven compliance matrix.

Provide this spreadsheet as part of the RFP and ask vendors to complete it as part of their submission. Lock and hide those columns, cells, and worksheets from the file submitted to vendors. Reveal those fields after submission to see how a vendor stacks up based on their responses. Include in your calculations weighted penalties for unsupported systems.

Proposal creation

Now that you are adequately armed with priority preferences by role (via the internal survey), you will have more insight into how each business unit will react to your final choice. Additionally, you’ll have a sense of what you can afford and what systems to prioritize (using the ITS Price Estimator) and, ultimately, a point-by-point accounting of each vendor’s capabilities (courtesy of the weighted compliance matrix). With this information, you can create an RFP document that better captures the actual deliverables for your project and have the background data needed to conduct a proper evaluation.

With this data in hand, you can quickly rule out submissions below a capability threshold that you identify before evaluating. The goal, after all, is to narrow down proposals quickly. This approach:

  • Greatly assists in the preparation of the RFP
  • Clearly defines the technical requirements
  • It helps determine an agreed-upon, prioritized outcome
  • Reduces internal debate process

The process helps build a right-sized transit spec while avoiding common pitfalls and finding that vendor who will be your forever friend.

Now that you’re ready to construct your RFP, you can focus on telling a precise story—your story—so that any vendor responding has a full grasp of everything your system needs to accomplish. Construct your RFP to follow a logical progression and use clearly-defined topics and industry-standard terminology. Avoid copying and pasting from other sources, which typically results in a messy, contradictory document. We recommend a sequence that follows this general approach:

  1. Introduction (overview of your agency and history)
  2. Objectives (description of the challenges you seek to resolve)
  3. Resources (provide details about your agency, e.g., number of vehicles, makes and model, installed hardware, contact points, who has the final say, etc.
  4. Scope (define every deliverable, feature, and intended use—a good place to include your compliance matrix)
  5. Submission requirements (we recommend the following:)
    1. Statement of interest
    2. About the vendor
    3. Technical proposal
    4. Implementation plan
    5. Warranty & support
    6. Price proposal
    7. Required forms

A significant consideration in this process is to allow enough time for vendors to create a thoughtful response to your solution. A good rule of thumb is six to 12 weeks from the RFP’s issue (or street) date.

We also encourage you not to put page limits on the proposal. These highly technical and complex systems allow the vendor the necessary time to explain the system and benefits thoroughly. Yes, this can lead to some lengthy proposals. Consider the amount of money you are about to spend and the consequences of selecting poorly. In this decision, we highly stress that you should err on the side of information. Short summaries could cause a potential vendor to omit details that could otherwise influence your decision one way or another—all to satisfy an arbitrary page count.

The added information serves as a handy reference to answer any questions you may have.

Proposal evaluation

You’ve now laid the groundwork for your evaluation with all your pre-proposal work. The tools and concepts presented above provide your transit agency with enough background information to help identify potential vendors to look for before the issuance of the RFP—and then compare their response against the pre-RFP evaluation. More importantly, you have established a list of criteria based on priority gathered by stakeholders at all levels of your organization.

Take your time to evaluate the results. Ask questions. Conduct pilot programs. Take the necessary steps to ensure that you’re making a comprehensive decision in your agency, taxpayers, and riders’ best interest.

Recommended Posts
Contact Us

We're not around right now. But you can send us an email and we'll get back to you, asap.