1. Home ›
  2. Solution or Vendor Evaluation

Solution or Vendor Evaluation

1. Solution Evaluation

  • Generally, there are 2 types of commercially available software solutions:
    • Software as a Service (SaaS): a cloud-based model that allows users to access software applications over the internet
    • Off-the-Shelf (OTS) Software: Standardized software that is generally available commercially and can be used “as is”.
      • This may or may not be implemented as a web application.

Solution Selection Process:

  1. Check for Internal Solutions
    • Review the Business Applications list to determine if there are any existing software products being used at the university that satisfy the system requirements.
  2. Pursue Request for Quote / Proposal
    • If there are no existing software products at the university that satisfy the system requirements, work with Procurement Services to create a Request for Quote (RFQ) or Request for Proposal (RFP).
    • Procurement has an excellent process in place for communicating with potential vendors.
  3. Work with Procurement
    • The system requirements should be documented in a comprehensive, prioritized list prior to RFQ/RFP.
      • Procurement service has a tool that uses the format specified in the Requirements Specification Template (Spreadsheet) to automatically apply a list of response options for each requirement and perform automatic scoring when the vendor’s responses are received to provide a quick side-by- side comparison.
      • The list of response selections is:
        • A – Meets requirement (out of box)
        • B – Meets requirement (requires configuration)
        • C – Partially meets requirement (comment required)
        • D – Future release meets requirement (specify version & date in comment)
        • E – Does not meet requirement
  4. Review Vendor Responses
    • Review any documentation provided by the vendor in response to the RFQ/RFP.
    • The vendor should be required to respond to each of your requirements to convey their product’s ability to provide the needed functionality.
    • Ask for demonstration of critical functionality, ask questions about responses that need “configuration” or “partially meets”, ask for Sandbox access
    • The project team should select the top 3 products and ask for a product demonstration by the vendors.
  5. Participate in Solution Demonstrations
    • Before the product demonstration takes place, prepare a list of questions or critical functionality that you want to the vendor to address or display.
    • You should definitely ask for explanation about any “must” or “should” requirements for which the vendor responded with one of the following:
      • B – Meets requirement (requires configuration)
        • What does the needed configuration entail?
      • C – Partially meets requirement (comment required)
        • Ask for an explanation about what the product does/doesn’t do related to the requirement and perhaps a demonstration of the functionality that is provided
      • D – Future release meets requirement (specify version & date in comment)
        • Ask for a commitment to the specified version & date (or ask for a version & date, if not provided)
        • Ensure that the proposed future release will satisfy the requirement fully
    • Ask about if a Test or Sandbox will be provided and if there is an additional cost.
    • Ask for immediate access to a Sandbox environment so the user’s can have an opportunity to use an instance of the product to help with decision making.
    • Ask for references for existing higher education clients.
    • Pitfalls
      • BEWARE of the typical sales pitch which may focus only on what the vendor wants to show you instead of what you need to see demonstrated!
      • Don’t get derailed by the “smoke and mirrors show” and become so enamored by the bells and whistles that you forget to focus on the critical functionality.
        • A pre-defined list of questions will help alleviate these pitfalls and help you stay on track
  6. Assist with Solution Selection
    • Aside from requirement fulfillment, the following should also be considered when selecting a product:
      • Customization Needs: Assess whether the product allows minor configurations or if extensive customization is required
      • Integration Capabilities: Are integrations with existing systems available? Does the vendor use APIs that are approved by the Office of Information Technology Standards & Architecture board?
      • Customer Support: Are there defined guarantees (Service Level Agreements (SLAs)) in place for response times and problem resolution? Does the vendor offer resources such as manuals, tutorials, or user guides to help users understand and effectively use their products?
      • User Experience (UX): Is the user interface intuitive and easy to navigate? Are training resources available?
      • Security & Compliance: Does it meet Information Security standards and any applicable regulations like GDPR, HIPAA, or ISO standards?
      • Total Cost of Ownership (TCO): Consider license costs, maintenance fees, training, and customization.
      • Scalability & Future Growth: Can the software support future growth and additional features?
    • Offer business analysis insight to the project team to help them select the best possible solution
    • If Procurement Services was NOT involved, the Solution Decision Matrix can be used to rate and compare product/solution options.

2. Vendor Evaluation

When assessing a vendor to perform a service (including development of a product), consider the following key factors, as they directly affect the quality of the product or service to be provided, and how smoothly issues can be resolved when they arise:

  1. Vendor Experience & Expertise
    • What is the vendor’s specific experience with providing the applicable product or services for higher education customers?
      • Provide 3-5 recent and relevant case studies and/or recent client testimonials with quantifiable results.
    • What are the key qualifications and certifications of the project team members who will be assigned to our engagement?
    • Will the assigned project team members be dedicated to our project throughout its duration?
  2. Methodology & Approach
    • Describe your approach for providing the service needed, including your resources, tools used, data gathering, analysis, and reporting.
    • What resources would you need from Notre Dame and how much time do you anticipate needing from each to complete this engagement successfully?
    • How do you interact and/or communicate with customers throughout the process?
    • How do you address potential risks and challenges associated with customer projects?
  3. Deliverables
    • What specific deliverables will be provided at each stage of the process?
    • How will you ensure the timely, accurate and high quality delivery of all deliverables?
    • How will you present your findings and recommendations to our organization?
      • Will you provide executive summaries, presentations, and workshops?
  4. Vendor Reputation: A good vendor reputation indicates that they are dependable and provide value to their clients. It includes factors such as:
    • Trustworthiness: How reliable is the vendor in delivering quality products or services?
    • Customer Satisfaction: How satisfied are the customers with the vendor’s offerings?
    • Experience and Track Record: How long has the vendor been in business, and what is their history with delivering successful solutions or services?
    • Industry Recognition: Is the vendor recognized for excellence by industry standards, awards, or certifications?
  5. Vendor Support
    • Vendor Support: This focuses on the level of assistance, guidance, and services the vendor provides to their customers. It typically includes:
    • Customer Service: How responsive and helpful is the vendor’s support team when issues arise?
    • Technical Support: Is there assistance available for troubleshooting or resolving technical problems with the product or service?
    • Training and Documentation: Does the vendor offer resources such as manuals, tutorials, or user guides to help users understand and effectively use their products?
  6. Pricing & Engagement Model
    • Please provide a detailed cost breakdown for the proposed services, including hourly rates, fixed fees, and any potential travel expenses.
    • Describe your engagement model. Will it be fixed-fee, time-and-materials, or a hybrid approach? How will changes in scope be handled?
    • Describe the team you would assign to this engagement and when you could start.

A decision matrix should be used to rate and compare service providers (vendors).